Warhammer_DLP_Helper/version-rust/README.md

version-rust

Rust rewrite of version-python (same repo, main branch still carries the flat python layout). Goal: one concise binary + webapp, eventually replacing the python tooling.

Why this structure

• main stays untouched (flat python layout) until the rewrite reaches parity.
• This branch re-homes the python code into version-python/ (via git mv, history preserved) and starts version-rust/ beside it.
• version-python/AGENTS.md still documents the python stage — its rules (secrets, failure policy, idempotency, rate limiting) apply to the rewrite too and are the contract the Rust version must keep.

Crate choices (as ported)

Concern	Python today	Rust
7z extraction	7z CLI, cp437 name repair, /var/tmp fallback	sevenz-rust2 (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active)
zip extraction	7z CLI	zip crate
RAR extraction (critical)	7z CLI	unrar.rs (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (sevenz-rust2 ArchiveWriter) anyway
tar / gz / bz2 / xz	7z CLI	deviation: system binaries (tar, gzip, bzip2, xz) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical
HTTP client	requests	reqwest (blocking first, async where the webapp needs it)
sha256 (gofile website token, --hash manifest)	hashlib	sha2
Fuzzy matching (sorter)	difflib-style ratios	exact port of CPython difflib.SequenceMatcher in src/difflib.rs (NOT strsim — ratios must be bit-identical for the thresholds)
CLI	argparse	clap (subcommands below)
Webapp	—	axum (CLI parity first, webapp second)
Discord export	DCE docker container (external)	unchanged: shell out to docker

Port status

All python entry points are ported; live network paths are tested offline only (see "Untested live paths").

Rust module	Python source	Tests
src/state.rs	state.py	✓
src/globalstore.rs	globalstore.py	✓
src/manifest.rs	state.py append_manifest	✓
src/difflib.rs	CPython difflib	bit-identical on generated vectors
src/warhammer_data.rs + data/warhammer_data.json	warhammer_data.py	✓ (order-sensitive, IndexMap)
src/discord.rs	dlphelper/discord.py	✓ (DCE docker export, URL extraction)
src/gofile.rs	dlphelper/gofile.py	✓ (website-token goldens)
src/zero807.rs	dlphelper/zero807.py	✓ (curl subprocess contract)
src/extract.rs	tools/extract_all.py	✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, .broken/)
src/catalogue.rs	dlphelper/catalogue.py	✓ (all python test cases)
src/sorter.rs	dlphelper/sorter.py	✓ (all python test_sorter_units cases)
src/gwshop.rs	dlphelper/gwshop.py	✓ (Algolia client, facet partition)
src/orchestrate.rs	dlp.py pipeline	✓ (record/failed-urls/subs/assume-done; retry loops offline)
src/main.rs	all CLIs	smoke-tested (see below)

CLI

dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global] [--assume-done FILE]

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]

Notes:

• The Discord token goes --token → $DISCORD_TOKEN → hidden prompt, and is only ever passed to the DCE container via env, never argv.
• sorter --catalog defaults to version-python/data/gwshop_catalog.json (falling back to data/gwshop_catalog.json), both relative to the current working directory — pass --catalog when running elsewhere.
• sorter --audit defaults to <library>/../_logs/sorter_applied.jsonl (bookkeeping stays out of the library, which may be a seeded torrent).
• The sorter implements the rules in version-python/docs/SORTING.md — that document is still the working agreement; change it first, then both codes.
• Retry loops, cooldowns and the "one gofile guest account per run" rule are ported verbatim from dlp.py; a failure in one URL never aborts the run.
• Ctrl-C: state.json is saved after every URL; re-run to resume (the python "interrupted" record on the in-flight URL is not replicated).

Deviations from the python version (known, accepted)

• tar/gz/bz2/xz extraction shells out to the system binaries (7z CLI in python); 7z/zip/rar use native Rust.
• The extractor also recognizes .tgz (a gzip-compressed tar). The python suffix set never included it, so python left those files unextracted; this is a deliberate Rust-only addition.
• 0807.st folder pages (https://0807.st/f/<id>) are expanded: the page is parsed and each listed file is downloaded from https://0807.st/<id>?dl=1 (a 302 to the CDN, with the real file name in the page) into <out_dir>/downloads/0807/<page-id>/. Python (and earlier Rust) saved the HTML page itself; the folder .zip export is rejected by the site as too large, so per-file download is the only way in.
• fs::copy fallback in the sorter does not preserve mtime (python shutil.copy2 does).
• state.json omits "files": [] for records without files (python writes the empty list); readers of both versions accept both.
• sorter --json prints keys alphabetically (serde_json, no preserve_order); python printed them in dataclass order. Content is identical.
• Negative --retries/--limit values are rejected by clap (python accepted them with odd semantics).

Live validation (2026-10-09/11, channel 1455993943471820842)

Initial run with --since 5d: DCE export via docker (token env-only), URL extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account, file-level skip-existing, retry on network errors), 0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json resume, RAR extraction through unrar (1.4G archive -> STLs), nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files, 82 GiB.

Update run with --since 6d --fresh: re-exported the channel, kept 35 of 445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links were 0807 folder pages (/f/<id>): the new expansion fetched 12 files / ~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into downloads/0807/<page-id>/. 36/36 URLs ok, 144 files, 105 GiB.

Extraction of the whole run: 106 archives, 0 failed, all .tgz units expanded; then sorter --apply against the production library Warhammer_Library_Live (12k + 5.5k new hardlinks, skip-existing, 0 copied, low-confidence doc beside the library).

Still not live-tested: a full Algolia gwshop browse (needs credentials at runtime).

Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test