Rust rewrite of version-python (same repo, main branch still carries the flat python layout). Goal: one concise binary + webapp, eventually replacing the python tooling.

Why this structure

main stays untouched (flat python layout) until the rewrite reaches parity.
This branch re-homes the python code into version-python/ (via git mv, history preserved) and starts version-rust/ beside it.
version-python/AGENTS.md still documents the python stage — its rules (secrets, failure policy, idempotency, rate limiting) apply to the rewrite too and are the contract the Rust version must keep.

Crate choices (as ported)

Concern	Python today	Rust
7z extraction	7z CLI, cp437 name repair, /var/tmp fallback	`sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active)
zip extraction	7z CLI	`zip` crate
RAR extraction (critical)	7z CLI	`unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway
tar / gz / bz2 / xz	7z CLI	deviation: system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical
HTTP client	`requests`	`reqwest` (blocking first, async where the webapp needs it)
sha256 (gofile website token, --hash manifest)	hashlib	`sha2`
Fuzzy matching (sorter)	difflib-style ratios	exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds)
CLI	argparse	`clap` (subcommands below)
Webapp	—	`axum` (CLI parity first, webapp second)
Discord export	DCE docker container (external)	unchanged: shell out to docker

Port status

All python entry points are ported; live network paths are tested offline only (see "Untested live paths").

Rust module	Python source	Tests
`src/state.rs`	state.py	✓
`src/globalstore.rs`	globalstore.py	✓
`src/manifest.rs`	state.py append_manifest	✓
`src/difflib.rs`	CPython difflib	bit-identical on generated vectors
`src/warhammer_data.rs` + `data/warhammer_data.json`	warhammer_data.py	✓ (order-sensitive, IndexMap)
`src/discord.rs`	dlphelper/discord.py	✓ (DCE docker export, URL extraction)
`src/gofile.rs`	dlphelper/gofile.py	✓ (website-token goldens)
`src/zero807.rs`	dlphelper/zero807.py	✓ (curl subprocess contract)
`src/extract.rs`	tools/extract_all.py	✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`)
`src/catalogue.rs`	dlphelper/catalogue.py	✓ (all python test cases)
`src/sorter.rs`	dlphelper/sorter.py	✓ (all python test_sorter_units cases)
`src/gwshop.rs`	dlphelper/gwshop.py	✓ (Algolia client, facet partition)
`src/orchestrate.rs`	dlp.py pipeline	✓ (record/failed-urls/subs/assume-done; retry loops offline)
`src/main.rs`	all CLIs	smoke-tested (see below)

CLI

dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global] [--assume-done FILE]

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]

Notes:

The Discord token goes --token → $DISCORD_TOKEN → hidden prompt, and is only ever passed to the DCE container via env, never argv.
sorter --catalog defaults to version-python/data/gwshop_catalog.json (falling back to data/gwshop_catalog.json), both relative to the current working directory — pass --catalog when running elsewhere.
sorter --audit defaults to <library>/../_logs/sorter_applied.jsonl (bookkeeping stays out of the library, which may be a seeded torrent).
The sorter implements the rules in version-python/docs/SORTING.md — that document is still the working agreement; change it first, then both codes.
Retry loops, cooldowns and the "one gofile guest account per run" rule are ported verbatim from dlp.py; a failure in one URL never aborts the run.
Ctrl-C: state.json is saved after every URL; re-run to resume (the python "interrupted" record on the in-flight URL is not replicated).

Deviations from the python version (known, accepted)

tar/gz/bz2/xz extraction shells out to the system binaries (7z CLI in python); 7z/zip/rar use native Rust.
fs::copy fallback in the sorter does not preserve mtime (python shutil.copy2 does).
state.json omits "files": [] for records without files (python writes the empty list); readers of both versions accept both.
sorter --json prints keys alphabetically (serde_json, no preserve_order); python printed them in dataclass order. Content is identical.
Negative --retries/--limit values are rejected by clap (python accepted them with odd semantics).

Untested live paths (validated offline only, per the failure-policy rules)

Actual Discord channel export via the DCE container (needs a token) — the docker invocation and export.json parsing are unit-tested, the network path is not.
Live gofile downloads (guest account creation, ensure_token) — website token derivation is golden-tested against python; the HTTP calls are not.
Live 0807 downloads — the curl -A "Mozilla/5.0" -e "https://0807.st/" contract is kept, timeouts are per python (--max-time 1800), but no live fetch was made.
Real .rar extraction — the extractor wiring is in place (unrar crate, unrar runtime library present), but no real RAR archive was available for a live test.
gwshop was smoke-tested for request shaping only; a full Algolia browse needs valid credentials (algolia.json or env).

Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test