Warhammer_DLP_Helper/README.md

Warhammer DLP Helper

Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts): export a Discord channel, extract the (often base64-encoded) download links, download per host, extract the archives, sort the files into the library, and record a manifest that will feed the torrent stage. One concise Rust binary + webapp (the webapp is the next stage).

Ported from the original Python implementation, which this version replaced. The Rust port of … provenance notes in the module headers name the Python sources it was derived from; AGENTS.md holds the project rules and docs/SORTING.md the library/sorting contract.

Crate choices (as ported)

Concern	Original Python	Rust
7z extraction	7z CLI, cp437 name repair, /var/tmp fallback	sevenz-rust2 (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active)
zip extraction	7z CLI	zip crate
RAR extraction (critical)	7z CLI	unrar.rs (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (sevenz-rust2 ArchiveWriter) anyway
tar / gz / bz2 / xz	7z CLI	deviation: system binaries (tar, gzip, bzip2, xz) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical
HTTP client	requests	reqwest (blocking first, async where the webapp needs it)
sha256 (gofile website token, --hash manifest)	hashlib	sha2
Fuzzy matching (sorter)	difflib-style ratios	exact port of CPython difflib.SequenceMatcher in src/difflib.rs (NOT strsim — ratios must be bit-identical for the thresholds)
CLI	argparse	clap (subcommands below)
Webapp	—	axum (CLI parity first, webapp second)
Discord export	DCE docker container (external)	unchanged: shell out to docker

Port status

All original Python entry points are ported and live-validated (see below).

Rust module	Python source	Tests
src/state.rs	state.py	✓
src/globalstore.rs	globalstore.py	✓
src/manifest.rs	state.py append_manifest	✓
src/difflib.rs	CPython difflib	bit-identical on generated vectors
src/warhammer_data.rs + data/warhammer_data.json	warhammer_data.py	✓ (order-sensitive, IndexMap)
src/discord.rs	dlphelper/discord.py	✓ (DCE docker export, URL extraction)
src/gofile.rs	dlphelper/gofile.py	✓ (website-token goldens)
src/zero807.rs	dlphelper/zero807.py	✓ (curl subprocess contract)
src/extract.rs	tools/extract_all.py	✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, .broken/)
src/catalogue.rs	dlphelper/catalogue.py	✓ (all python test cases)
src/sorter.rs	dlphelper/sorter.py	✓ (all python test_sorter_units cases)
src/gwshop.rs	dlphelper/gwshop.py	✓ (Algolia client, facet partition)
src/orchestrate.rs	dlp.py pipeline	✓ (record/failed-urls/subs/assume-done; retry loops offline)
src/main.rs	all CLIs	smoke-tested (see below)

CLI

dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global] [--assume-done FILE]

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]

Notes:

• The Discord token goes --token → $DISCORD_TOKEN → hidden prompt, and is only ever passed to the DCE container via env, never argv.
• sorter --catalog defaults to data/gwshop_catalog.json, relative to the current working directory — pass --catalog when running elsewhere.
• sorter --audit defaults to <library>/../_logs/sorter_applied.jsonl (bookkeeping stays out of the library, which may be a seeded torrent).
• The sorter implements the rules in docs/SORTING.md — that document is the working agreement; change it first, then the code.
• Retry loops, cooldowns and the "one gofile guest account per run" rule are ported verbatim from dlp.py; a failure in one URL never aborts the run.
• Ctrl-C: state.json is saved after every URL; re-run to resume (the python "interrupted" record on the in-flight URL is not replicated).

Deviations from the python version (known, accepted)

• tar/gz/bz2/xz extraction shells out to the system binaries (7z CLI in python); 7z/zip/rar use native Rust.
• The extractor also recognizes .tgz (a gzip-compressed tar). The python suffix set never included it, so python left those files unextracted; this is a deliberate Rust-only addition.
• 0807.st folder pages (https://0807.st/f/<id>) are expanded: the page is parsed and each listed file is downloaded from https://0807.st/<id>?dl=1 (a 302 to the CDN, with the real file name in the page) into <out_dir>/downloads/0807/<page-id>/. Python (and earlier Rust) saved the HTML page itself; the folder .zip export is rejected by the site as too large, so per-file download is the only way in.
• fs::copy fallback in the sorter does not preserve mtime (python shutil.copy2 does).
• state.json omits "files": [] for records without files (python writes the empty list); readers of both versions accept both.
• sorter --json prints keys alphabetically (serde_json, no preserve_order); python printed them in dataclass order. Content is identical.
• Negative --retries/--limit values are rejected by clap (python accepted them with odd semantics).

Live validation (2026-10-09/11, channel 1455993943471820842)

Initial run with --since 5d: DCE export via docker (token env-only), URL extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account, file-level skip-existing, retry on network errors), 0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json resume, RAR extraction through unrar (1.4G archive -> STLs), nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files, 82 GiB.

Update run with --since 6d --fresh: re-exported the channel, kept 35 of 445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links were 0807 folder pages (/f/<id>): the new expansion fetched 12 files / ~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into downloads/0807/<page-id>/. 36/36 URLs ok, 144 files, 105 GiB.

Extraction of the whole run: 106 archives, 0 failed, all .tgz units expanded; then sorter --apply against the production library Warhammer_Library_Live (12k + 5.5k new hardlinks, skip-existing, 0 copied, low-confidence doc beside the library).

Still not live-tested: a full Algolia gwshop browse (needs credentials at runtime).

Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test