# Warhammer DLP Helper

Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts):
export a Discord channel, extract the (often base64-encoded) download links,
download per host, extract the archives, sort the files into the library, and
record a manifest that will feed the torrent stage. One concise Rust binary +
webapp (the webapp is the next stage).

Ported from the original Python implementation, which this version replaced.
The `Rust port of …` provenance notes in the module headers name the Python
sources it was derived from; `AGENTS.md` holds the project rules and
`docs/SORTING.md` the library/sorting contract.

## Crate choices (as ported)

| Concern | Original Python | Rust |
|---|---|---|
| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
| zip extraction | 7z CLI | `zip` crate |
| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
| CLI | argparse | `clap` (subcommands below) |
| Webapp | — | `axum` (CLI parity first, webapp second) |
| Discord export | DCE docker container (external) | unchanged: shell out to docker |

## Port status

All original Python entry points are ported and live-validated (see below).

| Rust module | Python source | Tests |
|---|---|---|
| `src/state.rs` | state.py | ✓ |
| `src/globalstore.rs` | globalstore.py | ✓ |
| `src/manifest.rs` | state.py append_manifest | ✓ |
| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
| `src/main.rs` | all CLIs | smoke-tested (see below) |

## CLI

```
dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global] [--assume-done FILE]

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]
```

Notes:

- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
  only ever passed to the DCE container via env, never argv.
- `sorter --catalog` defaults to `data/gwshop_catalog.json`, relative to the
  current working directory — pass `--catalog` when running elsewhere.
- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
  (bookkeeping stays out of the library, which may be a seeded torrent).
- The sorter implements the rules in `docs/SORTING.md` — that document is the
  working agreement; change it first, then the code.
- Retry loops, cooldowns and the "one gofile guest account per run" rule are
  ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
  "interrupted" record on the in-flight URL is not replicated).

## Deviations from the python version (known, accepted)

- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
  (7z CLI in python); 7z/zip/rar use native Rust.
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
  suffix set never included it, so python left those files unextracted; this
  is a deliberate Rust-only addition.
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
  parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
  (a 302 to the CDN, with the real file name in the page) into
  `<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
  HTML page itself; the folder `.zip` export is rejected by the site as too
  large, so per-file download is the only way in.
- `fs::copy` fallback in the sorter does not preserve mtime
  (python `shutil.copy2` does).
- state.json omits `"files": []` for records without files
  (python writes the empty list); readers of both versions accept both.
- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
  python printed them in dataclass order. Content is identical.
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
  them with odd semantics).

## Live validation (2026-10-09/11, channel 1455993943471820842)

Initial run with `--since 5d`: DCE export via docker (token env-only), URL
extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
file-level skip-existing, retry on network errors), 0807 via curl
(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
82 GiB.

Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.

Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
expanded; then `sorter --apply` against the production library
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
low-confidence doc beside the library).

Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
runtime).

## Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

```
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
```