Review pass: fix bugs, silence clippy, correct docs

Bugs:
- extract: pick the OUTERMOST archive suffix (sfx.first(), not last())
  so x.tar.gz uses the gz->tar path
- gwshop: truncate the error snippet by chars, not bytes (UTF-8 panic)
- sorter: py_stem_split now matches pathlib (0 < i < len-1) so a
  trailing dot is not treated as a suffix
- sorter::apply_plan: a per-file link/copy/mkdir failure is logged and
  skipped instead of aborting the whole apply (never-abort contract)
- orchestrate/state: add State::mark_logged and use it for every
  per-URL record so a state-save error cannot abort a run

Cleanups:
- remove dead is_archive_name and unused SourceInfo.manifest field
- mark kept-but-unused capabilities with allow(dead_code) + rationale
- let-chains / is_ok_and / is_multiple_of / replace([..]) / sort_by_key
  and other idiomatic simplifications across modules
- difflib: SequenceMatcher owns its char vecs (bit-identical parity kept)
- --only now enforces gofile|0807 via value_parser

Docs:
- SORTING.md: drop phantom sorter_plan.jsonl, os.link -> std::fs::hard_link,
  zipfile -> the zip crate
- README: add --since to the dlp synopsis, clarify --subs/--assume-done
- .gitignore: ignore catalogue/sorter/_logs/subscriptions run artifacts

Validation: cargo fmt --check clean, cargo clippy --all-targets 0 warnings,
cargo test 87 passed / 0 failed.

.gitignore

@@ -1,24 +1,29 @@
-# Python
+/target
-__pycache__/
+
-*.pyc
+# Algolia credentials (search-only key, still never commit)
-.venv/
+algolia.json
-*.egg-info/
+gwshop_raw.json
+
+# Never commit credentials
+*.token
+.env
 
 # Run outputs (if OUT_DIR points into the repo)
 thread/
 downloads/
+extracted/
 export.json
 export.html
 state.json
 state.json.tmp
 failed_urls.txt
 manifest.jsonl
+catalogue.md
+catalogue.jsonl
+catalogue_moves.jsonl
+sorter_applied.jsonl
+sorter_low_confidence.txt
+subscriptions.json
+*.bak
+_logs/
 *.log
-
-# Never commit credentials
-*.token
-.env
-
-# Algolia credentials (search-only key, still never commit)
-algolia.json
-gwshop_raw.json

AGENTS.md

@@ -11,18 +11,25 @@ those channels point to two hosters and are often **base64-encoded** inside
 message content (obfuscated by the posting bot).
 
 The end goal is a **torrent-first platform** (think radarr/lidarr, but for
-proxy files). This repo is the current stage of that journey: a tool that
+proxy files). This repo is the Rust implementation of the current stage: one
+binary, `dlphelper`, that
 
 1. exports a Discord channel/thread (DiscordChatExporter via docker),
 2. extracts base64-encoded and plain download links,
 3. picks the right downloader per host automatically,
 4. downloads everything with per-file tracking, retries and resume,
-5. records a manifest that will later feed the torrent stage.
+5. extracts the archives into a plain-file tree,
+6. sorts the files into the library by system/group/faction/class/unit,
+7. records a manifest that will later feed the torrent stage.
 
-Deliberately **not** in scope (yet): .torrent creation, tracker upload,
+A webapp is the next stage. Deliberately **not** in scope (yet): `.torrent`
-deseeding, rename-into-library conventions. Design decisions should not close
+creation, tracker upload, deseeeding, rename-into-library conventions. Design
-the door on these — especially `manifest.jsonl` and the `--hash` option exist
+decisions should not close the door on these — especially `manifest.jsonl`
-for dedupe/cross-seeding later.
+and the `--hash` option exist for dedupe/cross-seeding later.
+
+This repository was ported from a Python implementation, which the Rust
+version has now replaced (the ported modules keep `Rust port of …` provenance
+notes in their headers).
 
 ## 2. Repository & environment
 
@@ -30,38 +37,37 @@ for dedupe/cross-seeding later.
   **private** repo, branch `main`. Push with the credentials already stored
   via git's credential store (`~/.git-credentials`); never paste them into
   commands or files.
-- The server is Gitea 1.27. To create *another* repo, use
+- Machine: Linux, Rust (edition 2024), docker is available. The repo lives on
-  `python3 tools/new_repo.py list|create <name>` (reads `~/.git-credentials`
+  **NFS**, where rustc SIGBUSes — always build/test with an explicit target
-  itself, never prints secrets). Never delete or overwrite existing repos.
+  directory:
-- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only
+
-  runtime dependency (see `requirements.txt`), docker is available.
+      CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
+
 - Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
   upstream reading material only.
 
 ## 3. Architecture
 
 ```
-dlp.py                     CLI orchestrator (entry point)
+Cargo.toml                  crate `dlphelper` (edition 2024)
-catalogue.py               CLI: classify the collection into system/faction/class
+src/main.rs                 CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
-sorter.py                  CLI: fuzzysort files into the library (plan/apply,
+src/orchestrate.rs          dlp pipeline: export -> extract URLs -> per-host download
-                           hardlinks, default catalog data/gwshop_catalog.json)
+src/discord.rs              DCE docker export (JSON) + base64/plain URL extraction
-dlphelper/
+src/gofile.rs               GoFile API client (one guest account per run)
-  discord.py               DCE docker export (JSON) + base64/plain URL extraction
+src/zero807.rs              curl downloader for 0807.st (direct links + /f/<id> folder pages)
-  gofile.py                GoFile API client (one guest account per run)
+src/state.rs                state.json store
-  zero807.py               curl-based downloader for 0807.st direct links
+src/globalstore.rs          cross-run URL store (~/.local/share/dlphelper/urls.json)
-  state.py                 state.json store + manifest.jsonl appender
+src/manifest.rs             manifest.jsonl appender
-  globalstore.py           cross-run URL store (~/.local/share/dlphelper/urls.json)
+src/extract.rs              recursive archive extraction into a plain-file tree
-  catalogue.py             scanner + keyword classifier + report writer
+src/catalogue.rs            scanner + keyword classifier + report writer
-  warhammer_data.py        taxonomy: GW-named systems/factions, GW shop
+src/sorter.rs               library sorter: inspect/classify/fuzzy-match, plan + apply
-                           categories as classes, GROUP_OF_FACTION map
+src/difflib.rs              faithful CPython SequenceMatcher port (bit-identical ratios)
-  gwshop.py                GW webshop catalog dump via Algolia (see §7.3)
+src/warhammer_data.rs       taxonomy tables (parsed from data/warhammer_data.json)
-  sorter.py                library sorter: inspect/classify/fuzzy-match
+src/gwshop.rs               GW webshop catalog dump via Algolia
-                           against the catalog, plan + hardlink apply
+src/util.rs                 small shared helpers
-parts/grab.sh              legacy groundstone script (kept, not used by dlp.py)
+data/gwshop_catalog.json    committed catalog snapshot (sorter default)
-tools/new_repo.py          Gitea repo helper (reads ~/.git-credentials itself)
+data/warhammer_data.json    committed taxonomy tables
-tools/debug_gofile.py      inspect raw gofile API response for one content id
+docs/SORTING.md             the library/sorting contract (change it before the code)
-tests/test_extract.py      offline extraction test (plain python3, no deps)
-tests/test_catalogue.py    offline classifier test
 ```
 
 Data flow:
@@ -78,6 +84,8 @@ graph LR
     F --> I[downloads/0807/...]
     E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
     J --> K[globalstore: urls.json, marks URL done forever]
+    J --> L[extract -> plain-file tree]
+    L --> M[sorter -> library, hardlinks + audit]
 ```
 
 Key facts about the hosters (learned the hard way, do not re-learn):
@@ -86,36 +94,48 @@ Key facts about the hosters (learned the hard way, do not re-learn):
   Creation is aggressively rate-limited — create **at most one account per
   run**. Content listing needs `Authorization: Bearer <token>`, an
   `X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
-  see `gofile.py`), and downloads need `Cookie: accountToken=<token>` plus
+  see `src/gofile.rs`), and downloads need `Cookie: accountToken=<token>` plus
   Referer/Origin headers.
 - **0807.st**: serves files directly but 403s anything that doesn't look like
   a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
   default multi-connection request does not. curl is the tool here.
+  Folder pages (`https://0807.st/f/<id>`) are expanded: the page is parsed and
+  each listed file is fetched from `https://0807.st/<id>?dl=1` into
+  `downloads/0807/<page-id>/`. The folder `.zip` export is refused by the site
+  ("folder too large"), so per-file download is the only route in.
 - Encoded links in the channels start as base64 strings matching
-  `aHR0cHM6Ly[89]...` — keep the extraction regex compatible with
+  `aHR0cHM6Ly[89]...`.
-  `parts/grab.sh`.
 
 ## 4. Usage (what "done" looks like)
 
     export DISCORD_TOKEN=...
-    python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>
+    dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash
 
-Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt`
+`--since 5d` (or `12h`, `2026-10-04`) limits a run to recent messages — handy
-(or just re-run — state.json skips everything already marked ok).
+for incremental updates. Re-runs are safe: `state.json` and the global store
+skip everything already marked ok.
 
-Several channels at once (`subscriptions.json`: list of {name, channel_id,
+Retry only failures:
-out_dir}):
 
-    python3 dlp.py --subs subscriptions.json
+    dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt
+
+Several channels at once (`subscriptions.json`: list of `{name, channel_id,
+out_dir}`):
+
+    dlphelper dlp --subs subscriptions.json
 
 Subs mode re-exports every channel by default (that is how new links get
-picked up) and relies on state.json + the global store to only download
+picked up) and relies on `state.json` + the global store to only download what
-what is actually new. Use a cron/systemd timer for watch-like behavior.
+is actually new. Use a cron/systemd timer for watch-like behavior.
 
-Manual import of the pre-dlp.py era (URLs you know are fully downloaded,
+Extract and sort:
-never re-fetch them again):
 
-    python3 dlp.py --assume-done old_urls.txt
+    dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
+    dlphelper sorter <out>/extracted --library <library> --apply
+
+Manual import of old URL lists (known fully downloaded, never re-fetch):
+
+    dlphelper dlp --assume-done old_urls.txt
 
 ## 5. Run artifacts (all in OUT_DIR, gitignored)
 
@@ -126,7 +146,11 @@ never re-fetch them again):
 | `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
 | `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
 | `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
-| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. |
+| `extracted/extract_manifest.jsonl` | one line per extraction `{archive, dest, unit}` (feeds `sorter --provenance`) |
+| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. `--global-state` moves it, `--no-global` disables it for a run. |
+
+Sorter bookkeeping (audit + low-confidence doc) is written to
+`<library>/../_logs/` by default — never inside the library (see §7).
 
 ## 6. Rules
 
@@ -136,27 +160,28 @@ never re-fetch them again):
 2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
    It is handed to the DCE container via the `TOKEN` env var, never as a
    command-line argument.
-3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy
+3. `~/.git-credentials` may only be read by tooling that needs it; never copy
    credentials into scripts, tests, or fixtures.
 4. Nothing downloaded and no export/state/manifest file is ever committed.
 
 ### Failure policy (the core requirement)
-5. A failure in one URL or file must **never** abort the run. Catch, record in
+5. A failure in one URL or file must **never** abort the run. Record it in
-   `state.json`, continue. The loops in `dlp.py` therefore have catch-alls —
+   `state.json` and continue. The per-URL loops in `src/orchestrate.rs`
-   keep them.
+   therefore have catch-alls — keep them.
 6. Cooldowns are the only permitted "stall": after N consecutive failures,
-   pause, then continue. Circuit-breakers that exit the process need a
+   pause, then continue. Circuit-breakers that exit the process need an
-   explicit justification in the PR/commit message.
+   explicit justification in the commit message.
 7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
-   Password-required gofile links are not retried (it can't help).
+   Password-required gofile links are not retried (it can't help); dead links
+   (HTTP 404/410) are never retried.
 
 ### Idempotency
-8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or**
+8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json`
-   the global store), skip existing non-empty files, write partial downloads
+   **or** the global store), skip existing non-empty files, write partial
-   to `*.part` and rename on success.
+   downloads to `*.part` and rename on success.
 9. `state.json` is the source of truth for resume; keep its schema backward
-   compatible when touching `state.py` (unknown keys are preserved, missing
+   compatible (unknown keys preserved, missing keys defaulted). Same for the
-   keys defaulted). Same for the global store schema in `globalstore.py`.
+   global store schema.
 10. `--assume-done` is a deliberate, manual import step for old URL lists.
     Never invoke it automatically, and never let the tool mark URLs done
     without a verified successful download.
@@ -170,86 +195,71 @@ never re-fetch them again):
     re-verifying the 403 behavior.
 
 ### Code style
-13. Python 3.12+, type hints on public functions, `logging` not `print` in
+13. Rust 2024, idiomatic. Use `log` (not `println!`) in library modules;
-    library code (prints are fine only for `--dry-run` output).
+    prints are fine only for CLI output (plan/dry-run).
-14. Stay on stdlib + `requests`. New dependencies need a real justification
+14. Keep new dependencies justified — the crate already covers HTTP
-    and belong in `requirements.txt`.
+    (`reqwest`), archives (`sevenz-rust2`, `zip`, `unrar`), fuzzy matching
-15. `dlphelper/` modules must stay import-safe (no side effects at import).
+    (the in-tree `difflib` port, **not** an external crate) and CLI (`clap`).
+    The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
+15. `src/difflib.rs` must stay bit-identical to CPython `difflib` — the sorter
+    thresholds depend on exact ratios; do not "simplify" it.
 
 ### Validation
-15. Minimum validation for any change: `python3 -m py_compile dlp.py
+16. Minimum validation for any change: `CARGO_TARGET_DIR=/var/tmp/dlphelper-target
-    catalogue.py dlphelper/*.py`, plus the offline tests (`python3
+    cargo test` (the offline suite must stay green). Document what you could
-    tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
+    *not* test (live gofile/0807/Discord paths need real links and tokens).
-    modules changed. Document what you could *not* test (e.g. live
+17. Full live runs need a Discord token from the user — ask for it, never ask
-    gofile/0807 downloads need real links and tokens).
+    for it to be pasted into a file or committed.
-17. Full live runs need a Discord token from the user — ask for it, never
-    ask for it to be pasted into a file or committed.
 
 ### Git
 18. Branch `main`, descriptive commit messages, no force-push, no history
     rewriting on the shared remote.
-19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
-    fine, deletion/renaming of existing repos requires the user.
 
-## 7. Roadmap (future stages, keep compatible)
+## 7. Sorter & library structure
 
-1. **Dedupe**: hash-based (sha256) duplicate detection across threads via
+`docs/SORTING.md` is the working agreement for the library hierarchy and the
-   `manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already
+sorting pipeline — **change it first, then the code**. Summary:
-   exists via the global store; content-level (same file, different link)
-   comes with hashing.
-2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
-   a watch dir, cross-seeding between direct-download and torrent copies.
-3. **Library organization (started)**: `catalogue.py` classifies every file
-   into system / faction / class and can move files into
-   `<library>/<System>/<Group>/<Faction>/<Class>/` with `--apply` (audited in
-   `catalogue_moves.jsonl`). The taxonomy lives in
-   `dlphelper/warhammer_data.py`; it is keyword-based and grows with the
-   collection - unclassified files are reported, never guessed. Manual fixes
-   go into an overrides JSON (`--overrides`). The library hierarchy follows
-   the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
-   working agreement for the sorting pipeline.
 
-   **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
+- Hierarchy: `<library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>`.
-   working, verified 2026-10-04 — how it was obtained, do not re-derive):**
+  Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
-   scraping warhammer.com directly is blocked — the shop HTML and all
+- Matching rules are fixed: the outer file name wins when it scores ≥
-   search/listing endpoints (`/en-WW/shop/*`,
+  threshold; inner archive names only vote as fallback. A fuzzy match never
-   `/app/resources/catalog/search/*`, `/app/resources/catalog/categories`)
+  jumps into another game's section (keyword systems constrain the catalog
-   sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any
+  pool; cross-system adoption needs `CROSS_SYSTEM_MIN = 0.75` and token
-   non-browser client, regardless of User-Agent; the sitemap declared in
+  containment). Deepest-matching-dir wins (junk dir names are never
-   robots.txt does not exist; deep paths under the allowed
+  candidates). Weak hits ≥ `WEAK_MIN = 0.50` are accepted but flagged
-   `/app/resources/catalog/product/*` reach a separate commerce backend
+  low-confidence. Tokenless normal adoption needs `+0.08`
-   (plain `404 Not found`) but the resource shape is unknown. What works:
+  (`TOKENLESS_BONUS`).
-   the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and
+- Sorting runs in the sorter's **default (non-provenance) mode**: every
-   its app-id plus the **public search-only API key** ship with the browser
+  file is classified on its own name (unclassified files land directly in
-   page; the user captured both from the browser devtools ("Copy as cURL"
+  `_unsorted/<file>`). `--provenance` (classify whole extraction dirs as
-   on one search request) and they are stored in the gitignored
+  units) exists but the live library does **not** use it; mixing the two
-   `algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the
+  modes creates duplicate placements.
-   key is search-only, never commit any other Algolia key. The full
+- `DROPPED_SYSTEMS` (default `{Middle-Earth}`) filters catalog entries at load
-   product index `prod-lazarus-product-en-ww` (~3500 products, plus one
+  and counts keyword hits as unclassified; `--drop-system`/`--keep-system`
-   sibling index per locale) is dumped with plain `requests`: the
+  adjust. The committed catalog snapshot keeps all systems.
-   public search-only key cannot use Algolia's `/browse` (403) and normal
+- Catalog source: the GW webshop via Algolia (`src/gwshop.rs`). The app-id +
-   pagination is capped at 1000 cumulative hits, so `gwshop.py`
+  public **search-only** key live in the gitignored `algolia.json` (or
-   **partitions the index on the `GameSystemsRoot` hierarchical facet** —
+  `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY`); never commit any other Algolia key.
-   each branch is fetched with an explicit filter, recursing deeper only
+  `data/gwshop_catalog.json` is the committed snapshot and the sorter default.
-   when a branch exceeds one page. (Live count 2026-10-04: 4183 records,
+- The dump partitions the index on the `GameSystemsRoot` hierarchical facet
-   of which 3491 are reachable via the facet; the ~692 left out are paints,
+  (the search-only key cannot `/browse`; plain pagination caps at 1000 hits).
-   Black Library books, White Dwarf issues, brushes, bases and gifts that
+  When a product is faceted under several branches of equal depth (e.g. a
-   carry no `GameSystemsRoot` path — no miniature kits among them.) Records are normalized into the
+  generic kit listed per chapter), the **first longest path wins** — Python's
-   source-agnostic catalog schema (system/group/faction/subfaction/class
+  `max(paths, key=len)` semantics, reproduced explicitly because Rust's
-   = the facet path, name/sku/productType attached) into
+  `max_by_key` returns the *last* maximum. Products carrying no
-   `data/gwshop_catalog.json`, which is the sorter's default catalog and
+  `GameSystemsRoot` path (paints, Black Library books, White Dwarf, bases,
-   is **committed as a snapshot**; refresh with
+  gifts — no miniature kits) are left out; the coverage shortfall is logged.
-   `python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog
+- Bookkeeping (`sorter_applied.jsonl`, `sorter_low_confidence.txt`) is written
-   sources stay pluggable and source-agnostic (miniset.net — reachable
+  to `<library>/../_logs/` by default — never inside the library, which may be
-   but stale, manual JSON overrides, `data/library_catalog.json`).
+  a seeded torrent whose file set is frozen.
-4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
-   primitive version; a real watch mode (per-subscription schedules, tear-off
-   notifications, discord pings on new links) builds on it.
 
 ## 8. Testing notes
 
-- `tests/` currently holds a synthetic `export.json` test for extraction
+- Tests live inline (`#[cfg(test)]`) in each module and run with `cargo test`.
-  (`tests/test_extract.py`, runnable with plain `python3`). Keep tests
+  Keep them dependency-free and offline — they must never hit
-  dependency-free and offline — they must never hit gofile/0807/Discord.
+  gofile/0807/Discord.
-- To exercise downloads without Discord: build a `--urls-file` from known
+- `tempfile` tests stage under `/var/tmp` when available, because `/tmp` is a
-  URLs and run with a scratch OUT_DIR.
+  small tmpfs that trips the extractor's free-space guard (`MIN_FREE_BYTES`).
+- To exercise downloads without Discord: build a `--urls-file` from known URLs
+  and run with a scratch OUT_DIR.

Cargo.toml

@@ -0,0 +1,31 @@
+[package]
+name = "dlphelper"
+version = "0.1.0"
+edition = "2024"
+description = "Torrent-first platform for Warhammer proxy models: Discord channel export, link extraction, per-host downloaders, manifest — Rust rewrite of the original Python tool."
+
+[dependencies]
+anyhow = "1"
+base64 = "0.22"
+chrono = "0.4"
+clap = { version = "4", features = ["derive"] }
+dirs = "6"
+env_logger = "0.11"
+fs4 = "0.13"
+hex = "0.4"
+indexmap = { version = "2", features = ["serde"] }
+log = "0.4"
+rand = "0.9"
+regex = "1"
+reqwest = { version = "0.12", features = ["blocking", "gzip", "json"] }
+rpassword = "7"
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+sha2 = "0.10"
+sevenz-rust2 = "0.23"
+unrar = "0.5"
+url = "2"
+zip = "2"
+
+[dev-dependencies]
+tempfile = "3"

README.md

@@ -1,111 +1,146 @@
 # Warhammer DLP Helper
 
-A small "radarr for Warhammer proxy files" tool: point it at a Discord
+Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts):
-channel/thread and an output directory, and it exports the channel, extracts
+export a Discord channel, extract the (often base64-encoded) download links,
-the encoded download links, and downloads everything - automatically choosing
+download per host, extract the archives, sort the files into the library, and
-the right downloader per host:
+record a manifest that will feed the torrent stage. One concise Rust binary +
+webapp (the webapp is the next stage).
 
-| Host        | Downloader | Notes |
+Ported from the original Python implementation, which this version replaced.
-|-------------|------------|-------|
+The `Rust port of …` provenance notes in the module headers name the Python
-| gofile.io   | built-in GoFile API client | one guest account per run (rate-limit friendly) |
+sources it was derived from; `AGENTS.md` holds the project rules and
-| 0807.st     | curl | needs browser-like headers (`-A`, `-e https://0807.st/`) |
+`docs/SORTING.md` the library/sorting contract.
 
-## Install
+## Crate choices (as ported)
 
-    pip install -r requirements.txt   # just `requests`
+| Concern | Original Python | Rust |
-    docker pull tyrrrz/discordchatexporter:stable
+|---|---|---|
+| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
+| zip extraction | 7z CLI | `zip` crate |
+| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
+| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
+| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
+| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
+| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
+| CLI | argparse | `clap` (subcommands below) |
+| Webapp | — | `axum` (CLI parity first, webapp second) |
+| Discord export | DCE docker container (external) | unchanged: shell out to docker |
 
-## Usage
+## Port status
 
-    export DISCORD_TOKEN="..."
+All original Python entry points are ported and live-validated (see below).
-    python3 dlp.py <channel_or_thread_id> /path/to/output
 
-The token can also be passed with `--token` (or you'll be prompted, input
+| Rust module | Python source | Tests |
-hidden). Useful options:
+|---|---|---|
+| `src/state.rs` | state.py | ✓ |
+| `src/globalstore.rs` | globalstore.py | ✓ |
+| `src/manifest.rs` | state.py append_manifest | ✓ |
+| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
+| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
+| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
+| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
+| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
+| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
+| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
+| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
+| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
+| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
+| `src/main.rs` | all CLIs | smoke-tested (see below) |
 
-    --urls-file failed_urls.txt   retry a previous failure list, no re-export
+## CLI
-    --password <pw>               for password-protected gofile links
-    --only gofile|0807            process only one host
-    --extract                     unzip archives into per-archive folders
-    --hash                        record sha256 per file in the manifest
-    --dry-run                     just show what would be downloaded
-    --fresh                       force a fresh channel export
 
-## Several channels / watching for updates
+```
+dlphelper dlp <channel_id> <out_dir> [--since WHEN] [--urls-file FILE] [--token]
+    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
+    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
+    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
+    [--global-state FILE] [--no-global]
 
-Put your channels into `subscriptions.json`:
+dlphelper dlp --subs FILE              # alternative mode: iterate every channel
+                                       # in the list (channel_id/out_dir unused)
 
-```json
+dlphelper dlp --assume-done FILE       # alternative mode: mark URLs as done, exit
-{
+
-  "subscriptions": [
+dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
-    {"name": "some-eldar",
+    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]
-     "channel_id": "1455993943471820842",
+
-     "out_dir": "/big/storage/warhammer/some-eldar"}
+dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
-  ]
+    [--apply] [--audit FILE] [--provenance FILE] [--json]
-}
+    [--drop-system NAME]... [--keep-system NAME]...
+
+dlphelper extract <SOURCES>... --out DIR
+
+dlphelper gwshop <out> [--index NAME] [--raw FILE]
 ```
 
-then run (e.g. from a daily cron/systemd timer):
+Notes:
 
-    python3 dlp.py --subs subscriptions.json
+- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
+  only ever passed to the DCE container via env, never argv.
+- `sorter --catalog` defaults to `data/gwshop_catalog.json`, relative to the
+  current working directory — pass `--catalog` when running elsewhere.
+- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
+  (bookkeeping stays out of the library, which may be a seeded torrent).
+- The sorter implements the rules in `docs/SORTING.md` — that document is the
+  working agreement; change it first, then the code.
+- `dlp --since` limits a run to recent messages (e.g. `5d`, `12h`,
+  `2026-10-04`); `--subs` and `--assume-done` are alternative modes that ignore
+  `<channel_id>`/`<out_dir>`.
+- Retry loops, cooldowns and the "one gofile guest account per run" rule are
+  ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
+- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
+  "interrupted" record on the in-flight URL is not replicated).
 
-Subs mode re-exports each channel to pick up new links; `state.json` and the
+## Deviations from the python version (known, accepted)
-global store make sure only genuinely new URLs are downloaded.
 
-## Cataloguing the collection
+- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
+  (7z CLI in python); 7z/zip/rar use native Rust.
+- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
+  suffix set never included it, so python left those files unextracted; this
+  is a deliberate Rust-only addition.
+- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
+  parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
+  (a 302 to the CDN, with the real file name in the page) into
+  `<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
+  HTML page itself; the folder `.zip` export is rejected by the site as too
+  large, so per-file download is the only way in.
+- `fs::copy` fallback in the sorter does not preserve mtime
+  (python `shutil.copy2` does).
+- state.json omits `"files": []` for records without files
+  (python writes the empty list); readers of both versions accept both.
+- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
+  python printed them in dataclass order. Content is identical.
+- Negative `--retries`/`--limit` values are rejected by clap (python accepted
+  them with odd semantics).
 
-`catalogue.py` classifies every downloaded file into game system, faction and
+## Live validation (2026-10-09/11, channel 1455993943471820842)
-force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
-Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
-taxonomy in `dlphelper/warhammer_data.py`:
 
-    # preview a dlp.py run dir (manifest + downloads)
+Initial run with `--since 5d`: DCE export via docker (token env-only), URL
-    python3 catalogue.py /path/to/run_dir
+extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
+file-level skip-existing, retry on network errors), 0807 via curl
+(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
+resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
+extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
+82 GiB.
 
-    # include the manual-era directories
+Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
-    python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
+445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
+were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
+~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
+`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
 
-    # catalogue arbitrary dirs only
+Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
-    python3 catalogue.py --scan /big/storage/manual
+expanded; then `sorter --apply` against the production library
+`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
+low-confidence doc beside the library).
 
-    # actually reorganize into a library tree (audited, only classified files)
+Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
-    python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
+runtime).
 
-Outputs `catalogue.md` (readable) and `catalogue.jsonl` (machine, with source
+## Build/test on NFS
-attribution per file). Unclassified files are listed, never guessed; fix them
-with an overrides JSON (`--overrides overrides.json`).
 
-## Already downloaded the old way?
+The repo lives on NFS; rustc SIGBUSes there. Use:
 
-`--assume-done` imports a plain URL list into the global store so those links
+```
-are never fetched again (no verification — use deliberately):
+CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
-
+```
-    python3 dlp.py --assume-done old_urls.txt
-
-## Output layout
-
-    OUT_DIR/
-      export.json           Discord channel export (cached between runs)
-      state.json            per-URL status; re-runs skip everything marked "ok"
-      failed_urls.txt       URLs that failed after all retries
-      manifest.jsonl        one line per downloaded file (url, source message, path, size)
-      downloads/
-        gofile/<content>/<folders>/<files>
-        0807/<id>.zip
-| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
-| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
-
-Runs are resumable and idempotent: interrupted downloads use `.part` files and
-finished ones are skipped on the next run. Individual failures never stop the
-run; after 3 consecutive failures it pauses for a cooldown (rate-limit
-protection) and continues.
-
-## Why not use Lysagxra/GoFileDownloader directly?
-
-That tool creates a new GoFile guest account **per URL** and calls
-`sys.exit(1)` on the first failure, which is exactly why runs died after ~6
-downloads. This project reimplements its (working) API scheme - headers,
-`X-Website-Token`, download links with the `accountToken` cookie - but creates
-one guest account per run, records failures per file, and keeps going.
-
-`parts/grab.sh` is the original groundstone script, kept for reference.

catalogue.py

@@ -1,118 +0,0 @@
-#!/usr/bin/env python3
-"""Catalogue CLI: classify the collection into system / faction / class.
-
-Examples:
-    # catalogue the dlp.py run dir (manifest + downloads)
-    python3 catalogue.py /path/to/run_dir
-
-    # additionally scan manual-era directories
-    python3 catalogue.py /path/to/run_dir --scan /big/storage/manual/Aeldari
-
-    # catalogue ONLY manual dirs, without any run dir
-    python3 catalogue.py --scan /big/storage/manual --out-md catalogue.md
-
-    # fix misclassifications (see overrides format below)
-    python3 catalogue.py <dir> --overrides overrides.json
-
-    # actually move files into <library>/<System>/<Faction>/<Class>/
-    python3 catalogue.py <dir> --apply --library /big/storage/warhammer_library
-
-overrides.json format (keys are file names or full paths, case-insensitive):
-    {
-      "some file.zip": {"faction": "Aeldari", "class": "Heavy Support",
-                        "system": "Warhammer 40,000"}
-    }
-
-Outputs (default into the first given dir, or cwd):
-    catalogue.md    human-readable, grouped system > faction > class
-    catalogue.jsonl one JSON line per file, with source attribution
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-import sys
-from pathlib import Path
-
-from dlphelper.catalogue import (
-    build_catalogue,
-    format_size,
-    apply_moves,
-    write_reports,
-)
-
-log = logging.getLogger("catalogue")
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Classify the Warhammer collection into factions and classes."
-    )
-    parser.add_argument("run_dir", nargs="?", type=Path,
-                        help="a dlp.py run dir (uses its manifest.jsonl and downloads/)")
-    parser.add_argument("--scan", action="append", default=[], type=Path, metavar="DIR",
-                        help="additional directory to scan for archive files (repeatable)")
-    parser.add_argument("--manifest", type=Path, default=None, metavar="FILE",
-                        help="explicit manifest.jsonl (default: <run_dir>/manifest.jsonl)")
-    parser.add_argument("--overrides", type=Path, default=None, metavar="FILE",
-                        help="manual classification overrides (JSON)")
-    parser.add_argument("--out-md", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--out-jsonl", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--apply", action="store_true",
-                        help="move files into the library structure (default: preview)")
-    parser.add_argument("--library", type=Path, default=None, metavar="DIR",
-                        help="library root for --apply (required with --apply)")
-    args = parser.parse_args()
-
-    logging.basicConfig(level=logging.INFO,
-                        format="%(asctime)s %(levelname)-7s %(message)s",
-                        datefmt="%H:%M:%S")
-
-    run_dir = args.run_dir
-    manifests = [args.manifest] if args.manifest else (
-        [run_dir / "manifest.jsonl"] if run_dir else []
-    )
-    scan_dirs = list(args.scan)
-    if run_dir and not args.manifest:
-        # run dir is scanned as a whole (covers files not yet in the manifest)
-        scan_dirs.append(run_dir)
-
-    if not manifests and not scan_dirs:
-        parser.error("give a run_dir or at least one --scan directory")
-
-    out_base = run_dir if run_dir else Path.cwd()
-    out_md = args.out_md or out_base / "catalogue.md"
-    out_jsonl = args.out_jsonl or out_base / "catalogue.jsonl"
-
-    entries = build_catalogue(manifests, scan_dirs, args.overrides)
-    write_reports(entries, out_md, out_jsonl)
-
-    classified = sum(1 for e in entries if e.faction != "Unclassified")
-    log.info("Catalogued %d files (%d classified) -> %s", len(entries), classified, out_md)
-
-    # compact per-faction summary to the console
-    summary: dict[tuple[str, str], list[int]] = {}
-    for e in entries:
-        key = (e.system, e.faction)
-        n, size = summary.get(key, (0, 0))
-        summary[key] = [n + 1, size + e.size]
-    for (system, faction), (n, size) in sorted(summary.items()):
-        log.info("  %-18s %-32s %4d files  %s", system, faction, n, format_size(size))
-
-    if args.apply:
-        if not args.library:
-            parser.error("--apply needs --library DIR")
-        if not args.run_dir:
-            parser.error("--apply needs a run_dir (files are moved out of it)")
-        moved = apply_moves(entries, args.library, args.library / "catalogue_moves.jsonl")
-        log.info("Moved %d files into %s (audit: catalogue_moves.jsonl)", moved, args.library)
-        # rewrite reports with updated paths
-        write_reports(entries, out_md, out_jsonl)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        sys.exit(130)

data/library_catalog.json

@@ -1,140 +0,0 @@
-{
-  "comment": "Canonical library catalog. Fields: system/group/faction/subfaction/class/name/aliases. Names and aliases are matched case-insensitively; subfaction and class may be null. Aligned with miniset.net faction names.",
-  "entries": [
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": "Space Wolves",
-      "class": "Fast Attack",
-      "name": "Thunderwolf Cavalry",
-      "aliases": ["thunder wolf", "thunderwolves", "thunder wolf cavalry"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": "Space Wolves",
-      "class": "Elites",
-      "name": "Wulfen",
-      "aliases": ["wulfen dreadnought"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of Chaos",
-      "faction": "Chaos Daemons",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Keeper of Secrets",
-      "aliases": ["keeper of secrets and shalaxi helbane", "shalaxi helbane", "keepers of secrets"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "Drukhari",
-      "subfaction": null,
-      "class": null,
-      "name": "The Twin Lance",
-      "aliases": ["twin lance"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Elites",
-      "name": "Terminator Squad",
-      "aliases": ["terminators", "terminator squad"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Elites",
-      "name": "Dreadnought",
-      "aliases": ["dread"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Captain",
-      "aliases": ["captain in gravis armour", "primaris captain"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Heavy Support",
-      "name": "Hellblaster Squad",
-      "aliases": ["hellblasters"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Troops",
-      "name": "Intercessor Squad",
-      "aliases": ["intercessors"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Commander Farsight",
-      "aliases": ["farsight"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Commander Shadowsun",
-      "aliases": ["shadowsun"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "Fast Attack",
-      "name": "Pathfinder Team",
-      "aliases": ["pathfinders"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "Dedicated Transport",
-      "name": "Devilfish",
-      "aliases": ["devil fish"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": null,
-      "name": "Assorted Drones",
-      "aliases": ["drones"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adepta Sororitas",
-      "subfaction": null,
-      "class": null,
-      "name": "Battle Sisters",
-      "aliases": ["battle sister squad", "battle sisters squad"]
-    }
-  ]
-}

dlp.py

@@ -1,544 +0,0 @@
-#!/usr/bin/env python3
-"""Warhammer DLP Helper.
-
-Give it a Discord channel/thread id and an output directory; it exports the
-channel, extracts the encoded download links (gofile.io and 0807.st), picks
-the right downloader for each link, and keeps going when individual links
-fail. Everything is recorded in state.json; failures also land in
-failed_urls.txt so a re-run (or --urls-file) retries exactly those.
-
-Already-downloaded URLs are recorded in a global store (default
-~/.local/share/dlphelper/urls.json), so a link that was downloaded for one
-channel is never re-downloaded for another - and re-runs of the same channel
-skip everything finished (see also --assume-done for importing old lists).
-
-Usage:
-    python3 dlp.py <channel_id> <out_dir> [options]
-    python3 dlp.py --subs subscriptions.json [options]
-    python3 dlp.py <channel_id> <out_dir> --urls-file failed_urls.txt
-    python3 dlp.py --assume-done old_urls.txt          # import past downloads
-"""
-
-from __future__ import annotations
-
-import argparse
-import getpass
-import hashlib
-import json
-import logging
-import os
-import random
-import sys
-import time
-import zipfile
-from itertools import chain
-from pathlib import Path
-
-from dlphelper import discord as dlp_discord
-from dlphelper import gofile as dlph_gofile
-from dlphelper import zero807 as dlph_zero807
-from dlphelper.globalstore import GlobalStore, default_global_path
-from dlphelper.state import State, append_manifest
-
-log = logging.getLogger("dlp")
-
-
-def sha256_of(path: Path) -> str:
-    digest = hashlib.sha256()
-    with path.open("rb") as fh:
-        for chunk in iter(lambda: fh.read(1024 * 1024), b""):
-            digest.update(chunk)
-    return digest.hexdigest()
-
-
-def resolve_token(cli_value: str | None) -> str:
-    if cli_value:
-        return cli_value
-    if os.environ.get("DISCORD_TOKEN"):
-        return os.environ["DISCORD_TOKEN"]
-    log.info("No Discord token given ($DISCORD_TOKEN); prompting (input hidden).")
-    token = getpass.getpass("Discord token: ")
-    if not token:
-        raise SystemExit("No token provided, aborting.")
-    return token
-
-
-def read_url_lines(path: Path) -> list[str]:
-    return [
-        line.strip()
-        for line in path.read_text(encoding="utf-8").splitlines()
-        if line.strip() and not line.strip().startswith("#")
-    ]
-
-
-def load_urls_file(path: Path) -> list[dlp_discord.FoundUrl]:
-    """Load a plain URL list (e.g. failed_urls.txt); no attribution available."""
-    found: dict[str, dlp_discord.FoundUrl] = {}
-    for url in read_url_lines(path):
-        host = dlp_discord.classify(url)
-        if host is None:
-            log.warning("Unsupported host in %s: %s", path.name, url)
-            continue
-        found.setdefault(url, dlp_discord.FoundUrl(url=url, host=host))
-    return sorted(found.values(), key=lambda f: f.url)
-
-
-def record_files(
-    state: State,
-    gstore: GlobalStore | None,
-    url: str,
-    files: list[dict],
-    source: dict,
-) -> None:
-    state.mark(url, status="ok", error=None, files=files)
-    if gstore is not None:
-        gstore.mark(url, channel=source.get("channel"), host=source.get("host"),
-                    file_count=len(files))
-    for f in files:
-        append_manifest(Path(source["manifest"]), {
-            "channel": source["channel"],
-            "url": url,
-            "host": source.get("host"),
-            "message_id": source.get("message_id"),
-            "timestamp": source.get("timestamp"),
-            "author": source.get("author"),
-            **f,
-        })
-
-
-def extract_zips(files: list[dict]) -> None:
-    """Optional: unzip each downloaded archive next to itself."""
-    for f in files:
-        path = Path(f["path"])
-        if path.suffix.lower() != ".zip" or not path.exists():
-            continue
-        target = path.with_suffix("")
-        if target.exists() and any(target.iterdir()):
-            log.info("Skip extracting %s (target exists)", path.name)
-            continue
-        log.info("Extracting %s -> %s", path.name, target.name)
-        try:
-            target.mkdir(parents=True, exist_ok=True)
-            with zipfile.ZipFile(path) as zf:
-                zf.extractall(target)
-        except (zipfile.BadZipFile, OSError) as exc:
-            log.warning("Extraction of %s failed: %s", path.name, exc)
-
-
-def run_gofile(
-    urls: list[dlp_discord.FoundUrl],
-    args: argparse.Namespace,
-    state: State,
-    source_by_url: dict[str, dict],
-    is_done,
-    gstore: GlobalStore | None,
-) -> None:
-    if not [f for f in urls if not is_done(f.url)]:
-        return  # nothing pending: no guest account creation, no API calls
-    client = dlph_gofile.GoFileClient(password=args.password)
-    try:
-        client.ensure_token()
-    except dlph_gofile.GoFileError as exc:
-        log.error("%s - marking all pending gofile links as failed.", exc)
-        for f in urls:
-            if is_done(f.url):
-                continue
-            state.mark(f.url, status="failed", error=str(exc), files=[])
-        return
-
-    consecutive_failures = 0
-    for f in urls:
-        if is_done(f.url):
-            log.info("Skip (already downloaded): %s", f.url)
-            continue
-        source = source_by_url[f.url]
-
-        for attempt in range(1, args.retries + 1):
-            try:
-                files_spec = client.list_files(f.url)
-                if not files_spec:
-                    raise dlph_gofile.GoFileEmpty("content exists but contains no files")
-                files: list[dict] = []
-                for relpath, name, link in files_spec:
-                    dest = Path(args.out_dir) / "downloads" / "gofile" / relpath / name
-                    size = client.download_file(link, dest)
-                    entry = {"path": str(dest), "size": size, "name": name}
-                    if args.hash:
-                        entry["sha256"] = sha256_of(dest)
-                    files.append(entry)
-                log.info("OK %s (%d files)", f.url, len(files))
-                record_files(state, gstore, f.url, files, source)
-                if args.extract:
-                    extract_zips(files)
-                consecutive_failures = 0
-                break
-            except dlph_gofile.GoFilePasswordRequired as exc:
-                # Retrying without a password never helps
-                log.error("PASSWORD REQUIRED %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileNotFound as exc:
-                # Dead links never come back - no retry
-                log.error("DEAD LINK %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileEmpty as exc:
-                # Emptied content is stable - no retry
-                log.error("EMPTY CONTENT %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileAuthError as exc:
-                # Token may have died mid-run (rate limit); re-auth once per attempt
-                log.warning("Auth error for %s: %s - re-authenticating", f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                client.reset()
-                try:
-                    client.ensure_token()
-                except dlph_gofile.GoFileError as reauth_exc:
-                    log.error("Re-authentication failed: %s", reauth_exc)
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except dlph_gofile.GoFileError as exc:
-                log.warning("Attempt %d/%d failed for %s: %s",
-                            attempt, args.retries, f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except Exception as exc:  # noqa: BLE001 - a bug must not stop the run
-                log.exception("Unexpected error for %s: %s", f.url, exc)
-                state.mark(f.url, error=f"unexpected: {exc}", files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except KeyboardInterrupt:
-                state.mark(f.url, error="interrupted", files=[])
-                raise
-        else:
-            rec = state.records.get(f.url, {})
-            state.mark(f.url, status="failed", error=rec.get("error", "unknown"), files=[])
-            consecutive_failures += 1
-            log.error("FAILED %s after %d attempts", f.url, args.retries)
-            if consecutive_failures >= args.cooldown_failures:
-                log.warning(
-                    "%d failures in a row - cooling down %ds before continuing "
-                    "(rate limit?). The run does not stop.",
-                    consecutive_failures, args.cooldown_seconds,
-                )
-                time.sleep(args.cooldown_seconds)
-
-
-def run_0807(
-    urls: list[dlp_discord.FoundUrl],
-    args: argparse.Namespace,
-    state: State,
-    source_by_url: dict[str, dict],
-    is_done,
-    gstore: GlobalStore | None,
-) -> None:
-    if not [f for f in urls if not is_done(f.url)]:
-        return
-    out_dir = Path(args.out_dir) / "downloads" / "0807"
-    consecutive_failures = 0
-    for f in urls:
-        if is_done(f.url):
-            log.info("Skip (already downloaded): %s", f.url)
-            continue
-        source = source_by_url[f.url]
-
-        for attempt in range(1, args.retries + 1):
-            try:
-                dest, size = dlph_zero807.download(f.url, out_dir)
-                entry: dict = {"path": str(dest), "size": size, "name": dest.name}
-                if args.hash:
-                    entry["sha256"] = sha256_of(dest)
-                log.info("OK %s (%s)", f.url, format_size(size))
-                record_files(state, gstore, f.url, [entry], source)
-                if args.extract:
-                    extract_zips([entry])
-                consecutive_failures = 0
-                break
-            except dlph_zero807.Zero807NotFound as exc:
-                # Dead links never come back - no retry
-                log.error("DEAD LINK %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_zero807.Zero807Error as exc:
-                log.warning("Attempt %d/%d failed for %s: %s",
-                            attempt, args.retries, f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except Exception as exc:  # noqa: BLE001 - a bug must not stop the run
-                log.exception("Unexpected error for %s: %s", f.url, exc)
-                state.mark(f.url, error=f"unexpected: {exc}", files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except KeyboardInterrupt:
-                state.mark(f.url, error="interrupted", files=[])
-                raise
-        else:
-            rec = state.records.get(f.url, {})
-            state.mark(f.url, status="failed", error=rec.get("error", "unknown"), files=[])
-            consecutive_failures += 1
-            log.error("FAILED %s after %d attempts", f.url, args.retries)
-            if consecutive_failures >= args.cooldown_failures:
-                log.warning(
-                    "%d failures in a row - cooling down %ds before continuing "
-                    "(HTTP 403 storm?). The run does not stop.",
-                    consecutive_failures, args.cooldown_seconds,
-                )
-                time.sleep(args.cooldown_seconds)
-
-
-def format_size(n: int) -> str:
-    size = float(n)
-    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
-        if size < 1024 or unit == "TiB":
-            return f"{size:.1f} {unit}"
-        size /= 1024
-    return f"{n} B"
-
-
-def write_failed_urls(out_dir: Path, state: State) -> Path:
-    dest = out_dir / "failed_urls.txt"
-    dest.parent.mkdir(parents=True, exist_ok=True)
-    failures = state.failures()
-    dest.write_text("\n".join(failures) + ("\n" if failures else ""), encoding="utf-8")
-    return dest
-
-
-def mark_assume_done(path: Path, gstore: GlobalStore | None) -> int:
-    """Record URLs as already downloaded (no files) so they are never fetched.
-
-    Manual import step for the pre-dlp.py era: point it at an old URL list you
-    know is fully downloaded. There is no verification - garbage in, skipped
-    forever.
-    """
-    if gstore is None:
-        raise SystemExit("--assume-done needs the global store (drop --no-global)")
-    count = 0
-    for url in read_url_lines(path):
-        gstore.mark(url, channel=None, host=dlp_discord.classify(url), file_count=0,
-                    note="assumed done via --assume-done")
-        count += 1
-    log.info("Marked %d URLs as done in the global store (%s)", count, gstore.path)
-    return count
-
-
-def process_channel(
-    channel_id: str,
-    out_dir: Path,
-    args: argparse.Namespace,
-    token: str | None,
-    gstore: GlobalStore | None,
-    force_export: bool = False,
-) -> int:
-    """Run the full pipeline for one channel; returns the number of failures."""
-    state = State(out_dir / "state.json")
-    manifest = out_dir / "manifest.jsonl"
-
-    def is_done(url: str) -> bool:
-        return state.is_done(url) or (gstore is not None and gstore.is_done(url))
-
-    if args.urls_file:
-        found = load_urls_file(args.urls_file)
-    else:
-        if token is None:
-            raise SystemExit("internal error: no token resolved")
-        export_path = dlp_discord.export_channel(
-            channel_id, out_dir, token, force=force_export or args.fresh
-        )
-        found = dlp_discord.extract_urls(export_path)
-
-    if args.only:
-        found = [f for f in found if f.host == args.only]
-
-    buckets: dict[str, list[dlp_discord.FoundUrl]] = {"gofile": [], "0807": []}
-    for f in found:
-        if f.host in buckets:
-            buckets[f.host].append(f)
-        else:  # classify() already filtered, but be defensive
-            log.warning("Skipping unsupported URL: %s", f.url)
-            state.mark(f.url, status="failed", error="unsupported host")
-
-    source_by_url: dict[str, dict] = {
-        f.url: {
-            "channel": channel_id,
-            "host": f.host,
-            "message_id": f.message_id,
-            "timestamp": f.timestamp,
-            "author": f.author,
-            "manifest": str(manifest),
-        }
-        for f in found
-    }
-
-    # --limit applies per host class to the *pending* URLs, so a test run
-    # downloads exactly N new things per host (done ones don't use up the quota)
-    if args.limit is not None:
-        for key, items in buckets.items():
-            pending = [f for f in items if not is_done(f.url)]
-            if args.shuffle:
-                random.shuffle(pending)
-            buckets[key] = pending[: args.limit]
-
-    pending = [f for f in found if not is_done(f.url)]
-    if args.limit is not None:
-        pending = [f for f in chain.from_iterable(buckets.values()) if not is_done(f.url)]
-    log.info(
-        "%s: %d URLs (%d gofile, %d 0807); %d done, %d to download",
-        channel_id, len(found),
-        len(buckets["gofile"]), len(buckets["0807"]),
-        len(found) - len(pending), len(pending),
-    )
-
-    if args.dry_run:
-        selected = {f.url for f in chain.from_iterable(buckets.values())}
-        for f in found:
-            if f.url in selected:
-                flag = "download"
-            elif is_done(f.url):
-                flag = "done"
-            else:
-                flag = "skip"  # cut by --limit / --only
-            print(f"[{f.host:6}] [{flag:8}] {f.url}")
-        return 0
-
-    if buckets["gofile"]:
-        run_gofile(buckets["gofile"], args, state, source_by_url, is_done, gstore)
-    if buckets["0807"]:
-        run_0807(buckets["0807"], args, state, source_by_url, is_done, gstore)
-
-    failed_file = write_failed_urls(out_dir, state)
-
-    ok = [u for u, r in state.records.items() if r.get("status") == "ok"]
-    bad = state.failures()
-    dead = sum(1 for r in state.records.values() if "dead link" in (r.get("error") or ""))
-    total_files = sum(len(r.get("files") or []) for r in state.records.values())
-    total_size = sum(
-        (f_.get("size") or 0) for r in state.records.values() for f_ in (r.get("files") or [])
-    )
-    log.info("%s: %d ok (%d files, %s) / %d failed (%d dead links) / %d seen",
-             out_dir, len(ok), total_files, format_size(total_size),
-             len(bad), dead, len(state.records))
-    if bad:
-        log.info("Failed URLs written to %s - re-run this tool or use "
-                 "python3 dlp.py <id> <out> --urls-file %s to retry them.",
-                 failed_file, failed_file.name)
-    return len(bad)
-
-
-def load_subscriptions(path: Path) -> list[dict]:
-    """Load subscriptions.json: [{"name": ..., "channel_id": ..., "out_dir": ...}]."""
-    data = json.loads(path.read_text(encoding="utf-8"))
-    entries = data.get("subscriptions", []) if isinstance(data, dict) else data
-    if not isinstance(entries, list):
-        raise SystemExit(f"{path}: expected a list or {'{'}\"subscriptions\": [...]{'}'}")
-    subs = []
-    for entry in entries:
-        if not isinstance(entry, dict) or "channel_id" not in entry or "out_dir" not in entry:
-            raise SystemExit(
-                f"{path}: each subscription needs 'channel_id' and 'out_dir' "
-                f"(got: {entry})"
-            )
-        subs.append(entry)
-    return subs
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Collect downloads from Discord channels (gofile + 0807)."
-    )
-    parser.add_argument("channel_id", nargs="?", help="Discord channel or thread id")
-    parser.add_argument("out_dir", nargs="?", help="directory for downloads and state")
-    parser.add_argument("--subs", type=Path, metavar="FILE",
-                        help="process a subscription list instead of one channel "
-                             "(entries: channel_id, out_dir, optional name)")
-    parser.add_argument("--token", help="Discord token (else $DISCORD_TOKEN or prompt)")
-    parser.add_argument("--urls-file", type=Path,
-                        help="skip the Discord export; read URLs from this file "
-                             "(e.g. failed_urls.txt)")
-    parser.add_argument("--password", help="password for password-protected gofile links")
-    parser.add_argument("--retries", type=int, default=3, help="attempts per URL (default 3)")
-    parser.add_argument("--retry-delay", type=int, default=5,
-                        help="base seconds between retries, multiplied per attempt (default 5)")
-    parser.add_argument("--cooldown-failures", type=int, default=3,
-                        help="consecutive failures before a cooldown pause (default 3)")
-    parser.add_argument("--cooldown-seconds", type=int, default=60,
-                        help="length of the cooldown pause (default 60)")
-    parser.add_argument("--only", choices=("gofile", "0807"),
-                        help="process only one host class")
-    parser.add_argument("--limit", type=int, metavar="N",
-                        help="download at most N pending URLs per host class "
-                             "(test runs)")
-    parser.add_argument("--shuffle", action="store_true",
-                        help="randomize which URLs a --limit picks")
-    parser.add_argument("--hash", action="store_true",
-                        help="record sha256 of each downloaded file")
-    parser.add_argument("--extract", action="store_true",
-                        help="unzip downloaded archives into per-archive folders")
-    parser.add_argument("--fresh", action="store_true",
-                        help="re-export the Discord channel even if export.json exists "
-                             "(subs mode does this by default)")
-    parser.add_argument("--no-fresh", action="store_true",
-                        help="subs mode: reuse cached exports instead of re-exporting")
-    parser.add_argument("--dry-run", action="store_true",
-                        help="export/extract/classify only, download nothing")
-    parser.add_argument("--global-state", type=Path, metavar="FILE",
-                        help=f"cross-run URL store (default: {default_global_path()})")
-    parser.add_argument("--no-global", action="store_true",
-                        help="disable the cross-run URL store for this run")
-    parser.add_argument("--assume-done", type=Path, metavar="FILE",
-                        help="mark every URL in the file as already downloaded, "
-                             "then exit (manual import of old download lists)")
-    args = parser.parse_args()
-
-    logging.basicConfig(
-        level=logging.INFO,
-        format="%(asctime)s %(levelname)-7s %(message)s",
-        datefmt="%H:%M:%S",
-    )
-
-    gstore = None
-    if not args.no_global:
-        gstore = GlobalStore(args.global_state or default_global_path())
-
-    if args.assume_done:
-        if args.channel_id or args.subs:
-            parser.error("--assume-done cannot be combined with a channel or --subs")
-        mark_assume_done(args.assume_done, gstore)
-        return
-
-    if args.subs:
-        if args.channel_id or args.out_dir or args.urls_file:
-            parser.error("--subs cannot be combined with channel_id/out_dir/--urls-file")
-        subs = load_subscriptions(args.subs)
-        token = resolve_token(args.token)
-        total_failed = 0
-        for entry in subs:
-            out_dir = Path(entry["out_dir"])
-            log.info("=== subscription: %s ===", entry.get("name") or entry["channel_id"])
-            total_failed += process_channel(
-                entry["channel_id"], out_dir, args, token, gstore,
-                force_export=not args.no_fresh,  # subs mode re-exports by default
-            )
-        if total_failed:
-            log.warning("%d URL(s) failed in total across subscriptions.", total_failed)
-            sys.exit(1)
-        return
-
-    if not args.channel_id or not args.out_dir:
-        parser.error("channel_id and out_dir are required (or use --subs/--assume-done)")
-
-    # A token is only needed when we will export a channel (--urls-file skips that)
-    token = None if args.urls_file else resolve_token(args.token)
-    failures = process_channel(args.channel_id, Path(args.out_dir), args, token, gstore)
-    if failures:
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        log.warning("Interrupted - state.json is saved; re-run to resume.")
-        sys.exit(130)

dlphelper/__init__.py

@@ -1,3 +0,0 @@
-"""Warhammer DLP Helper - collect and organize files shared in Discord channels."""
-
-__version__ = "0.1.0"

dlphelper/catalogue.py

@@ -1,299 +0,0 @@
-"""Catalogue: classify collected files into game system / faction / class.
-
-The catalogue is the bridge between "a pile of downloads" and the later
-library stage: it reads manifest.jsonl (with full source attribution) and/or
-scans arbitrary directories for archive files, classifies each file against
-the taxonomy in warhammer_data.py, and writes machine- (catalogue.jsonl) and
-human-readable (catalogue.md) reports.
-
-Classification is keyword-based and deliberately conservative: files that
-cannot be classified land in the "unclassified" bucket so gaps become visible
-instead of producing wrong assignments. A manual overrides file can pin
-individual files.
-
-Nothing is moved by default - use --apply for that (it moves files into
-<library-root>/<System>/<Faction>/<Class>/ and records every move in
-catalogue_moves.jsonl so it can be audited/undone).
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import re
-import shutil
-from dataclasses import dataclass, field
-from pathlib import Path
-
-from dlphelper.warhammer_data import (
-    FACTION_ALIASES,
-    GROUP_OF_FACTION,
-    SYSTEMS,
-    UNIT_CLASSES,
-)
-
-log = logging.getLogger(__name__)
-
-ARCHIVE_SUFFIXES = {".zip", ".7z", ".rar", ".tar", ".gz"}
-UNKNOWN = "Unknown"
-UNCLASSIFIED_FACTION = "Unclassified"
-UNCLASSIFIED_CLASS = "Unclassified"
-
-
-@dataclass
-class CatalogueEntry:
-    """One classified file."""
-
-    path: str
-    name: str
-    size: int
-    system: str = UNKNOWN
-    faction: str = UNCLASSIFIED_FACTION
-    klass: str = UNCLASSIFIED_CLASS
-    source: dict = field(default_factory=dict)  # manifest attribution if known
-
-
-def _clean_name(name: str) -> str:
-    stem = Path(name).stem
-    stem = re.sub(r"[_\.\+\[\]\(\)]+", " ", stem)
-    stem = re.sub(r"\s+", " ", stem).strip().lower()
-    return stem
-
-
-def _kw_regex(keyword: str) -> str:
-    """Word-boundary regex for a keyword, tolerating a plural 's'."""
-    return rf"(?<![a-z0-9]){re.escape(keyword)}s?(?![a-z0-9])"
-
-
-def _match_faction(clean: str) -> tuple[str, str] | None:
-    """Return (system, faction) with the most specific keyword hit."""
-    best: tuple[int, str, str] | None = None  # (len, system, faction)
-    for system, aliases in FACTION_ALIASES.items():
-        for keyword, faction in aliases.items():
-            if re.search(_kw_regex(keyword), clean):
-                if best is None or len(keyword) > best[0]:
-                    best = (len(keyword), system, faction)
-    return (best[1], best[2]) if best else None
-
-
-def _match_class(clean: str) -> str | None:
-    """Return the most specific force-organization class keyword hit."""
-    best: tuple[int, str] | None = None
-    for keyword, klass in UNIT_CLASSES.items():
-        if re.search(_kw_regex(keyword), clean):
-            if best is None or len(keyword) > best[0]:
-                best = (len(keyword), klass)
-    return best[1] if best else None
-
-
-def classify(name: str) -> tuple[str, str, str]:
-    """Classify a file name -> (system, faction, class)."""
-    clean = _clean_name(name)
-    faction_hit = _match_faction(clean)
-    klass = _match_class(clean)
-
-    if faction_hit:
-        system, faction = faction_hit
-        # Non-40k systems: keep their class if we found one, else Unclassified
-        if system != "Warhammer 40,000" and klass is None:
-            return system, faction, UNCLASSIFIED_CLASS
-        return system, faction, klass or UNCLASSIFIED_CLASS
-
-    # No faction match: a class keyword alone cannot attribute a system
-    return UNKNOWN, UNCLASSIFIED_FACTION, UNCLASSIFIED_CLASS
-
-
-def scan_manifest(manifest_path: Path) -> list[CatalogueEntry]:
-    """Entries from a manifest.jsonl (preferred: has source attribution)."""
-    entries: list[CatalogueEntry] = []
-    seen: set[str] = set()
-    if not manifest_path.exists():
-        return entries
-    for line in manifest_path.read_text(encoding="utf-8").splitlines():
-        line = line.strip()
-        if not line:
-            continue
-        try:
-            row = json.loads(line)
-        except ValueError:
-            log.warning("Skipping malformed manifest line: %s", line[:80])
-            continue
-        path = row.get("path") or ""
-        if path in seen:
-            continue
-        seen.add(path)
-        entries.append(CatalogueEntry(
-            path=path,
-            name=row.get("name") or Path(path).name,
-            size=int(row.get("size") or 0),
-            source={
-                "channel": row.get("channel"),
-                "url": row.get("url"),
-                "message_id": row.get("message_id"),
-                "timestamp": row.get("timestamp"),
-                "author": row.get("author"),
-            },
-        ))
-    return entries
-
-
-def scan_directory(directory: Path) -> list[CatalogueEntry]:
-    """Entries from walking a directory tree for archive files (manual era)."""
-    entries: list[CatalogueEntry] = []
-    for path in sorted(directory.rglob("*")):
-        if not path.is_file() or path.suffix.lower() not in ARCHIVE_SUFFIXES:
-            continue
-        try:
-            size = path.stat().st_size
-        except OSError:
-            size = 0
-        entries.append(CatalogueEntry(path=str(path), name=path.name, size=size))
-    return entries
-
-
-def load_overrides(path: Path | None) -> dict[str, dict]:
-    """overrides.json: {"/path/or/filename": {"faction": ..., "class": ..., "system": ...}}"""
-    if path is None or not path.exists():
-        return {}
-    return json.loads(path.read_text(encoding="utf-8"))
-
-
-def apply_overrides(entries: list[CatalogueEntry], overrides: dict[str, dict]) -> int:
-    count = 0
-    by_name = {e.name.lower(): e for e in entries}
-    for key, fix in overrides.items():
-        entry = by_name.get(key.lower())
-        if entry is None:
-            log.warning("Override target not found: %s", key)
-            continue
-        if "system" in fix:
-            entry.system = fix["system"]
-        if "faction" in fix:
-            entry.faction = fix["faction"]
-        if "class" in fix:
-            entry.klass = fix["class"]
-        count += 1
-    return count
-
-
-def format_size(n: int) -> str:
-    size = float(n)
-    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
-        if size < 1024 or unit == "TiB":
-            return f"{size:.1f} {unit}"
-        size /= 1024
-    return f"{n} B"
-
-
-def write_reports(
-    entries: list[CatalogueEntry],
-    out_md: Path,
-    out_jsonl: Path,
-) -> None:
-    # machine report
-    with out_jsonl.open("w", encoding="utf-8") as fh:
-        for e in entries:
-            fh.write(json.dumps({
-                "path": e.path,
-                "name": e.name,
-                "size": e.size,
-                "system": e.system,
-                "faction": e.faction,
-                "class": e.klass,
-                "source": {k: v for k, v in e.source.items() if v},
-            }, ensure_ascii=False) + "\n")
-
-    # human report
-    grouped: dict[str, dict[str, dict[str, list[CatalogueEntry]]]] = {}
-    for e in entries:
-        grouped.setdefault(e.system, {}).setdefault(e.faction, {}).setdefault(
-            e.klass, []).append(e)
-
-    lines: list[str] = ["# Warhammer Collection Catalogue", ""]
-    total_size = sum(e.size for e in entries)
-    classified = sum(1 for e in entries if e.faction != UNCLASSIFIED_FACTION)
-    lines.append(f"{len(entries)} files, {format_size(total_size)} total, "
-                 f"{classified} classified, "
-                 f"{len(entries) - classified} unclassified")
-    lines.append("")
-
-    for system in sorted(grouped):
-        factions = grouped[system]
-        all_sys_files = [e for f in factions.values() for c in f.values() for e in c]
-        sys_size = sum(e.size for e in all_sys_files)
-        lines.append(f"## {system} ({len(all_sys_files)} files, {format_size(sys_size)})")
-        for faction in sorted(factions):
-            classes = factions[faction]
-            fac_size = sum(e.size for c in classes.values() for e in c)
-            lines.append(f"### {faction} ({format_size(fac_size)})")
-            for klass in sorted(classes):
-                files = classes[klass]
-                lines.append(f"- **{klass}** ({len(files)} files, {format_size(sum(e.size for e in files))})")
-                for e in sorted(files, key=lambda x: x.name.lower()):
-                    src = ""
-                    if e.source.get("url"):
-                        src = f" — from {e.source['url']}"
-                    lines.append(f"  - {e.name} ({format_size(e.size)}){src}")
-            lines.append("")
-
-    out_md.write_text("\n".join(lines), encoding="utf-8")
-
-
-def apply_moves(
-    entries: list[CatalogueEntry],
-    library_root: Path,
-    moves_log: Path,
-) -> int:
-    """Move classified files into the library structure. Audited in moves_log."""
-    moves_log.parent.mkdir(parents=True, exist_ok=True)
-    moved = 0
-    with moves_log.open("a", encoding="utf-8") as logfh:
-        for e in entries:
-            if e.faction == UNCLASSIFIED_FACTION:
-                continue  # never move what we cannot classify
-            if e.system == UNKNOWN:
-                continue
-            src = Path(e.path)
-            if not src.exists():
-                log.warning("Cannot move, missing: %s", src)
-                continue
-            dest_dir = (
-                library_root / e.system
-                / GROUP_OF_FACTION.get(e.faction, "Unclassified")
-                / e.faction / e.klass
-            )
-            dest = dest_dir / src.name
-            if dest.exists():
-                log.warning("Skip, destination exists: %s", dest)
-                continue
-            dest_dir.mkdir(parents=True, exist_ok=True)
-            shutil.move(str(src), str(dest))
-            logfh.write(json.dumps({"from": str(src), "to": str(dest)}) + "\n")
-            e.path = str(dest)
-            moved += 1
-    return moved
-
-
-def build_catalogue(
-    manifests: list[Path],
-    scan_dirs: list[Path],
-    overrides_path: Path | None,
-) -> list[CatalogueEntry]:
-    entries: list[CatalogueEntry] = []
-    seen: set[str] = set()
-    for m in manifests:
-        for e in scan_manifest(m):
-            if e.path not in seen:
-                seen.add(e.path)
-                entries.append(e)
-    for d in scan_dirs:
-        for e in scan_directory(d):
-            if e.path not in seen:
-                seen.add(e.path)
-                entries.append(e)
-
-    for e in entries:
-        e.system, e.faction, e.klass = classify(e.name)
-
-    apply_overrides(entries, load_overrides(overrides_path))
-    return entries

dlphelper/discord.py

@@ -1,160 +0,0 @@
-"""Export a Discord channel/thread with DiscordChatExporter and extract URLs.
-
-The channels this tool targets encode their download links as base64 inside
-message content, and occasionally as plain links. We export the channel as
-JSON (richer metadata than the HTML export: message id, timestamp, author),
-decode every base64 candidate, and keep only URLs for supported hosts.
-"""
-
-from __future__ import annotations
-
-import base64
-import json
-import logging
-import os
-import re
-import subprocess
-from dataclasses import dataclass
-from pathlib import Path
-from urllib.parse import urlparse
-
-log = logging.getLogger(__name__)
-
-DCE_IMAGE = "tyrrrz/discordchatexporter:stable"
-
-# The channels encode links as base64 that always starts with this prefix
-# (https://...); grab.sh established this regex, keep it compatible.
-B64_RE = re.compile(r"aHR0cHM6Ly[89][A-Za-z0-9+/_-]*")
-URL_RE = re.compile(r"https://[^\s<>\"'\)\]\}]+")
-
-SUPPORTED_HOSTS = {
-    "gofile": ("gofile.io", "www.gofile.io"),
-    "0807": ("0807.st", "www.0807.st"),
-}
-
-
-@dataclass(frozen=True)
-class FoundUrl:
-    """One download URL found in the channel, with source attribution."""
-
-    url: str
-    host: str  # "gofile" | "0807" (set after classification)
-    message_id: str | None = None
-    timestamp: str | None = None
-    author: str | None = None
-    encoded: bool = False  # found via base64 decode vs. plain link
-
-
-def export_channel(channel_id: str, out_dir: Path, token: str, force: bool = False) -> Path:
-    """Export the channel to out_dir/export.json using the DCE docker image.
-
-    The token is passed to the container via the DISCORD_TOKEN env var (DCE
-    reads it natively) so it never appears on the docker command line. TOKEN
-    is set as well for DCE versions that only know that name.
-    """
-    dest = out_dir / "export.json"
-    if dest.exists() and not force:
-        log.info("Reusing existing export %s (pass --fresh to re-export)", dest)
-        return dest
-
-    out_dir.mkdir(parents=True, exist_ok=True)
-    env = dict(os.environ)
-    env["DISCORD_TOKEN"] = token
-    env["TOKEN"] = token
-    cmd = [
-        "docker", "run", "--rm",
-        "--user", f"{os.getuid()}:{os.getgid()}",
-        "-v", f"{out_dir.resolve()}:/out",
-        "-e", "DISCORD_TOKEN",
-        "-e", "TOKEN",
-        DCE_IMAGE,
-        "export", "-c", channel_id, "-f", "Json", "-o", "/out/export.json",
-    ]
-    log.info("Exporting channel %s via DiscordChatExporter ...", channel_id)
-    try:
-        subprocess.run(cmd, env=env, check=True)
-    except FileNotFoundError as exc:
-        raise RuntimeError(
-            "docker not found - the Discord export needs the docker image "
-            f"{DCE_IMAGE}"
-        ) from exc
-    except subprocess.CalledProcessError as exc:
-        raise RuntimeError(
-            f"DiscordChatExporter failed with exit code {exc.returncode} "
-            "(check the token and the channel id)"
-        ) from exc
-    if not dest.exists():
-        raise RuntimeError("DiscordChatExporter produced no output file")
-    return dest
-
-
-def classify(url: str) -> str | None:
-    """Return the host key ('gofile' | '0807') for a URL, else None."""
-    try:
-        hostname = (urlparse(url).hostname or "").lower()
-    except ValueError:
-        return None
-    for key, hosts in SUPPORTED_HOSTS.items():
-        if hostname in hosts:
-            return key
-    return None
-
-
-def _decode_b64(candidate: str) -> str | None:
-    s = candidate.replace("_", "/").replace("-", "+")
-    s = s.rstrip("=")
-    s += "=" * ((4 - len(s) % 4) % 4)
-    try:
-        decoded = base64.b64decode(s, validate=False).decode("utf-8", "strict")
-    except Exception:  # noqa: BLE001 - malformed candidate, just skip it
-        return None
-    # One encoded blob can contain the URL plus padding/newlines
-    stripped = decoded.strip()
-    line = stripped.splitlines()[0] if stripped else ""
-    return line or None
-
-
-def extract_urls(export_path: Path) -> list[FoundUrl]:
-    """Extract supported URLs from a DCE JSON export, deduplicated."""
-    data = json.loads(export_path.read_text(encoding="utf-8"))
-    messages = data if isinstance(data, list) else data.get("messages", [])
-
-    found: dict[str, FoundUrl] = {}
-    stats = {"b64": 0, "plain": 0, "unsupported": 0}
-
-    for msg in messages:
-        content = str(msg.get("content") or "")
-        if not content:
-            continue
-        author = (msg.get("author") or {}).get("name")
-        attrib = {
-            "message_id": str(msg.get("id") or "") or None,
-            "timestamp": msg.get("timestamp"),
-            "author": author,
-        }
-
-        for match in B64_RE.finditer(content):
-            stats["b64"] += 1
-            decoded = _decode_b64(match.group(0))
-            if not decoded:
-                continue
-            host = classify(decoded)
-            if host is None:
-                stats["unsupported"] += 1
-                continue
-            found.setdefault(decoded, FoundUrl(url=decoded, host=host, encoded=True, **attrib))
-
-        for match in URL_RE.finditer(content):
-            url = match.group(0).rstrip(".,;)")
-            host = classify(url)
-            if host is None:
-                continue
-            stats["plain"] += 1
-            found.setdefault(url, FoundUrl(url=url, host=host, encoded=False, **attrib))
-
-    log.info(
-        "Extraction: %d base64 candidates, %d plain links, %d unsupported/undecodable, "
-        "%d unique supported URLs",
-        stats["b64"], stats["plain"], stats["unsupported"], len(found),
-    )
-    return sorted(found.values(), key=lambda f: f.url)

dlphelper/globalstore.py

@@ -1,70 +0,0 @@
-"""Cross-run, cross-channel record of completed URLs.
-
-This is what prevents re-downloading your collection: state.json only knows
-about one OUT_DIR, but the same gofile link posted in a second channel would
-otherwise be downloaded again into that channel's OUT_DIR. The global store
-(default: ~/.local/share/dlphelper/urls.json) records every successfully
-downloaded URL once, forever, regardless of OUT_DIR.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import os
-import time
-from pathlib import Path
-
-log = logging.getLogger(__name__)
-
-
-def default_global_path() -> Path:
-    data_home = os.environ.get("XDG_DATA_HOME") or str(
-        Path.home() / ".local" / "share"
-    )
-    return Path(data_home) / "dlphelper" / "urls.json"
-
-
-class GlobalStore:
-    """JSON file mapping url -> {status, channel, host, files, updated}."""
-
-    def __init__(self, path: Path | None = None):
-        self.path = path
-        self.records: dict[str, dict] = {}
-        if self.path and self.path.exists():
-            try:
-                self.records = json.loads(self.path.read_text(encoding="utf-8"))
-                if not isinstance(self.records, dict):
-                    raise ValueError("global store root must be an object")
-            except (ValueError, OSError) as exc:
-                backup = self.path.with_suffix(".json.bak")
-                log.warning("Corrupt global store %s (%s); backed up to %s",
-                            self.path, exc, backup)
-                self.path.replace(backup)
-                self.records = {}
-
-    def is_done(self, url: str) -> bool:
-        if not self.records:
-            return False
-        rec = self.records.get(url)
-        return bool(rec) and rec.get("status") == "ok"
-
-    def mark(self, url: str, **fields) -> None:
-        if self.path is None:
-            return
-        rec = self.records.setdefault(url, {"status": "ok"})
-        for key, value in fields.items():
-            if value is not None:
-                rec[key] = value
-        rec["status"] = "ok"
-        rec["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
-        self.save()
-
-    def save(self) -> None:
-        if self.path is None:
-            return
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        tmp = self.path.with_suffix(".json.tmp")
-        tmp.write_text(json.dumps(self.records, indent=2, sort_keys=True),
-                       encoding="utf-8")
-        tmp.replace(self.path)

dlphelper/gofile.py

@@ -1,239 +0,0 @@
-"""Minimal GoFile API client - one guest account per run.
-
-The API interaction mirrors Lysagxra/GoFileDownloader (same headers, website
-token scheme and endpoints), but with two deliberate differences:
-
-1. A single guest account is created per run (GoFileDownloader creates one per
-   URL, which trips Gofile's rate limit after ~6 requests).
-2. Failures raise exceptions instead of sys.exit(), so the orchestrator can
-   track them per file/URL and keep going.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import logging
-import os
-import time
-from pathlib import Path
-from urllib.parse import urlparse
-
-import requests
-
-log = logging.getLogger(__name__)
-
-API = "https://api.gofile.io"
-USER_AGENT = "Mozilla/5.0"
-LOCALE = "en-US"
-TOKEN_SECRET = "12af056dacea0b"  # public website-token seed, see GoFileDownloader  # noqa: S105
-TOKEN_WINDOW_SEC = 14400
-CHUNK_SIZE = 1024 * 1024
-
-COMMON_HEADERS = {
-    "User-Agent": USER_AGENT,
-    "Accept-Encoding": "gzip",
-    "Accept": "*/*",
-    "Connection": "keep-alive",
-}
-EXTENDED_HEADERS = {
-    **COMMON_HEADERS,
-    "Sec-Fetch-Dest": "empty",
-    "Sec-Fetch-Mode": "cors",
-    "Sec-Fetch-Site": "same-site",
-    "Pragma": "no-cache",
-    "Cache-Control": "no-cache",
-}
-
-
-class GoFileError(Exception):
-    """Base error for GoFile operations."""
-
-
-class GoFilePasswordRequired(GoFileError):
-    """The content is password protected and none (or a wrong one) was given."""
-
-
-class GoFileNotFound(GoFileError):
-    """The content/file no longer exists (404 / status notFound).
-
-    Dead links never come back, so the orchestrator must not retry these.
-    """
-
-
-class GoFileEmpty(GoFileError):
-    """The content exists but contains no files (e.g. emptied folder).
-
-    Stable state like a dead link: no retry.
-    """
-
-
-class GoFileAuthError(GoFileError):
-    """The account token was rejected; the client can re-auth once."""
-
-
-def content_id_from_url(url: str) -> str:
-    """Extract the content id from a gofile.io/d/<id> URL."""
-    parts = [p for p in urlparse(url).path.split("/") if p]
-    if len(parts) >= 2 and parts[-2] == "d":
-        return parts[-1]
-    raise GoFileError(f"not a gofile content URL (expected gofile.io/d/<id>): {url}")
-
-
-class GoFileClient:
-    """GoFile API client that reuses one guest account token for the whole run."""
-
-    def __init__(self, password: str | None = None):
-        self._password_hash = (
-            hashlib.sha256(password.encode()).hexdigest() if password else None
-        )
-        self._token: str | None = None
-        self.session = requests.Session()
-
-    # -- authentication -----------------------------------------------------
-
-    def ensure_token(self, max_attempts: int = 8, wait_seconds: int = 20) -> str:
-        """Create a guest account; backs off and retries on rate limiting."""
-        if self._token:
-            return self._token
-
-        last_status = "unknown"
-        for attempt in range(1, max_attempts + 1):
-            try:
-                resp = self.session.post(
-                    f"{API}/accounts", headers=COMMON_HEADERS, timeout=15
-                )
-                data = resp.json()
-            except (ValueError, requests.RequestException) as exc:
-                last_status = f"request error: {exc}"
-                log.warning(
-                    "GoFile account creation attempt %d/%d failed: %s",
-                    attempt, max_attempts, last_status,
-                )
-            else:
-                if isinstance(data, dict) and data.get("status") == "ok":
-                    self._token = data["data"]["token"]
-                    log.info("GoFile guest account created (token reused for whole run)")
-                    return self._token
-                last_status = str(data.get("status", data)[:200] if data else data)
-                log.warning(
-                    "GoFile account creation attempt %d/%d failed: %s",
-                    attempt, max_attempts, last_status,
-                )
-            if attempt < max_attempts:
-                time.sleep(wait_seconds * attempt)
-        raise GoFileError(
-            f"GoFile account creation failed after {max_attempts} attempts "
-            f"(last: {last_status})"
-        )
-
-    def reset(self) -> None:
-        """Drop the current token so the next ensure_token() creates a new one."""
-        self._token = None
-
-    def _website_token(self) -> str:
-        window = int(time.time()) // TOKEN_WINDOW_SEC
-        seed = f"{USER_AGENT}::{LOCALE}::{self._token}::{window}::{TOKEN_SECRET}"
-        return hashlib.sha256(seed.encode()).hexdigest()
-
-    def _auth_headers(self) -> dict:
-        return {
-            **EXTENDED_HEADERS,
-            "Authorization": f"Bearer {self._token}",
-            "X-Website-Token": self._website_token(),
-            "X-BL": LOCALE,
-        }
-
-    # -- listing ------------------------------------------------------------
-
-    def list_files(self, url: str) -> list[tuple[str, str, str]]:
-        """Return (relpath, filename, download_link) for every file under a URL."""
-        cid = content_id_from_url(url)
-        results: list[tuple[str, str, str]] = []
-        self._walk(cid, [], results)
-        # Direct files at the content root get their own folder named after the id
-        normalized = [
-            (relpath if relpath else cid, name, link) for relpath, name, link in results
-        ]
-        return normalized
-
-    def _walk(self, cid: str, prefix: list[str], results: list[tuple[str, str, str]]) -> None:
-        query = "?cache=true&sortField=createTime&sortDirection=1"
-        if self._password_hash:
-            query += f"&password={self._password_hash}"
-        try:
-            resp = self.session.get(
-                f"{API}/contents/{cid}{query}", headers=self._auth_headers(), timeout=30
-            )
-        except requests.RequestException as exc:
-            raise GoFileError(f"content request for {cid} failed: {exc}") from exc
-
-        if resp.status_code == 404:
-            raise GoFileNotFound(f"content {cid} not found (dead link)")
-        if resp.status_code in (401, 403):
-            raise GoFileAuthError(f"content request denied ({resp.status_code}) for {cid}")
-        try:
-            data = resp.json()
-        except ValueError as exc:
-            raise GoFileError(f"non-JSON response for {cid}: {resp.text[:200]}") from exc
-
-        if data.get("status") == "notFound" or "notFound" in str(data.get("status", "")):
-            raise GoFileNotFound(f"content {cid} does not exist anymore (dead link)")
-        if data.get("status") != "ok":
-            raise GoFileError(f"content request failed for {cid}: {data.get('status')}")
-
-        d = data["data"]
-        if d.get("passwordStatus") not in (None, "passwordOk"):
-            raise GoFilePasswordRequired(
-                f"content {cid} is password protected (use --password)"
-            )
-
-        if d.get("type") == "folder":
-            here = prefix + [d["name"]]
-            for child in (d.get("children") or {}).values():
-                if child.get("type") == "folder":
-                    self._walk(child["id"], here, results)
-                else:
-                    results.append(("/".join(here), child["name"], child["link"]))
-        else:
-            results.append(("/".join(prefix), d["name"], d["link"]))
-
-    # -- downloading ----------------------------------------------------------
-
-    def download_file(self, link: str, dest: Path) -> int:
-        """Download one file; returns the file size. Skips existing files."""
-        dest.parent.mkdir(parents=True, exist_ok=True)
-        if dest.exists() and dest.stat().st_size > 0:
-            log.info("Skip existing: %s", dest)
-            return dest.stat().st_size
-
-        tmp = dest.with_name(dest.name + ".part")
-        headers = {
-            **EXTENDED_HEADERS,
-            "Cookie": f"accountToken={self._token}",
-            "Referer": link,
-            "Origin": link,
-        }
-        try:
-            with self.session.get(
-                link, headers=headers, stream=True, timeout=(10, 60), allow_redirects=True
-            ) as resp:
-                if resp.status_code == 404:
-                    raise GoFileNotFound(f"file {dest.name} not found (dead link)")
-                if resp.status_code != 200:
-                    raise GoFileError(f"HTTP {resp.status_code} while downloading {dest.name}")
-                total = int(resp.headers.get("Content-Length") or 0)
-                written = 0
-                with tmp.open("wb") as fh:
-                    for chunk in resp.iter_content(chunk_size=CHUNK_SIZE):
-                        fh.write(chunk)
-                        written += len(chunk)
-            if written == 0:
-                raise GoFileError(f"empty body for {dest.name}")
-            os.replace(tmp, dest)
-            return written
-        finally:
-            if tmp.exists() and not dest.exists():
-                try:
-                    tmp.unlink()
-                except OSError:
-                    pass

dlphelper/gwshop.py

@@ -1,353 +0,0 @@
-"""Games Workshop webshop catalog via Algolia.
-
-The warhammer.com shop is behind an AWS WAF challenge for non-browser
-clients, but its product search runs on Algolia and the app-id plus the
-public search-only key ship with every browser page. This module uses them
-to enumerate the whole product index and normalize it into the sorter's
-catalog schema (system/group/faction/subfaction/class/name/aliases).
-
-Credentials are read from (first hit wins):
-  1. env ALGOLIA_APP_ID + ALGOLIA_API_KEY
-  2. algolia.json next to the repo root (gitignored):
-       {"app_id": "...", "api_key": "..."}
-
-The key is the *search-only* public key (never an admin key). Treat any
-key found in algolia.json as still secret: never log or commit it.
-
-CLI:  python3 -m dlphelper.gwshop OUT.json [--index NAME] [--raw RAW.json]
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import logging
-import os
-import time
-from pathlib import Path
-from typing import Any
-
-import requests
-
-log = logging.getLogger(__name__)
-
-DEFAULT_INDEX = "prod-lazarus-product-en-ww"
-# The read host (-dsn) serves searches; credentials also work as query
-# parameters, which keeps every request a plain GET/POST with no headers
-# beyond content-type.
-_DSN_HOST = "https://{app_id}-dsn.algolia.net"
-
-_PAGE_SIZE = 1000  # Algolia max hitsPerPage per /query page
-_RETRY_DELAYS = (2, 5, 15)
-
-
-def load_credentials() -> tuple[str, str]:
-    """Resolve (app_id, api_key) from env or algolia.json."""
-    app_id = os.environ.get("ALGOLIA_APP_ID", "")
-    api_key = os.environ.get("ALGOLIA_API_KEY", "")
-    if app_id and api_key:
-        return app_id, api_key
-    cfg = Path(__file__).resolve().parent.parent / "algolia.json"
-    if cfg.is_file():
-        data = json.loads(cfg.read_text(encoding="utf-8"))
-        app_id = data.get("app_id", "") or app_id
-        api_key = data.get("api_key", "") or api_key
-    if not app_id or not api_key:
-        raise SystemExit(
-            "Algolia credentials missing: set ALGOLIA_APP_ID and "
-            "ALGOLIA_API_KEY, or create algolia.json with "
-            '{"app_id": "...", "api_key": "..."} (gitignored).'
-        )
-    return app_id, api_key
-
-
-class GwShopClient:
-    """Minimal Algolia search client for one application."""
-
-    def __init__(self, app_id: str, api_key: str) -> None:
-        self.base = _DSN_HOST.format(app_id=app_id)
-        self.params = {
-            "x-algolia-application-id": app_id,
-            "x-algolia-api-key": api_key,
-        }
-        self.session = requests.Session()
-
-    def _request(self, method: str, path: str, body: dict | None = None) -> Any:
-        url = f"{self.base}{path}"
-        last_err: Exception | None = None
-        for attempt, delay in enumerate((0,) + _RETRY_DELAYS, start=1):
-            if delay:
-                log.info("Algolia retry %d in %ds", attempt, delay)
-                time.sleep(delay)
-            try:
-                r = self.session.request(
-                    method,
-                    url,
-                    params=self.params,
-                    json=body,
-                    timeout=30,
-                )
-                if r.status_code == 200:
-                    return r.json()
-                last_err = RuntimeError(
-                    f"Algolia {method} {path} -> {r.status_code}: {r.text[:200]}"
-                )
-                if r.status_code in (401, 403):  # credentials: no point retrying
-                    break
-            except requests.RequestException as exc:
-                last_err = exc
-        raise RuntimeError(f"Algolia request failed: {last_err}") from last_err
-
-    def list_indices(self) -> list[dict]:
-        return self._request("GET", "/1/indexes").get("items", [])
-
-    # ---- full index retrieval -------------------------------------------
-    # The search-only key cannot use /browse (403) and normal pagination is
-    # capped at 1000 cumulative results. So we partition the index on the
-    # GameSystemsRoot hierarchical facet: each branch is fetched with an
-    # explicit filter, recursing deeper only when a branch exceeds one page.
-
-    _FACET = "GameSystemsRoot"
-
-    @staticmethod
-    def _escape(value: str) -> str:
-        return value.replace("\\", "\\\\").replace('"', '\\"')
-
-    def _filter_expr(self, path: list[tuple[int, str]]) -> str:
-        return " AND ".join(
-            f'{self._FACET}.lvl{lvl}:"{self._escape(value)}"'
-            for lvl, value in path
-        )
-
-    def _search_all(self, index: str, filters: str | None) -> tuple[list[dict], int]:
-        """All hits matching `filters` (must be <= pagination cap)."""
-        out: list[dict] = []
-        params = f"query=&hitsPerPage={_PAGE_SIZE}&page=0"
-        if filters:
-            params += f"&filters={requests.utils.quote(filters, safe='')}"
-        page = self._request(
-            "POST", f"/1/indexes/{index}/query", body={"params": params}
-        )
-        out.extend(page.get("hits", []))
-        return out, int(page.get("nbHits") or 0)
-
-    def _facet_counts(
-        self, index: str, level: int, filters: str | None
-    ) -> dict[str, int]:
-        params = f"query=&hitsPerPage=0&facets={self._FACET}.lvl{level}"
-        if filters:
-            params += f"&filters={requests.utils.quote(filters, safe='')}"
-        page = self._request(
-            "POST", f"/1/indexes/{index}/query", body={"params": params}
-        )
-        facets = page.get("facets") or {}
-        return facets.get(f"{self._FACET}.lvl{level}") or {}
-
-    def browse(self, index: str, hits_per_page: int = _PAGE_SIZE) -> list[dict]:
-        """Every record of the index, via facet-partitioned search."""
-        hits, total = self._search_all(index, None)
-        if total <= hits_per_page:
-            return hits
-        log.info("gwshop %s: %d hits, partitioning on %s", index, total, self._FACET)
-        out = self._partition(index, [], total)
-        covered = len({h.get("objectID") for h in out})
-        if covered < total:
-            log.warning(
-                "gwshop %s: %d of %d index records left out of the dump "
-                "(no fetchable %s branch path)",
-                index, total - covered, total, self._FACET,
-            )
-        return out
-
-    def _partition(
-        self, index: str, path: list[tuple[int, str]], total: int
-    ) -> list[dict]:
-        level = len(path)
-        filters = self._filter_expr(path) if path else None
-        counts = self._facet_counts(index, level, filters)
-        out: list[dict] = []
-        covered: set[Any] = set()
-        for value, count in sorted(counts.items()):
-            sub_path = path + [(level, value)]
-            sub_filters = self._filter_expr(sub_path)
-            if count <= _PAGE_SIZE:
-                hits, nb = self._search_all(index, sub_filters)
-                out.extend(hits)
-                covered.update(h.get("objectID") for h in hits)
-                log.info(
-                    "gwshop %s: %s -> %d hits", index, sub_filters[:80], nb
-                )
-            else:
-                sub = self._partition(index, sub_path, count)
-                out.extend(sub)
-                covered.update(h.get("objectID") for h in sub)
-        # nbHits summed across branches double-counts products listed in
-        # several branches, so coverage is counted per unique objectID.
-        residual = total - len(covered)
-        if residual > 0:
-            log.warning(
-                "gwshop %s: %d hits in branch carry no %s.lvl%d "
-                "(shallow facet path, left out of the dump)",
-                index, residual, self._FACET, level,
-            )
-        return out
-
-
-# ---------------------------------------------------------------------------
-# normalization
-# ---------------------------------------------------------------------------
-
-def _clean(node: str) -> str:
-    return node.strip()
-
-
-def hierarchy_paths(hit: dict) -> list[list[str]]:
-    """Every full path of the GameSystemsRoot hierarchical facet.
-
-    A hit may belong to several branches (e.g. an army box also being a
-    "Gaming Rules" entry); each branch is returned as its own path
-    [system, group, faction, subfaction_or_class, ...].
-    """
-    root = hit.get("GameSystemsRoot") or {}
-    paths: list[list[str]] = []
-    for lvl_key in sorted(root, key=lambda k: int(k[3:])):
-        for value in root.get(lvl_key) or []:
-            paths.append([_clean(p) for p in value.split(" > ")])
-    return paths
-
-
-def normalize(hit: dict) -> dict:
-    """One Algolia hit -> one or more catalog entries (usually one)."""
-    name = _clean(hit.get("name") or hit.get("objectID") or "")
-    entry: dict[str, Any] = {
-        "name": name,
-        "aliases": [],
-        "sku": hit.get("sku"),
-        "slug": hit.get("slug"),
-        "productType": hit.get("productType"),
-        "source": "gwshop",
-        "gw_paths": hierarchy_paths(hit),
-        "collections": hit.get("_collections") or [],
-    }
-    return entry
-
-
-def build_catalog(hits: list[dict]) -> dict:
-    """Index dump -> catalog document in the sorter's schema shape.
-
-    The hierarchy layer (system/group/faction/subfaction) is expanded from
-    gw_paths; name/aliases stay per product so the fuzzy matcher works on
-    the product name, exactly like data/library_catalog.json entries.
-    """
-    entries: list[dict] = []
-    seen: set[tuple] = set()
-    for hit in hits:
-        base = normalize(hit)
-        paths = base["gw_paths"]
-        if not paths:
-            key = (hit.get("objectID"), ())
-            if key in seen:
-                continue
-            seen.add(key)
-            entries.append(
-                {
-                    "system": None,
-                    "group": None,
-                    "faction": None,
-                    "subfaction": None,
-                    "class": None,
-                    "name": base["name"],
-                    "aliases": [],
-                    "sku": base["sku"],
-                    "slug": base["slug"],
-                    "productType": base["productType"],
-                    "source": "gwshop",
-                    "collections": base["collections"],
-                }
-            )
-            continue
-        # primary branch = longest path (most specific), e.g.
-        # [40k, Space Marines, Raven Guard, Walker]
-        primary = max(paths, key=len)
-        system = primary[0] if len(primary) > 0 else None
-        group = primary[1] if len(primary) > 1 else None
-        faction = primary[2] if len(primary) > 2 else None
-        # The LAST level is the class/category layer (GW uses shop
-        # categories there: Battleline, Walker, Vehicle, "Rules &
-        # Accessories", product lines like "Cadian"). Deeper sources
-        # (miniset style) may carry a real subfaction before it.
-        if len(primary) >= 5:
-            subfaction = primary[3]
-            klass = primary[4]
-        elif len(primary) == 4:
-            subfaction = None
-            klass = primary[3]
-        else:
-            subfaction = None
-            klass = None
-        key = (hit.get("objectID"), tuple(primary))
-        if key in seen:
-            continue
-        seen.add(key)
-        entries.append(
-            {
-                "system": system,
-                "group": group,
-                "faction": faction,
-                "subfaction": subfaction,
-                "class": klass,
-                "name": base["name"],
-                "aliases": [],
-                "sku": base["sku"],
-                "slug": base["slug"],
-                "productType": base["productType"],
-                "source": "gwshop",
-                "collections": base["collections"],
-            }
-        )
-    return {
-        "comment": (
-            "Generated by dlphelper.gwshop from the Games Workshop webshop "
-            "(Algolia index, deduplicated by product). Source-agnostic "
-            "catalog schema: system/group/faction/subfaction/class/name/aliases."
-        ),
-        "source": "gwshop",
-        "generated_entries": len(entries),
-        "entries": entries,
-    }
-
-
-def main() -> None:
-    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-    parser = argparse.ArgumentParser(
-        description="Dump the GW webshop catalog via Algolia."
-    )
-    parser.add_argument("out", help="output catalog JSON path")
-    parser.add_argument("--index", default=DEFAULT_INDEX)
-    parser.add_argument("--raw", help="also write the raw Algolia hits here")
-    args = parser.parse_args()
-
-    app_id, api_key = load_credentials()
-    client = GwShopClient(app_id, api_key)
-    hits = client.browse(args.index)
-    log.info("%d raw hits from %s", len(hits), args.index)
-
-    if args.raw:
-        Path(args.raw).write_text(
-            json.dumps(hits, ensure_ascii=False, indent=1), encoding="utf-8"
-        )
-        log.info("raw hits written to %s", args.raw)
-
-    catalog = build_catalog(hits)
-    Path(args.out).write_text(
-        json.dumps(catalog, ensure_ascii=False, indent=1), encoding="utf-8"
-    )
-    log.info(
-        "%d catalog entries written to %s",
-        catalog["generated_entries"],
-        args.out,
-    )
-
-
-if __name__ == "__main__":
-    main()

dlphelper/sorter.py

@@ -1,383 +0,0 @@
-"""Sorter: place collection files into the library via plan/apply.
-
-See docs/SORTING.md for the structure and the pipeline this implements:
-inspect (name + archive listing) -> classify (keyword tables) -> fuzzy match
-against the library catalog -> plan -> apply (hardlink, fallback copy).
-
-The sorter never deletes or modifies source files. Applying is append-only:
-targets that already exist are skipped, actions are audited in
-sorter_applied.jsonl.
-"""
-
-from __future__ import annotations
-
-import difflib
-import json
-import logging
-import os
-import shutil
-import subprocess
-import zipfile
-from dataclasses import dataclass, field
-from pathlib import Path
-
-from dlphelper.catalogue import _clean_name, _match_class, _match_faction
-from dlphelper.warhammer_data import GROUP_OF_FACTION
-
-log = logging.getLogger(__name__)
-
-ARCHIVE_SUFFIXES = {".zip", ".7z", ".rar", ".tar", ".gz"}
-UNSORTED = "_unsorted"
-UNCLASSIFIED_CLASS = "Unclassified"
-
-# Run-artifact files that must never be sorted into a library
-RUN_ARTIFACTS = {
-    "export.json", "export.html", "state.json", "manifest.jsonl",
-    "failed_urls.txt", "catalogue.md", "catalogue.jsonl",
-    "sorter_applied.jsonl", "catalogue_moves.jsonl", "urls.txt",
-    "urls_0807.txt", "urls_gofile.txt", "failed_0807.txt",
-}
-
-
-@dataclass
-class SortItem:
-    """One file to sort, plus everything learned about it."""
-
-    path: Path
-    size: int
-    name: str            # original file name
-    clean: str           # cleaned lower-case name
-    inner_names: list[str] = field(default_factory=list)  # archive listing
-    system: str = "Unknown"
-    group: str = UNSORTED
-    faction: str = UNSORTED
-    subfaction: str | None = None
-    klass: str | None = None
-    unit: str | None = None          # canonical catalog name, if matched
-    match_score: float = 0.0
-    match_kind: str = "none"         # catalog | implicit | unclassified
-    action: str = "link"             # link | copy
-    target: Path | None = None
-
-
-# ---------------------------------------------------------------------------
-# inspection
-# ---------------------------------------------------------------------------
-
-def list_zip(path: Path, limit: int = 40) -> list[str]:
-    """Inner names of a .zip without extracting."""
-    try:
-        with zipfile.ZipFile(path) as zf:
-            return zf.namelist()[:limit]
-    except (zipfile.BadZipFile, OSError, RuntimeError) as exc:
-        log.debug("zip listing failed for %s: %s", path.name, exc)
-        return []
-
-
-def list_7z(path: Path, limit: int = 40) -> list[str]:
-    """Inner names of a .7z via the installed 7z binary, without extracting."""
-    if not shutil.which("7z"):
-        return []
-    try:
-        proc = subprocess.run(
-            ["7z", "l", "-ba", "-slt", str(path)],
-            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, timeout=120,
-        )
-    except (OSError, subprocess.TimeoutExpired) as exc:
-        log.debug("7z listing failed for %s: %s", path.name, exc)
-        return []
-    names: list[str] = []
-    for block in proc.stdout.split("Path = ")[1:]:
-        name = block.splitlines()[0].strip() if block.splitlines() else ""
-        # -slt lists directories too; skip entries ending in /
-        if name and not name.endswith("/"):
-            names.append(name)
-        if len(names) >= limit:
-            break
-    return names
-
-
-def inspect(path: Path) -> SortItem:
-    size = path.stat().st_size if path.exists() else 0
-    item = SortItem(path=path, size=size, name=path.name, clean=_clean_name(path.name))
-    if not path.is_file():
-        return item
-    if path.suffix.lower() in ARCHIVE_SUFFIXES:
-        # An archive that is not an archive is almost always a saved HTML
-        # error/login page that got downloaded instead of the file.
-        if path.suffix.lower() == ".zip":
-            item.inner_names = list_zip(path)
-        else:
-            item.inner_names = list_7z(path)
-        if item.inner_names == [] and path.stat().st_size > 0:
-            head = b""
-            with path.open("rb") as fh:
-                head = fh.read(512)
-            if b"<!doctype html" in head.lower() or b"<html" in head.lower():
-                log.warning("%s is an HTML page, not an archive (bad download)", path.name)
-                item.match_kind = "invalid"
-    return item
-
-
-# ---------------------------------------------------------------------------
-# classification & matching
-# ---------------------------------------------------------------------------
-
-def classify_item(item: SortItem) -> None:
-    """Keyword classification from the file name, then refined by inner names."""
-    hit = _match_faction(item.clean)
-    if hit:
-        item.system, item.faction = hit
-    item.klass = _match_class(item.clean) or (
-        _match_class(_clean_name(" ".join(item.inner_names))) if item.inner_names else None
-    )
-
-    if item.faction == UNSORTED and item.inner_names:
-        # hash-named archives: vote on the inner names' classification
-        votes: dict[tuple[str, str], int] = {}
-        for inner in item.inner_names:
-            ihit = _match_faction(_clean_name(inner))
-            if ihit:
-                votes[ihit] = votes.get(ihit, 0) + 1
-        if votes:
-            best = max(votes.items(), key=lambda kv: kv[1])[0]
-            item.system, item.faction = best
-            if item.klass is None:
-                inner_classes = [
-                    _match_class(_clean_name(inner)) for inner in item.inner_names
-                ]
-                cvotes: dict[str, int] = {}
-                for c in inner_classes:
-                    if c:
-                        cvotes[c] = cvotes.get(c, 0) + 1
-                if cvotes:
-                    item.klass = max(cvotes.items(), key=lambda kv: kv[1])[0]
-
-    item.group = GROUP_OF_FACTION.get(item.faction, UNSORTED)
-
-
-# faction group layer (GW naming, shared with warhammer_data)
-
-
-def load_catalog(path: Path) -> list[dict]:
-    data = json.loads(path.read_text(encoding="utf-8"))
-    entries = data["entries"] if isinstance(data, dict) else data
-    for e in entries:
-        e["_clean"] = _clean_name(e["name"])
-        e["_clean_aliases"] = [_clean_name(a) for a in e.get("aliases", [])]
-    return entries
-
-
-def _best_ratio(clean: str, candidates: list[str]) -> float:
-    return max(
-        (difflib.SequenceMatcher(None, clean, c).ratio() for c in candidates),
-        default=0.0,
-    )
-
-
-def _best_in_pool(
-    item: SortItem, pool: list[dict], use_inner: bool = True
-) -> tuple[float, dict] | None:
-    """Best fuzzy score of `item` against all entries in `pool`.
-
-    Inner archive names are only consulted on request; a confident outer
-    (file) name must win over votes from random inner STL filenames.
-    """
-    best: tuple[float, dict] | None = None
-    for entry in pool:
-        cands = [entry["_clean"], *entry["_clean_aliases"]]
-        score = _best_ratio(item.clean, cands)
-        if use_inner and item.inner_names:
-            inner_clean = [_clean_name(n) for n in item.inner_names[:10]]
-            for c in inner_clean:
-                score = max(score, _best_ratio(c, cands) * 0.95)  # inner slightly discounted
-        if best is None or score > best[0]:
-            best = (score, entry)
-    return best
-
-
-def match_catalog(item: SortItem, catalog: list[dict], threshold: float) -> None:
-    """Fuzzy-match against catalog entries in the same faction (or all).
-
-    The matched entry is hierarchy-authoritative: group, subfaction and
-    class come from the catalog (GW shop categories), and when the keyword
-    classification had no faction the entry's system/faction is adopted too.
-    """
-    if item.faction == UNSORTED and not item.inner_names and not item.clean:
-        return
-    if item.match_kind == "invalid":
-        return  # HTML-page masquerading as an archive; never fuzzy-match it
-    same_faction = [e for e in catalog if e.get("faction") == item.faction]
-    pool = same_faction or catalog  # fall back to whole catalog if faction unknown
-    best = _best_in_pool(item, pool, use_inner=False)
-    if best is None or best[0] < threshold:
-        # Outer name not confident: let the archive's inner file names vote
-        # (this is the only signal for hash-named gofile archives).
-        best = _best_in_pool(item, pool, use_inner=True)
-    if best is None or best[0] < threshold:
-        # The keyword tables can lag behind GW's current categorization
-        # (e.g. units moved between factions); if the faction pool has no
-        # confident match, try the whole catalog before giving up.
-        if same_faction:
-            best = _best_in_pool(item, catalog, use_inner=False)
-            if best is None or best[0] < threshold:
-                best = _best_in_pool(item, catalog, use_inner=True)
-    if best and best[0] >= threshold:
-        entry = best[1]
-        item.unit = entry["name"]
-        if entry.get("group"):
-            item.group = entry["group"]
-        elif item.faction == UNSORTED:
-            item.group = GROUP_OF_FACTION.get(entry["faction"], UNSORTED)
-        if entry.get("subfaction"):
-            item.subfaction = entry["subfaction"]
-        # GW's class layer (shop category) beats the keyword guess, but do
-        # not *narrow* it: "Battleline" is GW's bucket for core infantry,
-        # a keyword-derived Hero/Elite is more specific for a proxy file.
-        catalog_klass = entry.get("class")
-        if catalog_klass and item.klass in (None, "", UNCLASSIFIED_CLASS):
-            item.klass = catalog_klass
-        item.match_score = best[0]
-        item.match_kind = "catalog"
-        # The catalog entry's faction is authoritative - but only adopt it
-        # when it is a real army faction. Generic GW branches like
-        # "Unit Type" or factionless entries would degrade a confident
-        # keyword faction to a junk folder.
-        entry_faction = entry.get("faction")
-        generic_factions = {"Unit Type", "Heroes", "Warriors", "Character"}
-        if item.faction == UNSORTED:
-            if entry_faction and entry_faction not in generic_factions:
-                item.system = entry["system"]
-                item.faction = entry_faction
-            if entry.get("group") and entry["group"] not in ("Unit Type",):
-                item.group = entry["group"]
-            else:
-                item.group = GROUP_OF_FACTION.get(item.faction, UNSORTED)
-        elif entry_faction and entry_faction not in generic_factions and entry_faction != item.faction:
-            # GW moved this unit to another faction (or the keyword tables
-            # are stale): trust the catalog.
-            item.system = entry["system"]
-            item.faction = entry_faction
-            if entry.get("group") and entry["group"] != "Unit Type":
-                item.group = entry["group"]
-            else:
-                item.group = GROUP_OF_FACTION.get(entry_faction, UNSORTED)
-        elif entry.get("group") and entry["group"] not in ("Unit Type",):
-            item.group = entry["group"]
-    else:
-        if item.faction == UNSORTED:
-            item.unit = item.name  # implicit entry: own (original) name
-            item.match_score = best[0] if best else 0.0
-            item.match_kind = "implicit"
-        else:
-            # Faction was known but nothing fuzzy-matched: keep the keyword
-            # faction and use the file's own name as the unit, but adopt the
-            # catalog's class layer when it adds information.
-            item.unit = item.name
-            item.match_score = best[0] if best else 0.0
-            item.match_kind = "implicit"
-
-
-def build_target(item: SortItem, library: Path) -> Path:
-    parts: list[str] = []
-    if item.system == "Unknown":
-        parts = [UNSORTED]
-    else:
-        parts = [item.system]
-        if item.faction != UNSORTED:
-            if item.group:
-                parts.append(item.group)
-            parts.append(item.faction)
-            if item.subfaction:
-                parts.append(item.subfaction)
-            parts.append(item.klass or UNCLASSIFIED_CLASS)
-            if item.unit:
-                parts.append(_display_name(item.unit))
-        else:
-            parts.append(UNSORTED)
-    target_dir = Path(library).joinpath(*parts)
-    return target_dir / item.name
-
-
-def _display_name(unit: str) -> str:
-    """Implicit entries carry the original filename; strip the extension."""
-    p = Path(unit)
-    return p.stem if p.suffix else unit
-
-
-# ---------------------------------------------------------------------------
-# plan / apply
-# ---------------------------------------------------------------------------
-
-def sort_plan(
-    files: list[Path],
-    library: Path,
-    catalog: list[dict],
-    threshold: float = 0.62,
-) -> list[SortItem]:
-    items = []
-    for f in sorted(files):
-        if not f.is_file():
-            continue
-        if (f.name.endswith(".part") or f.name.startswith(".")
-                or f.name in RUN_ARTIFACTS):
-            continue
-        item = inspect(f)
-        classify_item(item)
-        match_catalog(item, catalog, threshold)
-        item.target = build_target(item, library)
-        item.action = "link"
-        items.append(item)
-    return items
-
-
-def apply_plan(items: list[SortItem], audit_path: Path) -> tuple[int, int, int]:
-    """Create targets. Hardlink preferred; copy when filesystems differ.
-
-    Returns (linked, copied, skipped).
-    """
-    audit_path.parent.mkdir(parents=True, exist_ok=True)
-    linked = copied = skipped = 0
-    with audit_path.open("a", encoding="utf-8") as audit:
-        for item in items:
-            src, dest = item.path, item.target
-            if item.match_kind == "invalid":
-                log.error("Not sorting %s: it is an HTML page, not an archive", src.name)
-                skipped += 1
-                continue
-            if dest is None or not src.is_file():
-                skipped += 1
-                continue
-            if dest.exists():
-                log.info("Skip, target exists: %s", dest)
-                skipped += 1
-                continue
-            dest.parent.mkdir(parents=True, exist_ok=True)
-            if os.stat(src).st_dev == os.stat(dest.parent).st_dev:
-                os.link(src, dest)
-                item.action = "link"
-                linked += 1
-            else:
-                log.warning(
-                    "Different filesystem - COPYING instead of hardlinking %s", src.name
-                )
-                shutil.copy2(src, dest)
-                item.action = "copy"
-                copied += 1
-            audit.write(json.dumps({
-                "from": str(src), "to": str(dest), "action": item.action,
-                "unit": item.unit, "faction": item.faction, "class": item.klass,
-            }) + "\n")
-    return linked, copied, skipped
-
-
-def format_plan(items: list[SortItem]) -> str:
-    lines = []
-    for item in items:
-        if item.match_kind == "invalid":
-            lines.append(f"[x] {item.name} - NOT AN ARCHIVE (HTML page, bad download)")
-            continue
-        src = f"{item.name} ({item.size // (1024 * 1024)} MiB)" if item.size > 4 * 1024 * 1024 else item.name
-        flag = {"catalog": "+", "implicit": "~", "none": "?"}[item.match_kind]
-        lines.append(f"[{flag}] {src}\n      -> {item.target}")
-    return "\n".join(lines)

dlphelper/state.py

@@ -1,78 +0,0 @@
-"""Persistent per-URL state: makes runs resumable and failures trackable.
-
-state.json lives in the run's OUT_DIR and maps every URL seen to a record:
-
-    {
-      "status": "ok" | "failed" | "error",
-      "attempts": <int>,
-      "error": "<last error message>",
-      "files": [{"path": ..., "size": ...}, ...],
-      "updated": "<iso timestamp>"
-    }
-
-Re-running skips URLs whose status is "ok"; failed URLs are retried and also
-written to failed_urls.txt for easy manual retry lists.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import time
-from pathlib import Path
-
-log = logging.getLogger(__name__)
-
-
-class State:
-    """JSON-backed state store for one run directory."""
-
-    def __init__(self, path: Path):
-        self.path = path
-        self.records: dict[str, dict] = {}
-        if path.exists():
-            try:
-                self.records = json.loads(path.read_text(encoding="utf-8"))
-                if not isinstance(self.records, dict):
-                    raise ValueError("state root must be an object")
-            except (ValueError, OSError) as exc:
-                backup = path.with_suffix(".json.bak")
-                log.warning("Corrupt state file %s (%s); moving to %s and starting fresh",
-                            path, exc, backup)
-                path.replace(backup)
-                self.records = {}
-
-    def is_done(self, url: str) -> bool:
-        rec = self.records.get(url)
-        return bool(rec) and rec.get("status") == "ok"
-
-    def pending(self, urls: list[str]) -> list[str]:
-        return [u for u in urls if not self.is_done(u)]
-
-    def mark(self, url: str, **fields) -> dict:
-        rec = self.records.setdefault(url, {"status": "pending", "attempts": 0, "files": []})
-        for key, value in fields.items():
-            if value is not None:
-                rec[key] = value
-        rec["attempts"] = rec.get("attempts", 0) + 1
-        rec["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
-        self.save()
-        return rec
-
-    def failures(self) -> list[str]:
-        return [u for u, r in self.records.items() if r.get("status") != "ok"]
-
-    def save(self) -> None:
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        tmp = self.path.with_suffix(".json.tmp")
-        tmp.write_text(
-            json.dumps(self.records, indent=2, sort_keys=True), encoding="utf-8"
-        )
-        tmp.replace(self.path)
-
-
-def append_manifest(manifest_path: Path, row: dict) -> None:
-    """Append one JSON line to manifest.jsonl (per downloaded file)."""
-    manifest_path.parent.mkdir(parents=True, exist_ok=True)
-    with manifest_path.open("a", encoding="utf-8") as fh:
-        fh.write(json.dumps(row, ensure_ascii=False) + "\n")

dlphelper/warhammer_data.py

@@ -1,529 +0,0 @@
-"""Warhammer taxonomy and unit keyword tables used by the catalogue.
-
-Naming follows the Games Workshop webshop hierarchy (the same one
-dlphelper.gwshop dumps from Algolia):
-
-    <System>/<Group>/<Faction>/<Class>          e.g.
-    Warhammer 40,000/Space Marines/Ultramarines/Walker
-
-Two layers:
-- SYSTEMS: game system -> faction -> keyword aliases (matched against file
-  names). Faction detection is the primary signal; unit names double as
-  faction hints since they are listed under their faction. Space Marine
-  chapters are their own GW factions under the "Space Marines" group.
-- CLASSES: keyword -> class. With GW as the catalog source the class layer
-  holds GW shop categories (unit types: Battleline, Infantry, Walker,
-  Vehicle, Flyer, Monster, Mounted, Hero, Titanic).
-
-The unit table grows with the collection; it does not attempt to enumerate
-every unit Games Workshop ever released - unknown files land in the
-"unclassified" bucket of the catalogue report so gaps become visible.
-"""
-
-from __future__ import annotations
-
-# ---------------------------------------------------------------------------
-# Game systems and factions (keyword aliases, lowercase)
-# ---------------------------------------------------------------------------
-
-SYSTEMS: dict[str, dict[str, list[str]]] = {
-    "Warhammer 40,000": {
-        # -- Space Marines (chapters are their own GW factions) --------------
-        "Space Marines": [
-            "space marine", "astartes", "primaris", "intercessor", "reiver",
-            "aggressor", "inceptor", "hellblaster", "eradicator", "bladeguard",
-            "sternguard", "devastator", "tactical squad", "terminator squad",
-            "assault marine", "assault intercessor", "outrider", "scout squad",
-            "vanguard veteran", "invictor", "impulsor", "gladiator",
-            "land speeder", "drop pod", "inner circle", "honour guard",
-        ],
-        "Ultramarines": ["ultramarines", "guilliman", "calgar"],
-        "Blood Angels": [
-            "blood angel", "sanguinary guard", "sanguinor", "death company",
-            "baal predator", "mephiston", "dante", "astorath", "flesh tearers",
-            "lamberto", "furioso",
-        ],
-        "Dark Angels": [
-            "dark angel", "ravenwing", "deathwing", "inner circle companions",
-            "azrael", "asmodai", "belial", "samuel", "lion el'jonson",
-            "lion el jonson", "nephilim jetfighter",
-        ],
-        "Space Wolves": [
-            "space wolf", "space wolves", "wulfen", "thunderwolf", "wolf guard",
-            "rune priest", "wolf priest", "wolf lord", "logan grimnar",
-            "ragnar blackmane", "canis wolfborn", "fenrisian wolf",
-            "stormfang", "murderfang", "lukas the trickster",
-        ],
-        "Black Templars": [
-            "black templar", "crusader squad", "high marshal helbrecht",
-            "grimaldus", "the emperor's champion", "emperors champion",
-            "castellan", "land raider crusader",
-        ],
-        "Imperial Fists": [
-            "imperial fist", "rogal dorn", "lupos", "tor garadon",
-            "horatius hold", "dorn's arrow",
-        ],
-        "Raven Guard": [
-            "raven guard", "kayvaan shrike", "corvus", "sable brand",
-            "shadow captains",
-        ],
-        "Iron Hands": [
-            "iron hand", "iron hands", "ferrus manus", "iron father",
-            "medusan immortals",
-        ],
-        "White Scars": ["white scar", "jaghatai khan", "stormseer", "korsos"],
-        "Salamanders": [
-            "salamander", "vulkan", "he'stan", "adrax agatone",
-            "drake-scale", "flamecaster",
-        ],
-        "Grey Knights": [
-            "grey knight", "grand master", "brother-captain", "nemesis dreadknight",
-            "draigo", "castellan crowe", "sternguard knight",
-        ],
-        "Deathwatch": [
-            "deathwatch", "watch captain", "watch master", "kill team cassius",
-            "overkill", "corvus blackstar",
-        ],
-        "Crimson Fists": ["crimson fist", "kantor", "rynn's world"],
-        "Flesh Tearers": ["flesh tearer", "gabriel seth", "amissum"],
-        # -- Imperium --------------------------------------------------------
-        "Astra Militarum": [
-            "astra militarum", "imperial guard", "cadian", "catachan", "krieg",
-            "death korps", "tempestus", "scion", "valhallan", "steel legion",
-            "tallarn", "vostroyan", "tanith", "gaunt's ghosts", "leman russ",
-            "chimera", "basilisk", "hydra", "wyvern", "manticore", "deathstrike",
-            "sentinel", "taurox", "commissar", "bullgryn", "ogryn", "ratling",
-            "heavy weapon squad", "baneblade", "stormlord", "shadowsword",
-            "hellhammer", "malcador", "macharius", "valkyrie", "vulture",
-            "primaris psyker", "astropath", "priest",
-        ],
-        "Adeptus Mechanicus": [
-            "adeptus mechanicus", "mechanicum", "admech", "skitarii",
-            "cult mechanicus", "sicarian", "ruststalker", "infiltrator",
-            "kataphron", "kastelan", "ironstrider", "sydonian",
-            "onager", "dunecrawler", "maniple", "archmagos", "magos", "dominus",
-            "manipulus", "enginseer", "transuranic", "tech-priest", "cybernetica",
-            "thallax", "castellax", "vorax", "ordinatus", "belisarius cawl",
-        ],
-        "Adepta Sororitas": [
-            "adepta sororitas", "sisters of battle", "sister of battle",
-            "battle sister", "repentia", "retributor", "canoness", "dialogus",
-            "hospitaller", "seraphim", "zephyrim", "paragon warsuit",
-            "morvenn vahl", "junith", "immersa", "exorcist", "immolator",
-        ],
-        "Adeptus Custodes": [
-            "adeptus custodes", "custodes", "custodian", "allarus", "vertus praetor",
-            "venatari", "aquilon", "sagittarum", "hetaeron", "trajann", "constantin valdor",
-            "sisters of silence", "witchseeker", "prosecutor", "vigilator",
-            "caladius", "coronus", "telemon", "achillus", "galatus", "pallas",
-        ],
-        "Imperial Knights": [
-            "imperial knight", "knight errant", "knight paladin", "knight crusader",
-            "knight warden", "knight gallant", "knight lancer", "knight acheron",
-            "knight castellan", "knight valiant", "knight preceptor", "questoris",
-            "armiger", "warglaive", "helverin", "morialtis", "canis rex",
-            "cerastus", "atropos", "seneschal",
-        ],
-        "Imperial Agents": [
-            "inquisition", "inquisitor", "astra telepathica", "witch hunter",
-            "coteaz", "eisenhorn", "ravenor", "kyria draxus",
-            "officio assassinorum", "culexus", "vindicare", "callidus", "eversor",
-            "saint celestine", "living saint", "assassin",
-        ],
-        # -- Chaos -----------------------------------------------------------
-        "Chaos Space Marines": [
-            "chaos space marine", "black legion", "iron warriors", "word bearers",
-            "night lords", "alpha legion", "chaos terminator", "chosen", "havoc",
-            "raptor", "warp talon", "chaos cultist", "cultist", "daemon prince",
-            "chaos lord", "chaos sorcerer", "abaddon", "kharn", "ahriman",
-            "typhus", "fabius bile", "vashtorr", "defiler", "forgefiend",
-            "maulerfiend", "helldrake", "helbrute", "chaos spawn",
-            "obliterator", "chaos rhino", "chaos vindicator", "spartan assault",
-            "mastodon", "typhon",
-        ],
-        "Death Guard": [
-            "death guard", "nurgle", "plague marine", "deathshroud", "mortarion",
-            "plaguecaster", "poxwalker", "blightlord", "foetid bloat-drone",
-            "plagueburst crawler", "myphitic blight-hauler", "blight hauler",
-            "tallyman", "foul blightspawn", "biologus putrifier",
-            "plague surgeon", "necrosius",
-        ],
-        "Thousand Sons": [
-            "thousand sons", "tzeentch", "rubric marine", "rubricae",
-            "scarab occult", "magnus the red", "ahzek ahriman", "arhiman",
-            "exalted sorcerer", "osiron", "mutalith vortex beast", "tzaangor",
-            "prospero",
-        ],
-        "Chaos Daemons": [
-            "chaos daemon", "chaos daemons", "daemonette", "bloodletter",
-            "plaguebearer", "horror", "screamer", "flesh hound", "juggernaut",
-            "keeper of secrets", "skarbrand", "kairos fateweaver", "rotigus",
-            "be'lakor", "great unclean one", "lord of change", "soul grinder",
-            "skull cannon", "burning chariot", "exalted flamer",
-            "fiend of slaanesh", "plague drone", "nurgling", "herald",
-            "skulltaker", "khorne", "slaanesh", "skull lord",
-            "shalaxi helbane",
-        ],
-        "World Eaters": [
-            "world eater", "angron", "eightbound", "exalted eightbound",
-            "jakhals", "berserker", "skull lord",
-        ],
-        "Emperor's Children": [
-            "emperor's children", "emperors children", "fulgrim",
-            "noise marine", "lucius the eternal", "flawless blades",
-        ],
-        "Iron Warriors": ["iron warrior", "perturabo"],
-        "Red Corsairs": ["red corsair", "huron blackheart"],
-        "Chaos Knights": [
-            "chaos knight", "knight desecrator", "knight despoiler",
-            "knight rampager", "kharibdyss", "renegade knight",
-        ],
-        # -- Xenos -----------------------------------------------------------
-        "Orks": [
-            "ork", "orks", "orkz", "nob", "boyz", "grot", "weirdboy", "painboy",
-            "warboss", "ghazghkull", "meganob", "deff dread", "killa kan",
-            "deffkopta", "stormboy", "kommando", "burna boy", "loota",
-            "trukk", "battlewagon", "gorkanaut", "morkanaut", "stompa", "squig",
-            "zagstruk", "snikrot", "speed freek", "boomdakka", "flash gitz",
-            "wazbom blasta", "dakka jet", "burna-bomber", "blitza-bomber",
-            "da troopz", "dad and sonz", "ass sassins", "abby boi",
-        ],
-        "T'au Empire": [
-            "tau", "t'au", "ta'u", "kroot", "vespid", "battlesuit", "crisis suit",
-            "broadside", "r'varna", "ghostkeel", "stormsurge",
-            "fire warrior", "pathfinder team", "devilfish", "hammerhead",
-            "sky ray", "piranha", "sniper drone team", "ethereal",
-            "commander farsight", "commander shadowsun", "aun'va", "darkstrider",
-            "longstrike", "mont'ka", "assorted drones", "breacher",
-            "bomber", "tetra", "barracuda", "sun shark", "razorshark",
-            "kv128 stormsurge", "tx4 piranha", "kv46",
-        ],
-        "Tyranids": [
-            "tyranid", "hive tyrant", "swarmlord", "carnifex", "hormagaunt",
-            "termagant", "termagaunt", "genestealer brood", "zoanthrope", "venomthrope",
-            "exocrine", "haruspex", "tyranid prime", "tyranid warrior",
-            "ravener", "gargoyle", "spore mine", "mawloc", "trygon",
-            "harridan", "harpy", "crone", "tervigon", "barbed hierodule",
-            "hierodule", "hierophant", "neurothrope", "deathleaper", "lictor", "pyrovore",
-            "biovore", "broodlord", "bio-titan", "dominatrix",
-        ],
-        "Genestealer Cults": [
-            "genestealer cult", "genestealer", "aberrant", "acolyte", "neophyte hybrid",
-            "primus", "magus", "patriarch", "purestrain genestealer",
-            "sanctus", "kelermorph", "goliath truck", "rockgrinder",
-            "jackal alcaeus", "cult limo", "hijacked",
-        ],
-        "Necrons": [
-            "necron", "c'tan", "canoptek", "wraith", "spyder", "scarab",
-            "tesseract vault", "doomsday ark", "annihilation barge",
-            "ghost ark", "monolith", "doom scythe", "night scythe",
-            "imotekh", "szarekh", "nemesor zahndrekh", "vargard obyron",
-            "lokhrust", "skorpekh", "flayed one", "deathmark", "lychguard",
-            "necron warrior", "necron immortal", "ovoid", "dzaan",
-        ],
-        "Aeldari": [
-            "aeldari", "eldar", "craftworld", "ulthwe", "biel-tan", "iyanden",
-            "alaitoc", "saim-hann", "farseer", "warlock",
-            "spiritseer", "autarch", "asurmen", "guardian defender",
-            "ranger", "wraithguard", "wraithblades", "wraithlord",
-            "wraithknight", "dire avenger", "howling banshee",
-            "striking scorpion", "swooping hawk", "warp spider",
-            "fire dragon", "dark reaper", "shining spears", "windrider",
-            "vyper", "wave serpent", "fire prism", "night spinner",
-            "war walker", "hemlock wraithfighter", "crimson hunter",
-            "shade runner", "skyrunner",
-        ],
-        "Drukhari": [
-            "drukhari", "dark eldar", "kabal", "wych", "wrack", "haemonculus",
-            "coven", "reaver jetbike", "hellion", "scourge", "mandrake",
-            "kabalite warrior", "raider", "ravager", "venom", "razorwing",
-            "voidraven", "grotesque", "talos", "cronos",
-            "beast pack", "khymera", "razorwing flock", "asdrubael vect",
-            "lilith", "sathonyx", "luriel",
-        ],
-        "Harlequins": [
-            "harlequin", "death jester", "shadowseer", "solitaire",
-            "troupe", "starweaver", "voidweaver",
-        ],
-        "Leagues of Votann": [
-            "leagues of votann", "votann", "hearthkyn", "ironkin",
-            "brokhyr", "kahl", "cthonian", "transhyperic", "thegn",
-        ],
-    },
-    "Age of Sigmar": {
-        "Stormcast Eternals": [
-            "stormcast", "eternal", "liberator", "sequitor", "evocator",
-            "vindictor", "judicators", "retributors", "sacrosanct",
-            "dracothian", "lord-arcane", "knight-questor", "neave blacktalon",
-            "stormdrake", "krondys", "karazai",
-        ],
-        "Seraphon": ["seraphon", "saurus", "skink", "slann", "carnosaur",
-                     "bastiladon", "stegadon", "razordon", "salamander"],
-        "Orruk Warclans": ["orruk", "warclan", "kruleboyz", "ironjawz",
-                          "brute", "gorger", "weirdnob", "warchanter"],
-        "Gloomspite Gitz": ["gloomspite", "gitz", "moonclan", "spiderfang",
-                            "troggoth", "loontouched", "zarbag", "skragrot"],
-        "Sylvaneth": ["sylvaneth", "treelord", "dryad", "branchwraith",
-                      "alarielle", "kurnothi"],
-        "Ossiarch Bonereapers": ["ossiarch", "bonereaper", "mortek", "keldrek",
-                                 "katakros"],
-        "Nighthaunt": ["nighthaunt", "grimghast", "bladegheist", "chainghast",
-                       "kurdoss", "lady olynder"],
-        "Lumineth Realm-lords": ["lumineth", "aelf", "vanari", "teclis",
-                                 "light of eltharion"],
-        "Daughters of Khaine": ["daughters of khaine", "khainite", "witch aelf",
-                                "morgwaeth", "morathi"],
-        "Fyreslayers": ["fyreslayer", "vulkite", "hearthguard", "auric",
-                        "grimhold"],
-        "Kharadron Overlords": ["kharadron", "skyfarer", "endofters",
-                                "grundstok", "arcanaut"],
-        "Idoneth Deepkin": ["idoneth", "deepkin", "namarti", "isharann",
-                            "volturnos"],
-        "Blades of Khorne": ["blades of khorne", "khorne", "skullgrinder",
-                             "blood warrior", "wrathmonger"],
-        "Maggotkin of Nurgle": ["maggotkin", "nurgle", "blightking",
-                                "rotmire", "orghotts"],
-        "Slaves to Darkness": ["slaves to darkness", "chaos warrior",
-                               "chaos knight", "archaon", "gaunt summoner",
-                               "everchosen"],
-        "Skaven": ["skaven", "clanrat", "stormvermin", "verminlord",
-                   "grey seer", "warplock", "ratling gun"],
-        "Soulblight Gravelords": ["soulblight", "gravelord", "vampire",
-                                  "deathrattle", "koldhunter", "belladamma"],
-        "Cities of Sigmar": ["cities of sigmar", "freeguild", "windspeaker",
-                             "callis and toll", "hammerers"],
-    },
-    "The Horus Heresy": {
-        "Space Marine Legions": [
-            "legiones astartes", "pre-heresy", "horus heresy", "justaerin",
-            "seeker squad", "suzerain", "invictarus", "veteran tactical",
-            "contemptor", "leviathan pattern", "sicaran", "fellblade",
-            "kratos", "fire raptor", "eisenstein",
-        ],
-        "Mechanicum": ["mechanicum", "fulgurite", "magos", "thallax"],
-    },
-    "Other Games": {
-        "Kill Team": [
-            "kill team", "killteam", "killzone", "operative",
-            "hand of the archon", "void-dance troupe", "veteran guardsmen",
-            "spector squad", "wolf scouts", "scout squad kill",
-        ],
-        "Necromunda": [
-            "necromunda", "goliath gang", "escher", "orlock", "cawdor",
-            "van saar", "ash wastes", "underhive", "sumpkrocs",
-            "forge boss", "sumpkroc", "matriarch", "phyrr cat",
-        ],
-        "Adeptus Titanicus": [
-            "adepticus titanicus", "titan legio", "reaver titan",
-            "warhound titan", "warlord titan", "imperator titan",
-            "warmaster titan", "titanicus",
-        ],
-        "Warhammer Underworlds": ["warhammer underworlds", "underworlds",
-                                  "warband", "shadeborn", "embergard"],
-        "Aeronautica Imperialis": ["aeronautica", "t'au air caste",
-                                   "lightning fighter", "thunderbolt"],
-    },
-    "The Old World": {
-        "Kingdom of Bretonnia": ["bretonnia", "grail knight", "peasant",
-                                 "questing knight", "perrin"],
-        "Orc and Goblin Tribes": ["greenskin", "goblin", "night goblin",
-                                  "forest goblin", "black orc"],
-        "Tomb Kings of Khemri": ["tomb king", "khemri", "necrolith",
-                                 "skeleton archer"],
-        "Empire of Man": ["empire of man", "state troop", "greatsword",
-                          "empire knight"],
-        "Warriors of Chaos": ["chaos dwarf", "dog of war", "chaos marauder"],
-        "High Elf Realms": ["high elf", "eltharion", "phoenix guard"],
-        "Wood Elf Realms": ["wood elf", "glade guard", "waywatcher"],
-        "Beastmen Brayherds": ["beastmen", "brayherd", "ghorgon", "cygor"],
-        "Dwarfen Mountain Holds": ["dwarf", "ironbreaker", "hammerer"],
-    },
-    "Middle-Earth": {
-        "Rivendell": ["rivendell", "elrond", "elendil", "isildur", "gil-galad",
-                      "beorning", "grimbeorn", "rumil", "orophin"],
-        "Minas Tirith": ["minas tirith", "gondor", "boromir", "faramir"],
-        "Mordor": ["mordor", "orc", "orcs", "nazgul", "witch-king"],
-        "Iron Hills": ["iron hills", "dwarf", "mattock", "dwalin"],
-        "Moria": ["moria", "cave troll", "balrog"],
-    },
-}
-
-# ---------------------------------------------------------------------------
-# Classes (keyword -> class). GW shop categories / unit types.
-# ---------------------------------------------------------------------------
-
-CLASSES: list[tuple[str, list[str]]] = [
-    ("Titanic", [
-        "titan", "stompa", "gargantuan", "hierophant", "harridan", "hierodule",
-        "imperator", "warmaster", "tesseract vault", "plaguereaper",
-        "warhound", "warlord titan", "ta'unar", "spermato", "titandeath",
-    ]),
-    ("Fortification", ["fortification", "bunker", "bastion", "wall of martyrs"]),
-    ("Flyer", [
-        "flyer", "heldrake", "dakka jet", "burna-bomber", "blitza-bomber",
-        "bomber", "sky ray", "stormraven", "storm talon", "stormhawk",
-        "caestus", "valkyrie", "vulture", "thunderbolt", "lightning",
-        "marauder", "razorwing", "voidraven", "crimson hunter",
-        "hemlock", "doom scythe", "night scythe", "harpy", "crone",
-        "fire raptor", "barracuda", "sun shark", "razorshark",
-        "wazbom blasta", "nephilim jetfighter", "corvus blackstar",
-    ]),
-    ("Vehicle", [
-        "transport", "rhino", "razorback", "chimera", "taurox", "trukk",
-        "battlewagon", "devilfish", "wave serpent", "ghost ark",
-        "land raider", "repulsor", "drop pod", "starweaver", "voidweaver",
-        "goliath truck", "rockgrinder", "cult limo", "raider", "venom",
-        "spartan assault", "predator", "vindicator", "whirlwind",
-        "basilisk", "hydra", "wyvern", "manticore", "deathstrike",
-        "onager dunecrawler", "kastelan", "hammerhead", "broadside",
-        "stormsurge", "r'varna", "soul grinder", "defiler", "forgefiend",
-        "maulerfiend", "plagueburst crawler", "annihilation barge",
-        "doomsday ark", "fire prism", "night spinner", "exorcist",
-        "immolator", "caladius", "coronus grav", "pallas grav",
-        "impulsor", "gladiator", "baal predator", "land speeder",
-        "ironstrider", "sydonian", "sentinel", "taurox", "kataphron",
-    ]),
-    ("Walker", [
-        "dreadnought", "redemptor", "contemptor", "leviathan pattern",
-        "brutalis", "ballistus", "deredeo", "wulfen dreadnought",
-        "killa kan", "deff dread", "wraithlord", "wraithknight",
-        "telemon", "achillus", "galatus", "paragon warsuit",
-        "nemesis dreadknight", "invictor", "talos", "cronos",
-    ]),
-    ("Monster", [
-        "monster", "carnifex", "exocrine", "haruspex", "mawloc", "trygon",
-        "tervigon", "hierodule", "hierophant", "mutalith", "bloat-drone",
-        "kharibdyss", "gorkanaut", "morkanaut", "magnus", "mortalith",
-        "fiend of slaanesh", "plague drone", "cave troll", "balrog",
-        "carnosaur", "stegadon", "bastiladon", "grorgeous",
-    ]),
-    ("Mounted", [
-        "mounted", "thunderwolf", "outrider", "bike squad", "scout biker",
-        "reaver jetbike", "windrider", "skyrunner", "warbiker",
-        "allarus custodian on", "vertus praetor", "drakeling",
-        "boarboy", "gruntaz", "khan on", "dracothian guard",
-    ]),
-    ("Battleline", [
-        "battleline", "troops", "tactical squad", "intercessor",
-        "incursor", "infiltrator", "infantry squad", "conscript",
-        "krieg", "catachan", "cadian", "vostroyan", "penal legion",
-        "ratling", "veteran squad", "fire warrior", "pathfinder team",
-        "kroot", "termagant", "hormagaunt", "necron warrior",
-        "necron immortal", "guardian defender", "storm guardian",
-        "kabalite warrior", "boyz", "gretchin", "scion", "scout squad",
-        "cultist", "plague marine squad",
-    ]),
-    ("Infantry", [
-        "infantry", "veteran", "terminator", "sternguard", "bladeguard",
-        "aggressor", "reiver", "wulfen", "wolf guard", "deathshroud",
-        "blightlord", "poxwalker", "rubric marine", "rubricae",
-        "scarab occult", "tzaangor", "chosen", "possessed", "obliterator",
-        "daemonette", "bloodletter", "plaguebearer", "nurgling",
-        "juggernaut", "chaos cultist", "acolyte hybrid", "aberrant",
-        "neophyte", "purestrain", "genestealer", "lictor", "deathmark",
-        "flayed one", "lychguard", "wraithguard", "wraithblades",
-        "howling banshee", "striking scorpion", "warp spider",
-        "fire dragon", "incubi", "wrack", "grotesque", "assault marine",
-        "assault intercessor", "inceptor", "raptor", "warp talon",
-        "flesh hound", "screamer", "hellion", "scourge", "mandrake",
-        "swooping hawk", "shining spears", "canoptek wraith",
-        "canoptek scarab", "canoptek spyder", "ravener", "gargoyle",
-        "spore mine", "pyrovore", "khymera", "beast pack",
-        "razorwing flock", "zephyrim", "seraphim", "stormboy",
-        "deffkopta", "skorpekh destroyer", "skorpekh", "retributor",
-        "dark reaper", "sicarian", "ruststalker", "skitarii",
-        "hormagaunt brood", "termagant brood", "vanguard",
-    ]),
-    ("Hero", [
-        "hero", "hq", "captain", "chapter master", "primarch", "lieutenant",
-        "librarian", "chaplain", "apothecary", "farseer", "warlock",
-        "spiritseer", "autarch", "warboss", "big mek", "weirdboy",
-        "painboy", "commander", "ethereal", "hive tyrant", "swarmlord",
-        "tyranid prime", "neurothrope", "deathleaper", "patriarch",
-        "primus", "magus", "sanctus", "kelermorph", "overlord",
-        "imotekh", "szarekh", "zahndrekh", "obyron", "plaguecaster",
-        "necrosius", "chaos lord", "chaos sorcerer", "daemon prince",
-        "exalted champion", "herald", "skulltaker", "kairos", "rotigus",
-        "be'lakor", "great unclean one", "lord of change",
-        "keeper of secrets", "skarbrand", "kharn", "abaddon", "typhus",
-        "mortarion", "magnus", "ahriman", "lucius", "fabius bile",
-        "vashtorr", "canoness", "morvenn", "junith", "immersa",
-        "dialogus", "hospitaller", "trajann", "valdor", "inquisitor",
-        "coteaz", "eisenhorn", "ravenor", "draxus", "celestine",
-        "commissar", "priest", "enginseer", "dominus", "manipulus",
-        "archmagos", "belisarius cawl", "farsight", "shadowsun",
-        "aun'va", "darkstrider", "longstrike", "shade runner",
-    ]),
-    ("Rules & Accessories", [
-        "codex", "rulebook", "rules", "data cards", "datacards",
-        "warscroll", "index card", "cheat sheet", "reference",
-    ]),
-]
-
-# Flattened per-system faction keyword index for quick matching
-FACTION_ALIASES: dict[str, dict[str, str]] = {
-    system: {alias: faction for faction, aliases in factions.items() for alias in aliases}
-    for system, factions in SYSTEMS.items()
-}
-
-UNIT_CLASSES: dict[str, str] = {kw: klass for klass, kws in CLASSES for kw in kws}
-
-# GW group of each faction (matches the GameSystemsRoot.lvl1 values in the
-# Algolia index). Used when a keyword-classified file cannot be fuzzy-matched.
-GROUP_OF_FACTION: dict[str, str] = {
-    faction: group
-    for group, factions in {
-        "Space Marines": [
-            "Space Marines", "Ultramarines", "Blood Angels", "Dark Angels",
-            "Space Wolves", "Black Templars", "Imperial Fists", "Raven Guard",
-            "Iron Hands", "White Scars", "Salamanders", "Grey Knights",
-            "Deathwatch", "Crimson Fists", "Flesh Tearers",
-        ],
-        "Armies of the Imperium": [
-            "Astra Militarum", "Adeptus Mechanicus", "Adepta Sororitas",
-            "Adeptus Custodes", "Imperial Knights", "Imperial Agents",
-        ],
-        "Armies of Chaos": [
-            "Chaos Space Marines", "Death Guard", "Thousand Sons",
-            "Chaos Daemons", "Chaos Knights", "World Eaters",
-            "Emperor's Children", "Iron Warriors", "Red Corsairs",
-        ],
-        "Xenos Armies": [
-            "Orks", "T'au Empire", "Tyranids", "Genestealer Cults", "Necrons",
-            "Aeldari", "Drukhari", "Harlequins", "Leagues of Votann",
-        ],
-        "Grand Alliance Order": [
-            "Stormcast Eternals", "Seraphon", "Sylvaneth",
-            "Lumineth Realm-lords", "Daughters of Khaine", "Fyreslayers",
-            "Kharadron Overlords", "Idoneth Deepkin", "Cities of Sigmar",
-        ],
-        "Grand Alliance Destruction": [
-            "Orruk Warclans", "Gloomspite Gitz",
-        ],
-        "Grand Alliance Death": [
-            "Ossiarch Bonereapers", "Nighthaunt", "Soulblight Gravelords",
-        ],
-        "Grand Alliance Chaos": [
-            "Slaves to Darkness", "Blades of Khorne", "Maggotkin of Nurgle",
-            "Skaven",
-        ],
-        "Space Marine Legions": ["Space Marine Legions", "Mechanicum"],
-        "Kill Team": ["Kill Team"],
-        "Necromunda": ["Necromunda"],
-        "Adeptus Titanicus": ["Adeptus Titanicus"],
-        "Warhammer Underworlds": ["Warhammer Underworlds"],
-        "Aeronautica Imperialis": ["Aeronautica Imperialis"],
-        "Armies of the Old World": [
-            "Kingdom of Bretonnia", "Orc and Goblin Tribes",
-            "Tomb Kings of Khemri", "Empire of Man", "Warriors of Chaos",
-            "High Elf Realms", "Wood Elf Realms", "Beastmen Brayherds",
-            "Dwarfen Mountain Holds",
-        ],
-        "Rivendell": ["Rivendell"],
-        "Minas Tirith": ["Minas Tirith"],
-        "Mordor": ["Mordor"],
-        "Iron Hills": ["Iron Hills"],
-        "Moria": ["Moria"],
-    }.items()
-    for faction in factions
-}

dlphelper/zero807.py

@@ -1,91 +0,0 @@
-"""Downloader for 0807.st direct links via curl.
-
-0807.st serves the files directly but rejects requests that don't look like a
-browser (requires a User-Agent and the Referer of the site page), and it
-rejects multi-connection/range abuse patterns differently than curl - which is
-why curl is used instead of aria2c here (see chat history: curl worked,
-aria2c's default request got 403).
-
-Downloads go to a .part file first and are renamed on success, so an
-interrupted download is resumable (--continue-at -) and partial files are
-never mistaken for finished ones.
-"""
-
-from __future__ import annotations
-
-import logging
-import os
-import subprocess
-from pathlib import Path
-from urllib.parse import urlparse
-
-log = logging.getLogger(__name__)
-
-USER_AGENT = "Mozilla/5.0"
-REFERER = "https://0807.st/"
-
-
-class Zero807Error(Exception):
-    """Base error for 0807 downloads."""
-
-
-class Zero807NotFound(Zero807Error):
-    """The file no longer exists (HTTP 404/410). Not retryable."""
-
-
-def target_name(url: str) -> str:
-    """Derive the filename from the URL path (e.g. /6saCNie.zip -> 6saCNie.zip)."""
-    name = Path(urlparse(url).path).name
-    if not name:
-        raise Zero807Error(f"cannot derive a filename from URL: {url}")
-    return name
-
-
-def download(
-    url: str,
-    out_dir: Path,
-    timeout_seconds: int = 7200,
-    resume: bool = True,
-) -> tuple[Path, int]:
-    """Download one URL with curl; returns (final_path, size_bytes)."""
-    name = target_name(url)
-    out_dir.mkdir(parents=True, exist_ok=True)
-    dest = out_dir / name
-    tmp = dest.with_name(dest.name + ".part")
-
-    if dest.exists() and dest.stat().st_size > 0:
-        log.info("Skip existing: %s", dest)
-        return dest, dest.stat().st_size
-
-    cmd = [
-        "curl", "-fSL",
-        "--connect-timeout", "20",
-        "--max-time", str(timeout_seconds),
-        "--retry", "2", "--retry-delay", "5",
-        "-A", USER_AGENT,
-        "-e", REFERER,
-        "-o", str(tmp),
-    ]
-    if resume:
-        cmd.append("--continue-at")
-        cmd.append("-")
-    cmd.append(url)
-
-    log.info("Downloading %s -> %s", url, dest.name)
-    proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-    if proc.returncode != 0:
-        tail = [line for line in proc.stderr.strip().splitlines() if line][-3:]
-        stderr = " | ".join(tail)
-        # -f makes curl exit 22 on any HTTP error; dead links fail for good
-        if "404" in stderr or "410" in stderr:
-            raise Zero807NotFound(f"dead link (HTTP 404/410): {stderr}")
-        raise Zero807Error(f"curl exit code {proc.returncode}: {stderr}")
-
-    size = tmp.stat().st_size if tmp.exists() else 0
-    if size == 0:
-        tmp.unlink(missing_ok=True)
-        raise Zero807Error(f"empty file for {name}")
-    if dest.suffix.lower() == ".zip" and not tmp.read_bytes()[:2] == b"PK":
-        log.warning("%s does not look like a ZIP file (first bytes are not PK)", name)
-    os.replace(tmp, dest)
-    return dest, size

docs/SORTING.md

@@ -7,8 +7,7 @@ sorting-related code; change it here first, then in code.
 ## Target hierarchy (Games Workshop webshop-aligned)
 
 The hierarchy follows the official GW webshop taxonomy (dumped from the
-shop's Algolia index by `dlphelper/gwshop.py` into
+shop's Algolia index by `dlphelper gwshop` into `data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
-`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
 hierarchy treated as sub-factions are full GW factions, and the class layer
 holds GW shop categories (unit types / product lines) instead of the retired
 40k force-organization slots:
@@ -36,12 +35,11 @@ Layer rules:
 
 - Files without a faction but with a game land in `<game>/_unsorted/`.
 - Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
-- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
+- The keyword tables in `data/warhammer_data.json` (GW naming) provide the
-  system/faction/class signal; a catalog match then overrides the hierarchy
+  initial system/faction/class signal; a catalog match then overrides the
-  from the catalog entry, which is authoritative.
+  hierarchy from the catalog entry, which is authoritative.
-- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
+- The catalog is pluggable via `dlphelper sorter --catalog`; every entry
-  still usable via `sorter.py --catalog`; its sub-faction entries map into
+  uses the same schema. For GW-sourced entries `subfaction` stays empty.
-  the same schema. For GW-sourced entries `subfaction` stays empty.
 
 ## Canonical names
 
@@ -63,23 +61,32 @@ graph LR
 ```
 
 1. **Inspect** — read the file name; for archives, also read the *inner*
-   file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
+   file list (zip via the `zip` crate, 7z via `sevenz-rust2` — no external
-   no Python dependency). Inner names improve classification for hash-named
+   binary). Inner names improve classification for hash-named archives and
-   archives and are recorded in the report.
+   are recorded in the report.
 2. **Classify** — game/faction/class from the keyword tables
-   (`warhammer_data.py`), same as `catalogue.py`.
+   (`data/warhammer_data.json`).
-3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
+3. **Match** — fuzzy-match the (inner) name against the catalog
-   entries within the same faction; best sequence-match above
+   (`data/gwshop_catalog.json`) entries within the same faction; best
-   `--threshold` (default 0.62) wins. The catalog provides canonical names,
+   sequence-match above `--threshold` (default 0.62) wins. The catalog
-   sub-factions and classes.
+   provides canonical names, sub-factions and classes.
-4. **Plan** — default output: a dry-run table `source → target` plus
+4. **Plan** — default output: a dry-run table `source → target` on stdout
-   `sorter_plan.jsonl`. Nothing is touched.
+   (`--json` prints the same plan as JSON). Nothing is touched.
 5. **Apply** — create the target and **hardlink** the file there
-   (`os.link`); if the filesystems differ, fall back to a real copy with a
+   (`std::fs::hard_link`); if the filesystems differ, fall back to a real
-   warning. Originals are never modified or deleted — that is what keeps the
+   copy with a warning. Originals are never modified or deleted — that is
-   *arr-style workflow possible (remux/seed/sort independently). Every
+   what keeps the *arr-style workflow possible (remux/seed/sort
-   applied action is appended to `sorter_applied.jsonl` and re-runs skip
+   independently). Every applied action is appended to `sorter_applied.jsonl`
-   targets that already exist.
+   and re-runs skip targets that already exist.
+
+## Bookkeeping location
+
+The audit (`sorter_applied.jsonl`) and the low-confidence review doc
+(`sorter_low_confidence.txt`) are written to `<library>/../_logs/` by
+default — **never inside the library itself**, because the library may be
+a seeded torrent whose file set is frozen (a rewritten audit file would
+fail the client's recheck). `--audit FILE` overrides this; the low-
+confidence doc always lands next to the audit file.
 
 ## Why hardlinks, and the unzip question
 
@@ -88,14 +95,83 @@ graph LR
   sees. Limitation: hardlinks need source and library on the same
   filesystem; the sorter detects this and copies instead (warned).
 - Zip contents **can** be read without extracting (that is how the sorter
-  peeks). Modifying a member in place is technically possible with
+  peeks). Modifying a member in place is technically possible with the `zip`
-  `zipfile` but is not useful here.
+  crate but is not useful here.
 - Endgame (later stage): unpack archives into the unit folder so the
   library is plain files, easier to sort per-model and to torrent. The
   unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
   until the link-based flow is proven on a small batch.
 
-## Catalog file (`data/library_catalog.json`)
+## Extraction stage (`dlphelper extract`)
+
+That endgame stage now exists as a standalone command (the sorter stays
+plan/apply-only):
+
+    dlphelper extract <src_root>... --out <extracted_root>
+
+- Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
+  into a folder named after the archive, preserving relative structure;
+  nested archives are expanded recursively and deleted, so the tree ends
+  up with **no archives at all**.
+- Plain files are hardlinked into the tree (fallback copy) — nothing lost,
+  no duplicate data.
+- Sources are never modified. Failed extractions (HTML-page fake zips,
+  password archives, disk full) are moved to `<out>/.broken/` and the run
+  continues; re-runs skip already-extracted destinations.
+- `<out>/extract_manifest.jsonl` records `{archive, dest, unit}` per
+  extraction. By **default the sorter classifies each file on its own
+  name**; the optional `--provenance <manifest>` mode instead classifies
+  each extraction directory **once under the archive's name** (a loose
+  `Backpack.stl` alone is noise; under `The Twin Lance/` it belongs to
+  that unit) and keeps the archive's inner directory structure. The live
+  library uses the default (non-provenance) mode:
+
+      # default — what Warhammer_Library_Live uses
+      dlphelper sorter <extracted_root> --library <lib> --apply
+
+      # optional provenance mode (whole extraction dirs as units)
+      dlphelper sorter <extracted_root> --library <lib> \
+          --provenance <extracted_root>/extract_manifest.jsonl --apply
+
+Operational notes from the first full run (2026-10-06, sources
+`unsorted/1..4` → `extracted` ≈ 1.4 TB / 260k files / 0 archives left;
+library rebuilt from scratch: 218592 hardlinked, 30584 skipped as
+already-present duplicates, 0 copied):
+
+- The CIFS NAS mount reports **fabricated per-path inode numbers**
+  (`noserverino` behavior): two paths to the same hardlinked file show
+  different `st_ino`, and the values are stable across calls. To verify a
+  link, compare `st_nlink` (and content/size) — never inodes. Server-side
+  links are real: a fresh `std::fs::hard_link` bumps nlink (2→3 in the
+  checked case).
+- Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2
+  ("Cannot open output file"); the extractor then extracts to a `/var/tmp`
+  temp dir, repairs the names and moves the result over.
+- `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not
+  extracted.
+- Files under hidden trees (`.broken/`) are never planned; failed archives
+  (HTML-page fake zips, password archives, disk full) are quarantined there
+  for inspection and the run continues.
+- **Non-warhammer systems are dropped** (2026-10-07, user decision:
+  ("drop anything that isn't really warhammer"). The sorter default
+  (`DROPPED_SYSTEMS` in `src/sorter.rs`) is `{Middle-Earth}`:
+  catalog entries of dropped systems are filtered at load and keyword
+  hits for them count as unclassified, so nothing is ever placed into
+  those sections - the files stay in `_unsorted`. `--drop-system NAME`
+  drops more, `--keep-system NAME` re-enables one. The committed catalog
+  snapshot itself is untouched.
+- **Probe votes on full inner paths** (2026-10-07): the pack probe's
+  inner-name votes clean the whole relative path (dir names included),
+  not just the file stem - hash-named 0807 packs resolve their faction
+  from folders like `Catachan/Jungle Fighters/` instead of from one
+  root file named `lieutenant`. The token-containment boost is disabled
+  for single-token inner votes (one generic-ish word must not claim a
+  pack), and full-catalog/different-faction adoptions additionally
+  require token containment (`_contained_somewhere`). A matched catalog
+  unit's class layer always wins (no keyword class survives next to a
+  precise unit match).
+
+## Catalog file (`data/gwshop_catalog.json`)
 
 ```json
 {

parts/grab.sh

@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-THREAD_ID="${THREAD_ID:-1455993943471820842}"
-OUT_DIR="${OUT_DIR:-$PWD/thread}"
-HTML="$OUT_DIR/export.html"
-URLS="$OUT_DIR/urls.txt"   # aria2c input file: one URL per line
-
-# Token from env, otherwise prompt (input hidden)
-if [[ -z "${DISCORD_TOKEN:-}" ]]; then
-  read -rsp "Discord token: " DISCORD_TOKEN
-  echo
-fi
-
-mkdir -p "$OUT_DIR"
-
-# 1. Export the thread to HTML
-docker run --rm --user "$(id -u):$(id -g)" \
-  -v "$OUT_DIR:/out" \
-  tyrrrz/discordchatexporter:stable export \
-  -t "$DISCORD_TOKEN" -c "$THREAD_ID" -f HtmlDark -o /out/export.html
-
-unset DISCORD_TOKEN
-
-# 2. Find base64 strings starting with "https://", then decode them
-grep -oP 'aHR0cHM6Ly[89][A-Za-z0-9+/_-]*' "$HTML" \
-  | sort -u \
-  | while read -r s; do
-      s=$(printf '%s' "$s" | tr '_-' '/+' | tr -d '=')
-      while (( ${#s} % 4 )); do s="$s="; done
-      printf '%s' "$s" | base64 -d 2>/dev/null | tr -d '\r\n' || true
-      echo
-    done \
-  | grep -E '^https://[[:graph:]]+$' \
-  | awk '!seen[$0]++' > "$URLS"
-
-echo "Found $(wc -l < "$URLS") URLs -> $URLS"
-echo "Download with: aria2c -i \"$URLS\" -d \"$OUT_DIR/downloads\""

requirements.txt

@@ -1 +0,0 @@
-requests>=2.31

sorter.py

@@ -1,102 +0,0 @@
-#!/usr/bin/env python3
-"""Sorter CLI: fuzzysort collection files into the library (plan/apply).
-
-    # plan (default, touches nothing)
-    python3 sorter.py <dir> --library /path/to/library
-
-    # apply: hardlink files into the library (originals stay!)
-    python3 sorter.py <dir> --library /path/to/library --apply
-
-    # widen/narrow fuzzy matching
-    python3 sorter.py <dir> --library ... --threshold 0.5
-
-    # sort a single archive
-    python3 sorter.py "Thunderwolf Cavalry.7z" --library ...
-
-Flags:
-    --catalog FILE   default data/library_catalog.json
-    --audit FILE     default <library>/sorter_applied.jsonl
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import logging
-import sys
-from pathlib import Path
-
-from dlphelper.sorter import (
-    format_plan,
-    load_catalog,
-    sort_plan,
-    apply_plan,
-)
-
-log = logging.getLogger("sorter")
-
-DEFAULT_CATALOG = Path(__file__).resolve().parent / "data" / "gwshop_catalog.json"
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Sort files into the library structure (dry-run by default)."
-    )
-    parser.add_argument("source", type=Path,
-                        help="a directory to sort or a single file")
-    parser.add_argument("--library", type=Path, required=True, metavar="DIR",
-                        help="library root the files will be linked into")
-    parser.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG, metavar="FILE")
-    parser.add_argument("--threshold", type=float, default=0.62,
-                        help="fuzzy match threshold (default 0.62)")
-    parser.add_argument("--apply", action="store_true",
-                        help="create the targets (hardlink, fallback copy)")
-    parser.add_argument("--audit", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--json", action="store_true",
-                        help="print the plan as JSON instead of text")
-    args = parser.parse_args()
-
-    logging.basicConfig(level=logging.INFO,
-                        format="%(asctime)s %(levelname)-7s %(message)s",
-                        datefmt="%H:%M:%S")
-
-    source = args.source
-    files = [source] if source.is_file() else sorted(source.rglob("*"))
-    catalog = load_catalog(args.catalog)
-    audit = args.audit or (args.library / "sorter_applied.jsonl")
-
-    items = sort_plan(files, args.library, catalog, threshold=args.threshold)
-
-    if args.json:
-        print(json.dumps([
-            {
-                "source": str(i.path), "name": i.name, "size": i.size,
-                "system": i.system, "group": i.group, "faction": i.faction,
-                "subfaction": i.subfaction, "class": i.klass, "unit": i.unit,
-                "score": round(i.match_score, 3), "kind": i.match_kind,
-                "inner_count": len(i.inner_names),
-                "target": str(i.target),
-            }
-            for i in items
-        ], indent=2, ensure_ascii=False))
-    else:
-        print(format_plan(items))
-
-    n_catalog = sum(1 for i in items if i.match_kind == "catalog")
-    n_implicit = sum(1 for i in items if i.match_kind == "implicit")
-    n_none = sum(1 for i in items if i.match_kind == "none")
-    log.info("Plan: %d files -> %d catalog matches [+] / %d implicit [~] / "
-             "%d unclassifiable [?]",
-             len(items), n_catalog, n_implicit, n_none)
-
-    if args.apply:
-        linked, copied, skipped = apply_plan(items, audit)
-        log.info("Applied: %d hardlinked, %d copied, %d skipped (audit: %s)",
-                 linked, copied, skipped, audit)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        sys.exit(130)

src/difflib.rs

@@ -0,0 +1,264 @@
+//! Port of CPython's `difflib.SequenceMatcher` (Ratcliff/Obershelp) — the
+//! exact ratio algorithm the python sorter relies on. A different fuzzy
+//! metric (strsim etc.) would produce different scores and silently change
+//! which files land where, so this is a faithful port including
+//! real_quick_ratio/quick_ratio pruning and autojunk.
+//!
+//! Constructed like the python sorter uses it: `SequenceMatcher(None, a, b)`
+//! — no external junk patterns, autojunk enabled (it only matters for
+//! sequences of length >= 200).
+
+/// Longest matching block: (i in a, j in b, size).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct Match {
+    i: usize,
+    j: usize,
+    size: usize,
+}
+
+pub struct SequenceMatcher {
+    /// `a`/`b` as chars, collected once (python works on code points; the
+    /// matching routines index them repeatedly, so collecting here avoids
+    /// re-allocating on every `find_longest_match` call).
+    a_chars: Vec<char>,
+    b_chars: Vec<char>,
+    /// b[j] -> indices (purged of junk and popular elements, like `b2j`).
+    b2j: std::collections::HashMap<char, Vec<usize>>,
+    bjunk: std::collections::HashSet<char>,
+}
+
+impl SequenceMatcher {
+    pub fn new(a: &str, b: &str) -> Self {
+        // __chain_b with isjunk=None: no junk, but autojunk purges
+        // "popular" characters from long sequences (len(b) >= 200).
+        let a_chars: Vec<char> = a.chars().collect();
+        let b_chars: Vec<char> = b.chars().collect();
+        let mut b2j: std::collections::HashMap<char, Vec<usize>> = std::collections::HashMap::new();
+        for (idx, &elt) in b_chars.iter().enumerate() {
+            b2j.entry(elt).or_default().push(idx);
+        }
+        let bjunk = std::collections::HashSet::new(); // isjunk is None
+        let n = b_chars.len();
+        if n >= 200 {
+            let ntest = n / 100 + 1;
+            let mut bpopular = std::collections::HashSet::new();
+            for (&elt, idxs) in &b2j {
+                if idxs.len() > ntest {
+                    bpopular.insert(elt);
+                }
+            }
+            for &elt in &bpopular {
+                b2j.remove(&elt);
+            }
+        }
+        SequenceMatcher {
+            a_chars,
+            b_chars,
+            b2j,
+            bjunk,
+        }
+    }
+
+    fn isbjunk(&self, elt: char) -> bool {
+        self.bjunk.contains(&elt)
+    }
+
+    /// CPython SequenceMatcher.find_longest_match(alo, ahi, blo, bhi) over
+    /// char indices.
+    #[allow(clippy::needless_range_loop)] // `i` feeds the besti/bestj arithmetic
+    fn find_longest_match(&self, alo: usize, ahi: usize, blo: usize, bhi: usize) -> Match {
+        let a_chars = &self.a_chars;
+        let b_chars = &self.b_chars;
+        let nothing: Vec<usize> = Vec::new();
+
+        let mut besti = alo;
+        let mut bestj = blo;
+        let mut bestsize = 0usize;
+        // j2len[j] = length of the longest match ending at (a[i], b[j])
+        let mut j2len: std::collections::HashMap<usize, usize> = std::collections::HashMap::new();
+        for i in alo..ahi {
+            let mut newj2len: std::collections::HashMap<usize, usize> =
+                std::collections::HashMap::new();
+            for &j in self.b2j.get(&a_chars[i]).unwrap_or(&nothing) {
+                if j < blo {
+                    continue;
+                }
+                if j >= bhi {
+                    break;
+                }
+                let k = j2len.get(&(j.wrapping_sub(1))).copied().unwrap_or(0) + 1;
+                newj2len.insert(j, k);
+                if k > bestsize {
+                    besti = i + 1 - k;
+                    bestj = j + 1 - k;
+                    bestsize = k;
+                }
+            }
+            j2len = newj2len;
+        }
+
+        // Extend the best by non-junk elements on each end.
+        while besti > alo
+            && bestj > blo
+            && !self.isbjunk(b_chars[bestj - 1])
+            && a_chars[besti - 1] == b_chars[bestj - 1]
+        {
+            besti -= 1;
+            bestj -= 1;
+            bestsize += 1;
+        }
+        while besti + bestsize < ahi
+            && bestj + bestsize < bhi
+            && !self.isbjunk(b_chars[bestj + bestsize])
+            && a_chars[besti + bestsize] == b_chars[bestj + bestsize]
+        {
+            bestsize += 1;
+        }
+        // Then suck up matching junk on each side of it.
+        while besti > alo
+            && bestj > blo
+            && self.isbjunk(b_chars[bestj - 1])
+            && a_chars[besti - 1] == b_chars[bestj - 1]
+        {
+            besti -= 1;
+            bestj -= 1;
+            bestsize += 1;
+        }
+        while besti + bestsize < ahi
+            && bestj + bestsize < bhi
+            && self.isbjunk(b_chars[bestj + bestsize])
+            && a_chars[besti + bestsize] == b_chars[bestj + bestsize]
+        {
+            bestsize += 1;
+        }
+        Match {
+            i: besti,
+            j: bestj,
+            size: bestsize,
+        }
+    }
+
+    /// All matching blocks, in order (recursive queue algorithm).
+    pub fn get_matching_blocks(&self) -> Vec<Match> {
+        let la = self.a_chars.len();
+        let lb = self.b_chars.len();
+        let mut blocks: Vec<Match> = Vec::new();
+        let mut queue: Vec<(usize, usize, usize, usize)> = vec![(0, la, 0, lb)];
+        while let Some((alo, ahi, blo, bhi)) = queue.pop() {
+            let m = self.find_longest_match(alo, ahi, blo, bhi);
+            if m.size > 0 {
+                blocks.push(m);
+                if alo < m.i && blo < m.j {
+                    queue.push((alo, m.i, blo, m.j));
+                }
+                if m.i + m.size < ahi && m.j + m.size < bhi {
+                    queue.push((m.i + m.size, ahi, m.j + m.size, bhi));
+                }
+            }
+        }
+        blocks.sort_by_key(|m| (m.i, m.j, m.size));
+        blocks
+    }
+
+    fn calculate_ratio(matches: usize, la: usize, lb: usize) -> f64 {
+        let length = la + lb;
+        if length > 0 {
+            2.0 * matches as f64 / length as f64
+        } else {
+            1.0
+        }
+    }
+
+    /// Upper bound computed from the cardinality of the char multiset
+    /// intersection (python `quick_ratio`).
+    pub fn quick_ratio(&self) -> f64 {
+        let mut fullbcount: std::collections::HashMap<char, usize> =
+            std::collections::HashMap::new();
+        for &elt in &self.b_chars {
+            *fullbcount.entry(elt).or_default() += 1;
+        }
+        let mut avail: std::collections::HashMap<char, usize> = std::collections::HashMap::new();
+        let mut matches = 0usize;
+        for &elt in &self.a_chars {
+            let numb = avail
+                .get(&elt)
+                .copied()
+                .unwrap_or_else(|| fullbcount.get(&elt).copied().unwrap_or(0));
+            if numb > 0 {
+                avail.insert(elt, numb - 1);
+                matches += 1;
+            }
+        }
+        Self::calculate_ratio(matches, self.a_chars.len(), self.b_chars.len())
+    }
+
+    /// Trivial upper bound (python `real_quick_ratio`).
+    pub fn real_quick_ratio(&self) -> f64 {
+        let la = self.a_chars.len();
+        let lb = self.b_chars.len();
+        Self::calculate_ratio(la.min(lb), la, lb)
+    }
+
+    /// Ratcliff/Obershelp similarity in [0, 1], exactly as python's
+    /// `SequenceMatcher(None, a, b).ratio()`.
+    pub fn ratio(&self) -> f64 {
+        let matches: usize = self.get_matching_blocks().iter().map(|m| m.size).sum();
+        Self::calculate_ratio(matches, self.a_chars.len(), self.b_chars.len())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Expected values generated with CPython 3.12 difflib (see
+    /// tools/dump_difflib_vectors.py to regenerate).
+    #[rustfmt::skip]
+    const VECTORS: &[( &str, &str, f64, f64, f64 )] = &[
+        // a, b, real_quick_ratio, quick_ratio, ratio
+        ("telemon", "telemon heavy dreadnought", 0.4375, 0.4375, 0.4375),
+        ("aeldari rangers", "dark reapers", 0.8888888888888888, 0.7407407407407407, 0.6666666666666666),
+        ("aeldari rangers", "dwarf rangers", 0.9285714285714286, 0.7857142857142857, 0.7857142857142857),
+        ("guardian weapon team", "guardian squad", 0.8235294117647058, 0.5882352941176471, 0.5882352941176471),
+        ("frontline squad", "guardian squad", 0.9655172413793104, 0.6206896551724138, 0.5517241379310345),
+        ("2e catachany fellaz catachan jungle fighters x", "cadian command squad", 0.6060606060606061, 0.3939393939393939, 0.18181818181818182),
+        ("warp spiders", "warp spiders", 1.0, 1.0, 1.0),
+        ("thunderwolf cavalry", "thunderwolf mount", 0.9444444444444444, 0.6666666666666666, 0.6666666666666666),
+        ("lictor", "necron lieutenant with voidwalker", 0.3076923076923077, 0.3076923076923077, 0.2564102564102564),
+        ("aeldari pack", "harlequin troupe", 0.8571428571428571, 0.5, 0.35714285714285715),
+        ("xyzzy", "warp spiders", 0.5882352941176471, 0.0, 0.0),
+        ("guardian defender", "guardian squad", 0.9032258064516129, 0.6451612903225806, 0.6451612903225806),
+    ];
+
+    #[test]
+    fn matches_cpython_difflib_exactly() {
+        for (a, b, rqr, qr, r) in VECTORS {
+            let sm = SequenceMatcher::new(a, b);
+            assert_eq!(
+                sm.real_quick_ratio(),
+                *rqr,
+                "real_quick_ratio({a:?}, {b:?})"
+            );
+            assert_eq!(sm.quick_ratio(), *qr, "quick_ratio({a:?}, {b:?})");
+            assert_eq!(sm.ratio(), *r, "ratio({a:?}, {b:?})");
+        }
+    }
+
+    #[test]
+    fn empty_and_long_sequences() {
+        assert_eq!(SequenceMatcher::new("", "").ratio(), 1.0);
+        assert_eq!(SequenceMatcher::new("", "abc").ratio(), 0.0);
+        // autojunk path (len(b) >= 200): "unit unit" vs "unit " * 50
+        let long_b = "unit ".repeat(50);
+        let sm = SequenceMatcher::new("unit unit", &long_b);
+        assert_eq!(sm.ratio(), 0.0694980694980695);
+    }
+
+    #[test]
+    fn symmetry_of_ratio_value() {
+        // documented CPython behavior: ratio values are symmetric
+        let ab = SequenceMatcher::new("aeldari rangers", "dwarf rangers").ratio();
+        let ba = SequenceMatcher::new("dwarf rangers", "aeldari rangers").ratio();
+        assert_eq!(ab, ba);
+    }
+}

src/discord.rs

@@ -0,0 +1,516 @@
+//! DCE docker export (JSON) + base64/plain URL extraction — Rust port of
+//! `dlphelper/discord.py`.
+//!
+//! The channels this tool targets encode their download links as base64 inside
+//! message content, and occasionally as plain links. We export the channel as
+//! JSON (richer metadata than the HTML export: message id, timestamp, author),
+//! decode every base64 candidate, and keep only URLs for supported hosts.
+
+use std::collections::HashMap;
+use std::fs;
+use std::io;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+use std::sync::LazyLock;
+
+use anyhow::{Context, Result, bail};
+use base64::Engine as _;
+use base64::engine::general_purpose::STANDARD;
+use regex::Regex;
+use serde_json::Value;
+use url::Url;
+
+pub const DCE_IMAGE: &str = "tyrrrz/discordchatexporter:stable";
+
+// The channels encode links as base64 that always starts with this prefix
+// (https://...); grab.sh established this regex, keep it compatible.
+static B64_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"aHR0cHM6Ly[89][A-Za-z0-9+/_-]*").unwrap());
+static URL_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r#"https://[^\s<>"'\)\]\}]+"#).unwrap());
+
+/// Host key ("gofile" | "0807") -> host names accepted for it.
+pub const SUPPORTED_HOSTS: &[(&str, &[&str])] = &[
+    ("gofile", &["gofile.io", "www.gofile.io"]),
+    ("0807", &["0807.st", "www.0807.st"]),
+];
+
+/// One download URL found in the channel, with source attribution.
+#[derive(Debug, Clone, PartialEq)]
+pub struct FoundUrl {
+    pub url: String,
+    /// "gofile" | "0807" (set after classification)
+    pub host: &'static str,
+    pub message_id: Option<String>,
+    pub timestamp: Option<String>,
+    pub author: Option<String>,
+    /// found via base64 decode vs. plain link
+    pub encoded: bool,
+}
+
+/// Export the channel to out_dir/export.json using the DCE docker image.
+///
+/// The token is passed to the container via the DISCORD_TOKEN env var (DCE
+/// reads it natively) so it never appears on the docker command line. TOKEN
+/// is set as well for DCE versions that only know that name.
+pub fn export_channel(
+    channel_id: &str,
+    out_dir: &Path,
+    token: &str,
+    force: bool,
+) -> Result<PathBuf> {
+    let dest = out_dir.join("export.json");
+    if dest.exists() && !force {
+        log::info!(
+            "Reusing existing export {} (pass --fresh to re-export)",
+            dest.display()
+        );
+        return Ok(dest);
+    }
+
+    fs::create_dir_all(out_dir)?;
+    // Python Path.resolve() on an existing dir == canonicalize here (the
+    // create_dir_all above guarantees it exists).
+    let resolved = fs::canonicalize(out_dir)?;
+    let (uid, gid) = read_uid_gid()?;
+
+    log::info!(
+        "Exporting channel {} via DiscordChatExporter ...",
+        channel_id
+    );
+    // stdout/stderr are inherited, like the python subprocess.run(check=True).
+    let status = Command::new("docker")
+        .args([
+            "run",
+            "--rm",
+            "--user",
+            &format!("{uid}:{gid}"),
+            "-v",
+            &format!("{}:/out", resolved.display()),
+            "-e",
+            "DISCORD_TOKEN",
+            "-e",
+            "TOKEN",
+        ])
+        .arg(DCE_IMAGE)
+        .args([
+            "export",
+            "-c",
+            channel_id,
+            "-f",
+            "Json",
+            "-o",
+            "/out/export.json",
+        ])
+        // The token only travels via the environment, never on the argv.
+        .env("DISCORD_TOKEN", token)
+        .env("TOKEN", token)
+        .status();
+    match status {
+        Err(e) if e.kind() == io::ErrorKind::NotFound => {
+            bail!("docker not found - the Discord export needs the docker image {DCE_IMAGE}")
+        }
+        Err(e) => return Err(e).context("failed to spawn docker"),
+        Ok(s) if !s.success() => bail!(
+            "DiscordChatExporter failed with exit code {} (check the token and the channel id)",
+            s.code().unwrap_or(-1)
+        ),
+        Ok(_) => {}
+    }
+    if !dest.exists() {
+        bail!("DiscordChatExporter produced no output file");
+    }
+    Ok(dest)
+}
+
+/// Real uid/gid of this process. Linux only: parsed from /proc/self/status
+/// because std has no getuid (the first number on the Uid:/Gid: line is the
+/// real id, matching python's os.getuid()/os.getgid()).
+fn read_uid_gid() -> Result<(u32, u32)> {
+    let status = fs::read_to_string("/proc/self/status")
+        .context("cannot read /proc/self/status to determine uid/gid for docker --user")?;
+    let uid = parse_proc_id(&status, "Uid:")?;
+    let gid = parse_proc_id(&status, "Gid:")?;
+    Ok((uid, gid))
+}
+
+fn parse_proc_id(status_text: &str, key: &str) -> Result<u32> {
+    for line in status_text.lines() {
+        if let Some(rest) = line.strip_prefix(key) {
+            let first = rest
+                .split_whitespace()
+                .next()
+                .with_context(|| format!("empty {key} line in /proc/self/status"))?;
+            return first.parse().with_context(|| {
+                format!("cannot parse {key} value {first:?} in /proc/self/status")
+            });
+        }
+    }
+    bail!("{key} line not found in /proc/self/status")
+}
+
+/// Return the host key ('gofile' | '0807') for a URL, else None.
+///
+/// Mirrors python `urlparse(url).hostname.lower()`: unparseable URLs and URLs
+/// without a hostname yield None. The url crate normalizes the host to
+/// lowercase, same as urlparse does.
+pub fn classify(url: &str) -> Option<&'static str> {
+    let parsed = Url::parse(url).ok()?;
+    let hostname = parsed.host_str()?;
+    SUPPORTED_HOSTS
+        .iter()
+        .find(|(_, hosts)| hosts.contains(&hostname))
+        .map(|(key, _)| *key)
+}
+
+/// Decode one base64 candidate into the URL it contains, or None.
+///
+/// Python quirk kept: "_" is remapped to "/" and "-" to "+" (in that order),
+/// trailing "=" stripped, then padded back to a multiple of 4 before a strict
+/// standard-alphabet decode and a strict UTF-8 check. One encoded blob can
+/// contain the URL plus padding/newlines: whitespace is trimmed and only the
+/// first line is returned.
+fn decode_b64(candidate: &str) -> Option<String> {
+    let mut s: String = candidate
+        .chars()
+        .map(|c| match c {
+            '_' => '/',
+            '-' => '+',
+            other => other,
+        })
+        .collect();
+    while s.ends_with('=') {
+        s.pop();
+    }
+    for _ in 0..(4 - s.len() % 4) % 4 {
+        s.push('=');
+    }
+    let decoded = STANDARD.decode(s.as_bytes()).ok()?;
+    let text = String::from_utf8(decoded).ok()?;
+    // strip whitespace, keep the first line ("" -> None)
+    text.trim()
+        .lines()
+        .next()
+        .filter(|l| !l.is_empty())
+        .map(String::from)
+}
+
+/// Python `str(x or "")` for a JSON value: falsy values (null, false, 0, "",
+/// empty array/object) become the empty string, strings pass through, and any
+/// other value is stringified. DCE content/id fields are always strings; the
+/// other shapes only exist for parity with the python code.
+fn python_str_or_empty(value: Option<&Value>) -> String {
+    let falsy = match value {
+        None | Some(Value::Null) => true,
+        Some(Value::Bool(b)) => !b,
+        Some(Value::Number(n)) => n.as_f64().is_some_and(|f| f == 0.0),
+        Some(Value::String(s)) => s.is_empty(),
+        Some(Value::Array(a)) => a.is_empty(),
+        Some(Value::Object(o)) => o.is_empty(),
+    };
+    if falsy {
+        return String::new();
+    }
+    match value.unwrap() {
+        Value::String(s) => s.clone(),
+        Value::Number(n) => n.to_string(),
+        Value::Bool(_) => "True".to_string(), // python str(True)
+        other => other.to_string(),
+    }
+}
+
+/// Extract supported URLs from a DCE JSON export, deduplicated.
+///
+/// The root may be a list of messages or an object with a "messages" array;
+/// anything else yields no URLs. The first occurrence of a URL wins
+/// (python dict.setdefault semantics).
+pub fn extract_urls(export_path: &Path) -> Result<Vec<FoundUrl>> {
+    let text = fs::read_to_string(export_path)?;
+    let data: Value = serde_json::from_str(&text)?;
+    let empty: Vec<Value> = Vec::new();
+    let messages: &[Value] = match &data {
+        Value::Array(list) => list,
+        Value::Object(obj) => obj
+            .get("messages")
+            .and_then(Value::as_array)
+            .map(|a| a.as_slice())
+            .unwrap_or(&empty),
+        _ => &empty,
+    };
+
+    let mut found: HashMap<String, FoundUrl> = HashMap::new();
+    let mut stat_b64 = 0u64;
+    let mut stat_plain = 0u64;
+    let mut stat_unsupported = 0u64;
+
+    for msg in messages {
+        let content = python_str_or_empty(msg.get("content"));
+        if content.is_empty() {
+            continue;
+        }
+        let author = msg
+            .get("author")
+            .and_then(|a| a.get("name"))
+            .and_then(Value::as_str)
+            .map(str::to_owned);
+        let message_id = {
+            let s = python_str_or_empty(msg.get("id"));
+            (!s.is_empty()).then_some(s)
+        };
+        let timestamp = msg
+            .get("timestamp")
+            .and_then(Value::as_str)
+            .map(str::to_owned);
+
+        for m in B64_RE.find_iter(&content) {
+            stat_b64 += 1;
+            let Some(decoded) = decode_b64(m.as_str()) else {
+                continue;
+            };
+            let Some(host) = classify(&decoded) else {
+                stat_unsupported += 1;
+                continue;
+            };
+            found.entry(decoded.clone()).or_insert(FoundUrl {
+                url: decoded,
+                host,
+                message_id: message_id.clone(),
+                timestamp: timestamp.clone(),
+                author: author.clone(),
+                encoded: true,
+            });
+        }
+
+        for m in URL_RE.find_iter(&content) {
+            let url = m
+                .as_str()
+                .trim_end_matches(['.', ',', ';', ')'])
+                .to_string();
+            let Some(host) = classify(&url) else {
+                continue;
+            };
+            stat_plain += 1;
+            found.entry(url.clone()).or_insert(FoundUrl {
+                url,
+                host,
+                message_id: message_id.clone(),
+                timestamp: timestamp.clone(),
+                author: author.clone(),
+                encoded: false,
+            });
+        }
+    }
+
+    log::info!(
+        "Extraction: {} base64 candidates, {} plain links, {} unsupported/undecodable, {} unique supported URLs",
+        stat_b64,
+        stat_plain,
+        stat_unsupported,
+        found.len()
+    );
+    let mut out: Vec<FoundUrl> = found.into_values().collect();
+    out.sort_by(|a, b| a.url.cmp(&b.url));
+    Ok(out)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use base64::engine::general_purpose::URL_SAFE_NO_PAD;
+    use serde_json::json;
+    use tempfile::tempdir;
+
+    fn b64(s: &str) -> String {
+        STANDARD.encode(s.as_bytes())
+    }
+
+    /// The two-message fixture from tests/test_extract.py.
+    fn fixture_messages() -> Value {
+        json!([
+            {
+                "id": "1",
+                "timestamp": "2026-01-01T00:00:00",
+                "author": {"name": "poster"},
+                "content": format!("chaos stuff: {}", b64("https://gofile.io/d/abc123")),
+            },
+            {
+                "id": "2",
+                "timestamp": "2026-01-02T00:00:00",
+                "author": {"name": "poster"},
+                "content": format!(
+                    "0807 zip: {}\ndupe of 1: {}\nplain: https://0807.st/plainLink.zip and \
+                     https://discord.com/channels/1/2 (unsupported)\njunk: {} and \
+                     not-our-domain base64: aHR0cHM6Ly9leGFtcGxlLm9yZw==",
+                    b64("https://0807.st/6saCNie.zip"),
+                    b64("https://gofile.io/d/abc123"),
+                    b64("https://pixeldrain.com/u/xyz"),
+                ),
+            },
+        ])
+    }
+
+    fn write_export(dir: &Path, root: Value) -> PathBuf {
+        let path = dir.join("export.json");
+        fs::write(&path, serde_json::to_string(&root).unwrap()).unwrap();
+        path
+    }
+
+    #[test]
+    fn extract_urls_matches_python_fixture() {
+        let dir = tempdir().unwrap();
+        let export = write_export(dir.path(), json!({"messages": fixture_messages()}));
+        let found = extract_urls(&export).unwrap();
+
+        let urls: Vec<&str> = found.iter().map(|f| f.url.as_str()).collect();
+        assert_eq!(
+            urls,
+            vec![
+                "https://0807.st/6saCNie.zip",
+                "https://0807.st/plainLink.zip",
+                "https://gofile.io/d/abc123",
+            ]
+        );
+
+        let g = found
+            .iter()
+            .find(|f| f.url == "https://gofile.io/d/abc123")
+            .unwrap();
+        assert_eq!(g.host, "gofile");
+        assert_eq!(g.message_id.as_deref(), Some("1"));
+        assert_eq!(g.timestamp.as_deref(), Some("2026-01-01T00:00:00"));
+        assert_eq!(g.author.as_deref(), Some("poster"));
+        assert!(g.encoded);
+
+        let z = found
+            .iter()
+            .find(|f| f.url == "https://0807.st/6saCNie.zip")
+            .unwrap();
+        assert_eq!(z.host, "0807");
+        assert_eq!(z.message_id.as_deref(), Some("2"));
+        assert!(z.encoded);
+
+        let p = found
+            .iter()
+            .find(|f| f.url == "https://0807.st/plainLink.zip")
+            .unwrap();
+        assert_eq!(p.host, "0807");
+        assert_eq!(p.message_id.as_deref(), Some("2"));
+        assert_eq!(p.author.as_deref(), Some("poster"));
+        assert!(!p.encoded);
+    }
+
+    #[test]
+    fn extract_urls_accepts_bare_list_root() {
+        let dir = tempdir().unwrap();
+        let export = write_export(dir.path(), fixture_messages());
+        let found = extract_urls(&export).unwrap();
+        assert_eq!(found.len(), 3);
+    }
+
+    #[test]
+    fn extract_urls_empty_and_malformed_roots_yield_nothing() {
+        let dir = tempdir().unwrap();
+        assert!(
+            extract_urls(&write_export(dir.path(), json!({})))
+                .unwrap()
+                .is_empty()
+        );
+        assert!(
+            extract_urls(&write_export(dir.path(), json!({"other": []})))
+                .unwrap()
+                .is_empty()
+        );
+        // message without content is skipped
+        let export = write_export(dir.path(), json!([{"id": "7"}]));
+        assert!(extract_urls(&export).unwrap().is_empty());
+    }
+
+    #[test]
+    fn classify_matches_python_hostnames() {
+        assert_eq!(classify("https://0807.st/x.zip"), Some("0807"));
+        assert_eq!(classify("https://www.0807.st/x.zip"), Some("0807"));
+        assert_eq!(classify("https://gofile.io/d/x"), Some("gofile"));
+        assert_eq!(classify("https://www.gofile.io/d/x"), Some("gofile"));
+        assert_eq!(classify("https://example.org/x"), None);
+        // url crate lowercases hosts, like urlparse().hostname.lower()
+        assert_eq!(classify("https://GOFILE.IO/d/x"), Some("gofile"));
+        // edge cases: no scheme, empty string, bare host
+        assert_eq!(classify("0807.st/x.zip"), None);
+        assert_eq!(classify(""), None);
+        assert_eq!(classify("https://"), None);
+    }
+
+    #[test]
+    fn decode_b64_handles_padding_and_rstrip() {
+        // candidate as matched by B64_RE carries no '=': padding is added
+        let unpadded = b64("https://gofile.io/d/abc123")
+            .trim_end_matches('=')
+            .to_string();
+        assert_eq!(
+            decode_b64(&unpadded),
+            Some("https://gofile.io/d/abc123".to_string())
+        );
+        // trailing '=' is stripped before re-padding
+        assert_eq!(
+            decode_b64("aHR0cHM6Ly9leGFtcGxlLm9yZw=="),
+            Some("https://example.org".to_string())
+        );
+    }
+
+    #[test]
+    fn decode_b64_remaps_urlsafe_alphabet() {
+        // U+13C0 encodes to 4Y+A (standard) / 4Y-A (urlsafe); the python
+        // quirk remaps '-' back to '+' before decoding. ('_' -> '/' can never
+        // occur in a candidate that decodes to valid UTF-8, so '-' covers the
+        // remap path.)
+        let candidate = URL_SAFE_NO_PAD.encode("\u{13C0}");
+        assert!(candidate.contains('-'), "expected '-' in {candidate}");
+        assert_eq!(decode_b64(&candidate), Some("\u{13C0}".to_string()));
+    }
+
+    #[test]
+    fn decode_b64_keeps_first_line_after_whitespace() {
+        let candidate = STANDARD.encode("https://gofile.io/d/abc123\nnoise  ");
+        assert_eq!(
+            decode_b64(&candidate),
+            Some("https://gofile.io/d/abc123".to_string())
+        );
+    }
+
+    #[test]
+    fn decode_b64_rejects_invalid_utf8_and_garbage() {
+        // decodes to \xff\xfe — not valid UTF-8
+        assert_eq!(decode_b64("//8+"), None);
+        assert_eq!(decode_b64("aHR0cHM6Ly["), None);
+        assert_eq!(decode_b64(""), None);
+    }
+
+    #[test]
+    fn parse_proc_id_reads_first_number() {
+        let text = "Name:\tx\nUid:\t1000\t1000\t1000\t1000\nGid:\t989\t989\t989\t989\n";
+        assert_eq!(parse_proc_id(text, "Uid:").unwrap(), 1000);
+        assert_eq!(parse_proc_id(text, "Gid:").unwrap(), 989);
+        assert!(parse_proc_id("Name:\tx\n", "Uid:").is_err());
+    }
+
+    #[test]
+    fn read_uid_gid_matches_proc_self() {
+        // Linux-only binary (docker export); /proc/self/status must exist.
+        let (uid, gid) = read_uid_gid().unwrap();
+        let text = fs::read_to_string("/proc/self/status").unwrap();
+        let want_uid = parse_proc_id(&text, "Uid:").unwrap();
+        let want_gid = parse_proc_id(&text, "Gid:").unwrap();
+        assert_eq!((uid, gid), (want_uid, want_gid));
+    }
+
+    #[test]
+    fn export_channel_reuses_existing_export_without_docker() {
+        let dir = tempdir().unwrap();
+        let dest = dir.path().join("export.json");
+        fs::write(&dest, "[]").unwrap();
+        // force=false must short-circuit before any docker invocation; the
+        // token must not end up in the error/log path either.
+        let got = export_channel("123", dir.path(), "s3cret-token", false).unwrap();
+        assert_eq!(got, dest);
+    }
+}

src/globalstore.rs

@@ -0,0 +1,199 @@
+//! Cross-run, cross-channel record of completed URLs — Rust port of
+//! `dlphelper/globalstore.py`.
+//!
+//! state.json only knows about one OUT_DIR; this store (default
+//! `~/.local/share/dlphelper/urls.json`) records every successfully
+//! downloaded URL once, forever, regardless of OUT_DIR — so the same gofile
+//! link posted in a second channel is not downloaded again.
+
+use std::collections::BTreeMap;
+use std::fs;
+use std::path::PathBuf;
+
+use log::warn;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+use crate::util::{now_local_iso, save_sorted};
+
+/// Per-URL record in urls.json: `{"status", "updated", "channel", "host",
+/// "file_count", ...}`. Only fields actually set are written (the python
+/// version never writes `attempts`/`files` here).
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct GlobalRecord {
+    #[serde(default)]
+    pub status: String,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub updated: Option<String>,
+    /// channel / host / file_count and any unknown fields.
+    #[serde(flatten, default)]
+    pub extra: BTreeMap<String, Value>,
+}
+
+impl GlobalRecord {
+    pub fn is_done(&self) -> bool {
+        self.status == "ok"
+    }
+}
+
+/// Default location: `$XDG_DATA_HOME/dlphelper/urls.json`, else
+/// `~/.local/share/dlphelper/urls.json`. An empty XDG_DATA_HOME falls back to
+/// home, matching the python version.
+pub fn default_global_path() -> Option<PathBuf> {
+    if let Ok(xdg) = std::env::var("XDG_DATA_HOME")
+        && !xdg.is_empty()
+    {
+        return Some(PathBuf::from(xdg).join("dlphelper").join("urls.json"));
+    }
+    dirs::home_dir().map(|home| {
+        home.join(".local")
+            .join("share")
+            .join("dlphelper")
+            .join("urls.json")
+    })
+}
+
+/// JSON file mapping url -> record. A `None` path disables the store
+/// (all writes become no-ops), like passing `gstore=None` in python.
+pub struct GlobalStore {
+    pub path: Option<PathBuf>,
+    pub records: BTreeMap<String, GlobalRecord>,
+}
+
+impl GlobalStore {
+    pub fn disabled() -> Self {
+        Self {
+            path: None,
+            records: BTreeMap::new(),
+        }
+    }
+
+    /// Load urls.json; a corrupt or unreadable file is moved to `*.json.bak`
+    /// and we start fresh (same behavior as the python version).
+    pub fn load(path: impl Into<PathBuf>) -> Self {
+        let path = path.into();
+        if path.exists() {
+            let parsed = fs::read_to_string(&path).ok().and_then(|text| {
+                serde_json::from_str::<BTreeMap<String, GlobalRecord>>(&text).ok()
+            });
+            if let Some(records) = parsed {
+                return Self {
+                    path: Some(path),
+                    records,
+                };
+            }
+            let backup = path.with_extension("json.bak");
+            warn!(
+                "Corrupt global store {} ; backed up to {}",
+                path.display(),
+                backup.display()
+            );
+            let _ = fs::rename(&path, &backup);
+        }
+        Self {
+            path: Some(path),
+            records: BTreeMap::new(),
+        }
+    }
+
+    pub fn is_done(&self, url: &str) -> bool {
+        self.records.get(url).is_some_and(GlobalRecord::is_done)
+    }
+
+    /// Record a URL as done (plus attribution fields when known), then save.
+    /// No-ops when the store is disabled. `note` mirrors the python `note=`
+    /// kwarg (e.g. "assumed done via --assume-done") and lands in `extra`.
+    pub fn mark(
+        &mut self,
+        url: &str,
+        channel: Option<&str>,
+        host: Option<&str>,
+        file_count: Option<u64>,
+        note: Option<&str>,
+    ) -> anyhow::Result<()> {
+        if self.path.is_none() {
+            return Ok(());
+        }
+        let rec = self.records.entry(url.to_string()).or_default();
+        rec.status = "ok".to_string();
+        if let Some(c) = channel {
+            rec.extra.insert("channel".into(), Value::from(c));
+        }
+        if let Some(h) = host {
+            rec.extra.insert("host".into(), Value::from(h));
+        }
+        if let Some(n) = file_count {
+            rec.extra.insert("file_count".into(), Value::from(n));
+        }
+        if let Some(note) = note {
+            rec.extra.insert("note".into(), Value::from(note));
+        }
+        rec.updated = Some(now_local_iso());
+        self.save()
+    }
+
+    pub fn save(&self) -> anyhow::Result<()> {
+        match &self.path {
+            Some(path) => save_sorted(path, &self.records),
+            None => Ok(()),
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::fs;
+
+    #[test]
+    fn disabled_store_is_a_noop() {
+        let mut gs = GlobalStore::disabled();
+        gs.mark("u", Some("c"), Some("gofile"), Some(2), None)
+            .unwrap();
+        assert!(gs.records.is_empty());
+    }
+
+    #[test]
+    fn mark_persists_and_reloads() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("urls.json");
+        {
+            let mut gs = GlobalStore::load(&path);
+            gs.mark("u1", Some("chan"), Some("gofile"), Some(3), None)
+                .unwrap();
+            gs.mark("u2", None, None, None, None).unwrap();
+        }
+        let gs = GlobalStore::load(&path);
+        assert!(gs.is_done("u1"));
+        assert!(gs.is_done("u2"));
+        assert!(!gs.is_done("u3"));
+
+        let v: serde_json::Value =
+            serde_json::from_str(&fs::read_to_string(&path).unwrap()).unwrap();
+        assert_eq!(v["u1"]["status"], "ok");
+        assert_eq!(v["u1"]["channel"], "chan");
+        assert_eq!(v["u1"]["host"], "gofile");
+        assert_eq!(v["u1"]["file_count"], 3);
+        // no attempts/files keys here — that's state.json's job
+        assert!(v["u1"].get("attempts").is_none());
+        assert!(v["u2"].get("channel").is_none());
+    }
+
+    #[test]
+    fn corrupt_store_is_backed_up() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("urls.json");
+        fs::write(&path, "{not json").unwrap();
+        let gs = GlobalStore::load(&path);
+        assert!(gs.records.is_empty());
+        assert!(dir.path().join("urls.json.bak").exists());
+    }
+
+    #[test]
+    fn default_path_honors_empty_xdg_fallback() {
+        // Only assert shape; don't touch the process env (tests run in parallel).
+        if default_global_path().is_none() {
+            panic!("home dir should be discoverable in tests");
+        }
+    }
+}

src/gofile.rs

@@ -0,0 +1,675 @@
+//! GoFile API client (one guest account per run) — Rust port of
+//! `dlphelper/gofile.py`.
+//!
+//! The API interaction mirrors Lysagxra/GoFileDownloader (same headers,
+//! website-token scheme and endpoints), with the same two deliberate
+//! differences as the python version:
+//!
+//! 1. A single guest account is created per run (GoFileDownloader creates one
+//!    per URL, which trips Gofile's rate limit after ~6 requests).
+//! 2. Failures return errors instead of exiting, so the orchestrator can
+//!    track them per file/URL and keep going.
+//!
+//! Retry semantics (see python AGENTS.md §6): `NotFound`, `Empty` and
+//! `PasswordRequired` are never retried by the orchestrator; `AuthError`
+//! allows one re-auth (`reset()` + `ensure_token()`).
+
+use std::fmt;
+use std::fs;
+use std::io::{Read, Write};
+use std::path::Path;
+use std::sync::OnceLock;
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+use hex::encode as hex_encode;
+use reqwest::blocking::Client;
+use reqwest::header::{AUTHORIZATION, COOKIE, HeaderMap, HeaderName, HeaderValue, ORIGIN, REFERER};
+use sha2::{Digest, Sha256};
+
+pub const API: &str = "https://api.gofile.io";
+pub const USER_AGENT: &str = "Mozilla/5.0";
+pub const LOCALE: &str = "en-US";
+/// Public website-token seed, see GoFileDownloader. Fine to hardcode; the
+/// resulting tokens must never be logged (AGENTS.md §6).
+pub const TOKEN_SECRET: &str = "12af056dacea0b";
+pub const TOKEN_WINDOW_SEC: u64 = 14400;
+pub const CHUNK_SIZE: usize = 1024 * 1024;
+
+const COMMON_HEADER_PAIRS: &[(&str, &str)] = &[
+    ("User-Agent", USER_AGENT),
+    ("Accept-Encoding", "gzip"),
+    ("Accept", "*/*"),
+    ("Connection", "keep-alive"),
+];
+
+const EXTENDED_HEADER_PAIRS: &[(&str, &str)] = &[
+    ("User-Agent", USER_AGENT),
+    ("Accept-Encoding", "gzip"),
+    ("Accept", "*/*"),
+    ("Connection", "keep-alive"),
+    ("Sec-Fetch-Dest", "empty"),
+    ("Sec-Fetch-Mode", "cors"),
+    ("Sec-Fetch-Site", "same-site"),
+    ("Pragma", "no-cache"),
+    ("Cache-Control", "no-cache"),
+];
+
+/// Common headers sent to every endpoint (python `COMMON_HEADERS`).
+fn common_headers() -> HeaderMap {
+    static M: OnceLock<HeaderMap> = OnceLock::new();
+    M.get_or_init(|| build_header_map(COMMON_HEADER_PAIRS))
+        .clone()
+}
+
+/// Common + fetch/cache-control headers (python `EXTENDED_HEADERS`).
+fn extended_headers() -> HeaderMap {
+    static M: OnceLock<HeaderMap> = OnceLock::new();
+    M.get_or_init(|| build_header_map(EXTENDED_HEADER_PAIRS))
+        .clone()
+}
+
+fn build_header_map(pairs: &'static [(&'static str, &'static str)]) -> HeaderMap {
+    let mut map = HeaderMap::new();
+    for (name, value) in pairs {
+        let name: HeaderName = name
+            .parse()
+            .unwrap_or_else(|_| panic!("static header name {name:?} is invalid"));
+        let value = HeaderValue::from_static(value);
+        map.insert(name, value);
+    }
+    map
+}
+
+/// Insert a dynamically computed header. Unlike python (which would raise on
+/// an invalid value and let the orchestrator's catch-all handle it), a parse
+/// failure here logs a warning and skips the header — the request then fails
+/// server-side and is recorded, and the run keeps going.
+fn set_header(map: &mut HeaderMap, name: HeaderName, value: &str) {
+    match HeaderValue::from_str(value) {
+        Ok(v) => {
+            map.insert(name, v);
+        }
+        Err(_) => {
+            log::warn!("skipping invalid {} header value", name.as_str());
+        }
+    }
+}
+
+/// Error type mirroring the python exception classes. The orchestrator maps
+/// variants to retry policies: `NotFound`/`Empty`/`PasswordRequired` are
+/// final states, `AuthError` permits one re-auth, `Error` is retryable.
+#[allow(dead_code)] // variants reserved for the orchestrator (e.g. Empty)
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum GoFileError {
+    /// python `GoFileError` (base): transient/unclassified failure.
+    Error(String),
+    /// python `GoFilePasswordRequired`.
+    PasswordRequired(String),
+    /// python `GoFileNotFound`: dead link, never retry.
+    NotFound(String),
+    /// python `GoFileEmpty`: content exists but contains no files.
+    Empty(String),
+    /// python `GoFileAuthError`: token rejected, one re-auth allowed.
+    AuthError(String),
+}
+
+impl fmt::Display for GoFileError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            GoFileError::Error(m)
+            | GoFileError::PasswordRequired(m)
+            | GoFileError::NotFound(m)
+            | GoFileError::Empty(m)
+            | GoFileError::AuthError(m) => write!(f, "{m}"),
+        }
+    }
+}
+
+impl std::error::Error for GoFileError {}
+
+/// Extract the content id from a gofile.io/d/<id> URL.
+///
+/// Matches the lenient python `urlparse` behavior: a URL without a scheme
+/// still works (`"gofile.io/d/x"` → `"x"`); only the path is inspected.
+pub fn content_id_from_url(url: &str) -> Result<String, GoFileError> {
+    let path = match url::Url::parse(url) {
+        Ok(u) => u.path().to_string(),
+        // python urlparse is lenient about a missing scheme but still strips
+        // query/fragment; mirror that ("gofile.io/d/x?y=1" → path as parsed).
+        Err(_) => url.split(['?', '#']).next().unwrap_or(url).to_string(),
+    };
+    let parts: Vec<&str> = path.split('/').filter(|p| !p.is_empty()).collect();
+    if parts.len() >= 2 && parts[parts.len() - 2] == "d" {
+        return Ok(parts[parts.len() - 1].to_string());
+    }
+    Err(GoFileError::Error(format!(
+        "not a gofile content URL (expected gofile.io/d/<id>): {url}"
+    )))
+}
+
+/// sha256 over `bytes` as lowercase hex (python `hashlib.sha256(...).hexdigest()`).
+fn sha256_hex(bytes: &[u8]) -> String {
+    hex_encode(Sha256::digest(bytes))
+}
+
+/// Truncate to the first 200 chars (python `text[:200]` counts codepoints).
+fn truncate_200(s: &str) -> String {
+    s.chars().take(200).collect()
+}
+
+/// GoFile API client that reuses one guest account token for the whole run.
+pub struct GoFileClient {
+    password_hash: Option<String>,
+    token: Option<String>,
+    session: Client,
+}
+
+impl GoFileClient {
+    /// `password` is hashed (sha256 hex) once; the API wants the hash.
+    pub fn new(password: Option<&str>) -> Self {
+        let password_hash = password.map(|p| sha256_hex(p.as_bytes()));
+        // Client-level timeouts mirror python's download timeout tuple
+        // (connect 10s); the read-side 60s from python's (10, 60) is not
+        // expressible in reqwest blocking, so the download request gets a
+        // generous total timeout instead (see download_file).
+        let session = Client::builder()
+            .default_headers(common_headers())
+            .connect_timeout(Duration::from_secs(10))
+            .build()
+            .unwrap_or_else(|e| panic!("failed to build HTTP client: {e}"));
+        Self {
+            password_hash,
+            token: None,
+            session,
+        }
+    }
+
+    /// Create a guest account; backs off and retries on rate limiting.
+    /// Defaults match python: 8 attempts, 20s × attempt between them.
+    pub fn ensure_token(&mut self) -> Result<String, GoFileError> {
+        self.ensure_token_with(8, 20)
+    }
+
+    /// Same with explicit attempt/wait parameters (python keyword args).
+    pub fn ensure_token_with(
+        &mut self,
+        max_attempts: u32,
+        wait_seconds: u64,
+    ) -> Result<String, GoFileError> {
+        if let Some(token) = &self.token {
+            return Ok(token.clone());
+        }
+
+        let mut last_status = "unknown".to_string();
+        for attempt in 1..=max_attempts {
+            match self.try_create_account() {
+                Ok(token) => {
+                    self.token = Some(token.clone());
+                    log::info!("GoFile guest account created (token reused for whole run)");
+                    return Ok(token);
+                }
+                Err(status) => {
+                    last_status = status;
+                    log::warn!(
+                        "GoFile account creation attempt {}/{} failed: {}",
+                        attempt,
+                        max_attempts,
+                        last_status
+                    );
+                }
+            }
+            if attempt < max_attempts {
+                std::thread::sleep(Duration::from_secs(wait_seconds * u64::from(attempt)));
+            }
+        }
+        Err(GoFileError::Error(format!(
+            "GoFile account creation failed after {max_attempts} attempts (last: {last_status})"
+        )))
+    }
+
+    /// POST /accounts with the common headers; returns the token or the
+    /// status string that describes the failure (python's `last_status`).
+    fn try_create_account(&self) -> Result<String, String> {
+        let resp = self
+            .session
+            .post(format!("{API}/accounts"))
+            .headers(common_headers())
+            .timeout(Duration::from_secs(15))
+            .send();
+        let data: serde_json::Value = match resp {
+            Ok(r) => match r.json() {
+                Ok(v) => v,
+                Err(e) => return Err(format!("request error: {e}")),
+            },
+            Err(e) => return Err(format!("request error: {e}")),
+        };
+        if data.get("status").and_then(|s| s.as_str()) == Some("ok") {
+            return data
+                .pointer("/data/token")
+                .and_then(|t| t.as_str())
+                .map(str::to_string)
+                .ok_or_else(|| "request error: missing data.token".to_string());
+        }
+        // python: str(data.get("status", data)[:200] if data else data)
+        let status = match data.get("status") {
+            Some(serde_json::Value::String(s)) => s.clone(),
+            Some(other) => other.to_string(),
+            None => serde_json::to_string(&data).unwrap_or_default(),
+        };
+        Err(truncate_200(&status))
+    }
+
+    /// Drop the current token so the next `ensure_token()` creates a new one.
+    pub fn reset(&mut self) {
+        self.token = None;
+    }
+
+    /// sha256 over `"{UA}::{LOCALE}::{token}::{window}::{TOKEN_SECRET}"`.
+    /// `window` is an argument so tests can pin it.
+    fn website_token_for(&self, window: u64) -> String {
+        // python f-string renders a missing token as "None".
+        let token = self.token.as_deref().unwrap_or("None");
+        let seed = format!("{USER_AGENT}::{LOCALE}::{token}::{window}::{TOKEN_SECRET}");
+        sha256_hex(seed.as_bytes())
+    }
+
+    /// Website token for the current 4h window (python `_website_token`).
+    fn website_token(&self) -> String {
+        let window = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .map(|d| d.as_secs())
+            .unwrap_or(0)
+            / TOKEN_WINDOW_SEC;
+        self.website_token_for(window)
+    }
+
+    /// EXTENDED_HEADERS + Authorization, X-Website-Token, X-BL.
+    fn auth_headers(&self) -> HeaderMap {
+        let mut headers = extended_headers();
+        let bearer = format!("Bearer {}", self.token.as_deref().unwrap_or("None"));
+        set_header(&mut headers, AUTHORIZATION, &bearer);
+        set_header(
+            &mut headers,
+            header_name("x-website-token"),
+            &self.website_token(),
+        );
+        set_header(&mut headers, header_name("x-bl"), LOCALE);
+        headers
+    }
+
+    /// Return (relpath, filename, download_link) for every file under a URL.
+    ///
+    /// Direct files at the content root get their own folder named after the
+    /// content id (empty relpath → cid, like the python normalization).
+    pub fn list_files(&self, url: &str) -> Result<Vec<(String, String, String)>, GoFileError> {
+        let cid = content_id_from_url(url)?;
+        let mut results = Vec::new();
+        self.walk(&cid, &[], &mut results)?;
+        Ok(results
+            .into_iter()
+            .map(|(relpath, name, link)| {
+                let relpath = if relpath.is_empty() {
+                    cid.clone()
+                } else {
+                    relpath
+                };
+                (relpath, name, link)
+            })
+            .collect())
+    }
+
+    /// Recursive walk of /contents/<id>, mirroring python `_walk`.
+    fn walk(
+        &self,
+        cid: &str,
+        prefix: &[String],
+        results: &mut Vec<(String, String, String)>,
+    ) -> Result<(), GoFileError> {
+        let mut query = String::from("?cache=true&sortField=createTime&sortDirection=1");
+        if let Some(password_hash) = &self.password_hash {
+            query.push_str("&password=");
+            query.push_str(password_hash);
+        }
+        let resp = self
+            .session
+            .get(format!("{API}/contents/{cid}{query}"))
+            .headers(self.auth_headers())
+            .timeout(Duration::from_secs(30))
+            .send()
+            .map_err(|e| GoFileError::Error(format!("content request for {cid} failed: {e}")))?;
+
+        if resp.status() == reqwest::StatusCode::NOT_FOUND {
+            return Err(GoFileError::NotFound(format!(
+                "content {cid} not found (dead link)"
+            )));
+        }
+        let status_code = resp.status();
+        if status_code == reqwest::StatusCode::UNAUTHORIZED
+            || status_code == reqwest::StatusCode::FORBIDDEN
+        {
+            return Err(GoFileError::AuthError(format!(
+                "content request denied ({status_code}) for {cid}"
+            )));
+        }
+        let text = resp
+            .text()
+            .map_err(|e| GoFileError::Error(format!("content request for {cid} failed: {e}")))?;
+        let data: serde_json::Value = serde_json::from_str(&text).map_err(|_| {
+            GoFileError::Error(format!(
+                "non-JSON response for {cid}: {}",
+                truncate_200(&text)
+            ))
+        })?;
+
+        // python: data.get("status") == "notFound" or "notFound" in str(status)
+        let status = json_status(&data);
+        if status.contains("notFound") {
+            return Err(GoFileError::NotFound(format!(
+                "content {cid} does not exist anymore (dead link)"
+            )));
+        }
+        if status != "ok" {
+            return Err(GoFileError::Error(format!(
+                "content request failed for {cid}: {status}"
+            )));
+        }
+
+        let d = match data.get("data") {
+            Some(d @ serde_json::Value::Object(_)) => d,
+            _ => {
+                return Err(GoFileError::Error(format!(
+                    "content request failed for {cid}: missing data object"
+                )));
+            }
+        };
+
+        let password_ok = match d.get("passwordStatus") {
+            None | Some(serde_json::Value::Null) => true,
+            Some(serde_json::Value::String(s)) => s == "passwordOk",
+            Some(_) => false,
+        };
+        if !password_ok {
+            return Err(GoFileError::PasswordRequired(format!(
+                "content {cid} is password protected (use --password)"
+            )));
+        }
+
+        if d.get("type").and_then(|t| t.as_str()) == Some("folder") {
+            let name = required_str(d, "name", cid)?;
+            let mut here = prefix.to_vec();
+            here.push(name);
+            // serde_json maps children to a BTreeMap, so children are visited
+            // in key-sorted order instead of the API's insertion order (the
+            // orchestrator downloads all of them either way).
+            let children = d.get("children").and_then(|c| c.as_object());
+            if let Some(children) = children {
+                for (_key, child) in children {
+                    if child.get("type").and_then(|t| t.as_str()) == Some("folder") {
+                        let child_id = required_str(child, "id", cid)?;
+                        self.walk(&child_id, &here, results)?;
+                    } else {
+                        let child_name = required_str(child, "name", cid)?;
+                        let link = required_str(child, "link", cid)?;
+                        results.push((here.join("/"), child_name, link));
+                    }
+                }
+            }
+        } else {
+            let name = required_str(d, "name", cid)?;
+            let link = required_str(d, "link", cid)?;
+            results.push((prefix.join("/"), name, link));
+        }
+        Ok(())
+    }
+
+    /// Download one file; returns the file size. Skips existing files.
+    ///
+    /// Deviation from python's `timeout=(10, 60)`: reqwest blocking cannot
+    /// express "connect 10s / read 60s" per request. Connect is capped at
+    /// 10s client-wide; the download request gets a generous 2h total
+    /// timeout instead of python's uncapped streaming (60s per read) —
+    /// a stalled connection now aborts after at most 2h instead of 60s.
+    pub fn download_file(&self, link: &str, dest: &Path) -> Result<u64, GoFileError> {
+        if let Some(parent) = dest.parent() {
+            fs::create_dir_all(parent).map_err(|e| {
+                GoFileError::Error(format!("cannot create {}: {e}", parent.display()))
+            })?;
+        }
+        if dest.exists() {
+            let size = fs::metadata(dest).map(|m| m.len()).unwrap_or(0);
+            if size > 0 {
+                log::info!("Skip existing: {}", dest.display());
+                return Ok(size);
+            }
+        }
+
+        let name = dest
+            .file_name()
+            .map(|n| n.to_string_lossy().into_owned())
+            .unwrap_or_default();
+        let tmp = dest.with_file_name(format!("{name}.part"));
+
+        let result = self.download_to_tmp(link, dest, &tmp, &name);
+        // python `finally`: never leave a .part file behind on failure.
+        if tmp.exists() && !dest.exists() {
+            let _ = fs::remove_file(&tmp);
+        }
+        result
+    }
+
+    fn download_to_tmp(
+        &self,
+        link: &str,
+        dest: &Path,
+        tmp: &Path,
+        name: &str,
+    ) -> Result<u64, GoFileError> {
+        let mut headers = extended_headers();
+        set_header(
+            &mut headers,
+            COOKIE,
+            &format!("accountToken={}", self.token.as_deref().unwrap_or("None")),
+        );
+        set_header(&mut headers, REFERER, link);
+        set_header(&mut headers, ORIGIN, link);
+
+        let mut resp = self
+            .session
+            .get(link)
+            .headers(headers)
+            // python: timeout=(10, 60) — per-read 60s is not expressible in
+            // reqwest blocking, so use a generous total timeout instead.
+            .timeout(Duration::from_secs(2 * 3600))
+            .send()
+            .map_err(|e| GoFileError::Error(format!("download of {name} failed: {e}")))?;
+
+        let status = resp.status();
+        if status == reqwest::StatusCode::NOT_FOUND {
+            return Err(GoFileError::NotFound(format!(
+                "file {name} not found (dead link)"
+            )));
+        }
+        if status != reqwest::StatusCode::OK {
+            return Err(GoFileError::Error(format!(
+                "HTTP {status} while downloading {name}"
+            )));
+        }
+
+        let mut file = fs::File::create(tmp)
+            .map_err(|e| GoFileError::Error(format!("cannot create {}: {e}", tmp.display())))?;
+        let mut written = 0u64;
+        let mut buf = vec![0u8; CHUNK_SIZE];
+        loop {
+            let n = resp
+                .read(&mut buf)
+                .map_err(|e| GoFileError::Error(format!("download of {name} failed: {e}")))?;
+            if n == 0 {
+                break;
+            }
+            file.write_all(&buf[..n])
+                .map_err(|e| GoFileError::Error(format!("cannot write {name}: {e}")))?;
+            written += n as u64;
+        }
+        file.flush()
+            .map_err(|e| GoFileError::Error(format!("cannot write {name}: {e}")))?;
+
+        if written == 0 {
+            return Err(GoFileError::Error(format!("empty body for {name}")));
+        }
+        // os.replace semantics: rename over an existing destination.
+        fs::rename(tmp, dest)
+            .map_err(|e| GoFileError::Error(format!("cannot move {name} into place: {e}")))?;
+        Ok(written)
+    }
+}
+
+/// python `str(data.get("status", ""))` — missing status renders as "None".
+fn json_status(data: &serde_json::Value) -> String {
+    match data.get("status") {
+        None => "None".to_string(),
+        Some(serde_json::Value::String(s)) => s.clone(),
+        Some(other) => other.to_string(),
+    }
+}
+
+/// Fetch a required string field; a missing field mirrors python's KeyError
+/// (which would surface to the orchestrator as an unexpected failure).
+fn required_str(value: &serde_json::Value, key: &str, cid: &str) -> Result<String, GoFileError> {
+    value
+        .get(key)
+        .and_then(|v| v.as_str())
+        .map(str::to_string)
+        .ok_or_else(|| {
+            GoFileError::Error(format!("content {cid}: malformed entry (missing {key})"))
+        })
+}
+
+fn header_name(name: &'static str) -> HeaderName {
+    HeaderName::from_static(name)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn content_id_from_url_accepts_gofile_d_links() {
+        assert_eq!(
+            content_id_from_url("https://gofile.io/d/ABC123").unwrap(),
+            "ABC123"
+        );
+        // query string, fragment, trailing slash, deeper path
+        assert_eq!(
+            content_id_from_url("https://gofile.io/d/ABC123?x=1").unwrap(),
+            "ABC123"
+        );
+        assert_eq!(
+            content_id_from_url("https://gofile.io/d/ABC123/#anchor").unwrap(),
+            "ABC123"
+        );
+        // python urlparse is lenient about a missing scheme
+        assert_eq!(content_id_from_url("gofile.io/d/ABC123").unwrap(), "ABC123");
+        assert_eq!(
+            content_id_from_url("gofile.io/d/ABC123?x=1").unwrap(),
+            "ABC123"
+        );
+    }
+
+    #[test]
+    fn content_id_from_url_rejects_non_content_urls() {
+        let err = content_id_from_url("https://gofile.io/ABC123").unwrap_err();
+        assert!(
+            matches!(err, GoFileError::Error(ref m)
+                if m == "not a gofile content URL (expected gofile.io/d/<id>): https://gofile.io/ABC123"),
+            "unexpected: {err}"
+        );
+        assert!(content_id_from_url("https://example.com/foo/bar").is_err());
+        assert!(content_id_from_url("https://gofile.io/d/").is_err());
+    }
+
+    #[test]
+    fn website_token_for_is_deterministic_and_matches_python() {
+        let mut client = GoFileClient::new(None);
+        client.token = Some("testtoken".to_string());
+        // golden value computed with python hashlib for the same seed
+        assert_eq!(
+            client.website_token_for(0),
+            "1791d396eab9b0ef100bcb3c9211c80973ee8dd9b143fc9f388ced6a0fb3b0c0"
+        );
+        // stable across calls, 64 lowercase hex chars
+        let again = client.website_token_for(0);
+        assert_eq!(client.website_token_for(0), again);
+        assert_eq!(again.len(), 64);
+        assert!(
+            again
+                .bytes()
+                .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase())
+        );
+        // different window / token → different token
+        assert_ne!(client.website_token_for(1), client.website_token_for(0));
+        client.token = None;
+        // python renders a missing token as "None"
+        assert_eq!(
+            client.website_token_for(7),
+            "fb9a698ec598396eb77ab8b562fb1e30f01401dcf793701b290585e6393a657f"
+        );
+    }
+
+    #[test]
+    fn header_maps_match_python_keys() {
+        let common = common_headers();
+        let mut keys: Vec<String> = common.keys().map(|k| k.as_str().to_string()).collect();
+        keys.sort();
+        assert_eq!(
+            keys,
+            ["accept", "accept-encoding", "connection", "user-agent"]
+        );
+        assert_eq!(common.get("Accept-Encoding").unwrap(), "gzip");
+
+        let extended = extended_headers();
+        let mut ekeys: Vec<String> = extended.keys().map(|k| k.as_str().to_string()).collect();
+        ekeys.sort();
+        assert_eq!(
+            ekeys,
+            [
+                "accept",
+                "accept-encoding",
+                "cache-control",
+                "connection",
+                "pragma",
+                "sec-fetch-dest",
+                "sec-fetch-mode",
+                "sec-fetch-site",
+                "user-agent",
+            ]
+        );
+        assert_eq!(extended.get("Sec-Fetch-Site").unwrap(), "same-site");
+    }
+
+    #[test]
+    fn error_display_is_the_message() {
+        assert_eq!(
+            GoFileError::NotFound("content x not found (dead link)".into()).to_string(),
+            "content x not found (dead link)"
+        );
+        assert_eq!(GoFileError::PasswordRequired("pw".into()).to_string(), "pw");
+        assert_eq!(GoFileError::Empty("empty".into()).to_string(), "empty");
+        assert_eq!(GoFileError::AuthError("auth".into()).to_string(), "auth");
+        assert_eq!(GoFileError::Error("base".into()).to_string(), "base");
+    }
+
+    #[test]
+    fn sha256_hex_matches_python_hexdigest() {
+        // hashlib.sha256(b"abc").hexdigest()
+        assert_eq!(
+            sha256_hex(b"abc"),
+            "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
+        );
+    }
+
+    #[test]
+    fn truncate_200_counts_chars_not_bytes() {
+        let s = "ä".repeat(300);
+        assert_eq!(truncate_200(&s).chars().count(), 200);
+    }
+}

src/gwshop.rs

@@ -0,0 +1,662 @@
+//! GW webshop catalog dump via Algolia — Rust port of
+//! `dlphelper/gwshop.py`.
+//!
+//! The warhammer.com shop is behind an AWS WAF challenge for non-browser
+//! clients, but its product search runs on Algolia and the app-id plus the
+//! public search-only key ship with every browser page. This module uses
+//! them to enumerate the whole product index and normalize it into the
+//! sorter's catalog schema (system/group/faction/subfaction/class/name).
+//!
+//! Credentials are read from (first hit wins):
+//!   1. env ALGOLIA_APP_ID + ALGOLIA_API_KEY
+//!   2. algolia.json next to the repo root (gitignored):
+//!      {"app_id": "...", "api_key": "..."}
+//!
+//! The key is the *search-only* public key (never an admin key). Treat any
+//! key found in algolia.json as still secret: never log or commit it.
+//!
+//! CLI: `dlphelper gwshop OUT.json [--index NAME] [--raw RAW.json]` (wired
+//! in main.rs).
+
+use std::collections::BTreeMap;
+use std::path::{Path, PathBuf};
+use std::time::Duration;
+
+use anyhow::{Context as _, anyhow};
+use serde_json::{Value, json};
+
+use crate::util;
+
+pub const DEFAULT_INDEX: &str = "prod-lazarus-product-en-ww";
+// The read host (-dsn) serves searches; credentials also work as query
+// parameters, which keeps every request a plain GET/POST with no headers
+// beyond content-type.
+const DSN_HOST: &str = "https://{}-dsn.algolia.net";
+const PAGE_SIZE: usize = 1000; // Algolia max hitsPerPage per /query page
+const RETRY_DELAYS: [u64; 3] = [2, 5, 15];
+
+/// Resolve (app_id, api_key) from env or algolia.json.
+pub fn load_credentials() -> anyhow::Result<(String, String)> {
+    let mut app_id = std::env::var("ALGOLIA_APP_ID").unwrap_or_default();
+    let mut api_key = std::env::var("ALGOLIA_API_KEY").unwrap_or_default();
+    if !app_id.is_empty() && !api_key.is_empty() {
+        return Ok((app_id, api_key));
+    }
+    // algolia.json lives at the repo root, which is also the crate root, so
+    // CARGO_MANIFEST_DIR (baked at build time) points straight at it. Fall
+    // back to the current working directory for runs from elsewhere.
+    let candidates = [
+        Some(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("algolia.json")),
+        Some(PathBuf::from("algolia.json")),
+    ];
+    for candidate in candidates.into_iter().flatten() {
+        if candidate.is_file() {
+            let data: Value = serde_json::from_str(
+                &std::fs::read_to_string(&candidate)
+                    .with_context(|| format!("reading {}", candidate.display()))?,
+            )?;
+            app_id = data
+                .get("app_id")
+                .and_then(Value::as_str)
+                .filter(|s| !s.is_empty())
+                .map(str::to_string)
+                .unwrap_or(app_id);
+            api_key = data
+                .get("api_key")
+                .and_then(Value::as_str)
+                .filter(|s| !s.is_empty())
+                .map(str::to_string)
+                .unwrap_or(api_key);
+            break;
+        }
+    }
+    if app_id.is_empty() || api_key.is_empty() {
+        anyhow::bail!(
+            "Algolia credentials missing: set ALGOLIA_APP_ID and ALGOLIA_API_KEY, \
+             or create algolia.json with {{\"app_id\": \"...\", \"api_key\": \"...\"}} \
+             (gitignored)."
+        );
+    }
+    Ok((app_id, api_key))
+}
+
+/// Minimal Algolia search client for one application.
+pub struct GwShopClient {
+    base: String,
+    params: Vec<(String, String)>,
+    session: reqwest::blocking::Client,
+}
+
+impl GwShopClient {
+    pub fn new(app_id: &str, api_key: &str) -> Self {
+        GwShopClient {
+            base: DSN_HOST.replace("{}", app_id),
+            params: vec![
+                ("x-algolia-application-id".into(), app_id.into()),
+                ("x-algolia-api-key".into(), api_key.into()),
+            ],
+            session: reqwest::blocking::Client::new(),
+        }
+    }
+
+    fn request(&self, method: &str, path: &str, body: Option<Value>) -> anyhow::Result<Value> {
+        let url = format!("{base}{path}", base = self.base, path = path);
+        let mut last_err: Option<String> = None;
+        for (i, delay) in std::iter::once(0u64).chain(RETRY_DELAYS).enumerate() {
+            let attempt = i + 1;
+            if delay > 0 {
+                log::info!("Algolia retry {attempt} in {delay}s");
+                std::thread::sleep(Duration::from_secs(delay));
+            }
+            let mut req = match method {
+                "GET" => self.session.get(&url),
+                _ => self.session.post(&url),
+            }
+            .query(&self.params)
+            .timeout(Duration::from_secs(30));
+            if let Some(body) = &body {
+                req = req.json(body);
+            }
+            let resp = req.send();
+            match resp {
+                Ok(r) => {
+                    let status = r.status().as_u16();
+                    if status == 200 {
+                        return r.json().context("decoding Algolia response");
+                    }
+                    let text = r.text().unwrap_or_default();
+                    // Truncate by chars, not bytes: a byte slice could split a
+                    // multi-byte character and panic (python sliced `text[:200]`).
+                    let snippet: String = text.chars().take(200).collect();
+                    last_err = Some(format!("Algolia {method} {path} -> {status}: {snippet}"));
+                    if status == 401 || status == 403 {
+                        // credentials: no point retrying
+                        break;
+                    }
+                }
+                Err(exc) => last_err = Some(exc.to_string()),
+            }
+        }
+        Err(anyhow!(
+            "Algolia request failed: {}",
+            last_err.unwrap_or_default()
+        ))
+    }
+
+    /// Enumerate the application's indices (python `list_indices`). Kept as
+    /// a client capability even though the CLI does not use it yet.
+    #[allow(dead_code)]
+    pub fn list_indices(&self) -> anyhow::Result<Vec<Value>> {
+        let v = self.request("GET", "/1/indexes", None)?;
+        Ok(v.get("items")
+            .and_then(Value::as_array)
+            .cloned()
+            .unwrap_or_default())
+    }
+
+    // ---- full index retrieval -------------------------------------------
+    // The search-only key cannot use /browse (403) and normal pagination is
+    // capped at 1000 cumulative results. So we partition the index on the
+    // GameSystemsRoot hierarchical facet: each branch is fetched with an
+    // explicit filter, recursing deeper only when a branch exceeds one page.
+
+    const FACET: &'static str = "GameSystemsRoot";
+
+    fn escape(value: &str) -> String {
+        value.replace('\\', "\\\\").replace('"', "\\\"")
+    }
+
+    fn filter_expr(&self, path: &[(usize, &str)]) -> String {
+        path.iter()
+            .map(|(lvl, value)| format!("{}.lvl{}:\"{}\"", Self::FACET, lvl, Self::escape(value)))
+            .collect::<Vec<_>>()
+            .join(" AND ")
+    }
+
+    fn search_all(
+        &self,
+        index: &str,
+        filters: Option<&str>,
+    ) -> anyhow::Result<(Vec<Value>, usize)> {
+        // All hits matching `filters` (must be <= pagination cap).
+        let mut params = format!("query=&hitsPerPage={PAGE_SIZE}&page=0");
+        if let Some(filters) = filters {
+            params += &format!("&filters={}", urlencoding(filters));
+        }
+        let page = self.request(
+            "POST",
+            &format!("/1/indexes/{index}/query"),
+            Some(json!({ "params": params })),
+        )?;
+        let hits = page
+            .get("hits")
+            .and_then(Value::as_array)
+            .cloned()
+            .unwrap_or_default();
+        let nb = page.get("nbHits").and_then(Value::as_u64).unwrap_or(0) as usize;
+        Ok((hits, nb))
+    }
+
+    fn facet_counts(
+        &self,
+        index: &str,
+        level: usize,
+        filters: Option<&str>,
+    ) -> anyhow::Result<BTreeMap<String, usize>> {
+        let mut params = format!("query=&hitsPerPage=0&facets={}.lvl{level}", Self::FACET);
+        if let Some(filters) = filters {
+            params += &format!("&filters={}", urlencoding(filters));
+        }
+        let page = self.request(
+            "POST",
+            &format!("/1/indexes/{index}/query"),
+            Some(json!({ "params": params })),
+        )?;
+        let facets = page.get("facets").and_then(Value::as_object);
+        let counts = facets
+            .and_then(|f| f.get(&format!("{}.lvl{level}", Self::FACET)))
+            .and_then(Value::as_object);
+        let mut out = BTreeMap::new();
+        for (value, count) in counts.into_iter().flatten() {
+            if let Some(n) = count.as_u64() {
+                out.insert(value.clone(), n as usize);
+            }
+        }
+        Ok(out)
+    }
+
+    /// Every record of the index, via facet-partitioned search.
+    pub fn browse(&self, index: &str) -> anyhow::Result<Vec<Value>> {
+        let (hits, total) = self.search_all(index, None)?;
+        if total <= PAGE_SIZE {
+            return Ok(hits);
+        }
+        log::info!(
+            "gwshop {index}: {total} hits, partitioning on {}",
+            Self::FACET
+        );
+        let out = self.partition(index, &[], total)?;
+        let covered = out
+            .iter()
+            .filter_map(|h| h.get("objectID").cloned())
+            .collect::<std::collections::HashSet<Value>>()
+            .len();
+        if covered < total {
+            log::warn!(
+                "gwshop {index}: {} of {total} index records left out of the dump \
+                 (no fetchable {} branch path)",
+                total - covered,
+                Self::FACET
+            );
+        }
+        Ok(out)
+    }
+
+    fn partition(
+        &self,
+        index: &str,
+        path: &[(usize, String)],
+        total: usize,
+    ) -> anyhow::Result<Vec<Value>> {
+        let level = path.len();
+        let filter_refs: Vec<(usize, &str)> = path.iter().map(|(l, v)| (*l, v.as_str())).collect();
+        let filters = if filter_refs.is_empty() {
+            None
+        } else {
+            Some(self.filter_expr(&filter_refs))
+        };
+        let counts = self.facet_counts(index, level, filters.as_deref())?;
+        let mut out: Vec<Value> = Vec::new();
+        let mut covered: std::collections::HashSet<String> = std::collections::HashSet::new();
+        for (value, count) in &counts {
+            let mut sub_path: Vec<(usize, String)> = path.to_vec();
+            sub_path.push((level, value.clone()));
+            let sub_filters = self.filter_expr(
+                &sub_path
+                    .iter()
+                    .map(|(l, v)| (*l, v.as_str()))
+                    .collect::<Vec<_>>(),
+            );
+            if *count <= PAGE_SIZE {
+                let (hits, nb) = self.search_all(index, Some(&sub_filters))?;
+                log::info!(
+                    "gwshop {index}: {} -> {nb} hits",
+                    &sub_filters[..sub_filters.len().min(80)]
+                );
+                covered.extend(hits.iter().filter_map(|h| {
+                    h.get("objectID")
+                        .and_then(Value::as_str)
+                        .map(str::to_string)
+                }));
+                out.extend(hits);
+            } else {
+                let sub = self.partition(index, &sub_path, *count)?;
+                covered.extend(sub.iter().filter_map(|h| {
+                    h.get("objectID")
+                        .and_then(Value::as_str)
+                        .map(str::to_string)
+                }));
+                out.extend(sub);
+            }
+        }
+        // nbHits summed across branches double-counts products listed in
+        // several branches, so coverage is counted per unique objectID.
+        let residual = total.saturating_sub(covered.len());
+        if residual > 0 {
+            log::warn!(
+                "gwshop {index}: {residual} hits in branch carry no {}.lvl{level} \
+                 (shallow facet path, left out of the dump)",
+                Self::FACET
+            );
+        }
+        Ok(out)
+    }
+}
+
+/// python `requests.utils.quote(filters, safe='')` — percent-encode
+/// everything except unreserved characters.
+fn urlencoding(s: &str) -> String {
+    let mut out = String::new();
+    for byte in s.bytes() {
+        match byte {
+            b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
+                out.push(byte as char)
+            }
+            _ => out.push_str(&format!("%{byte:02X}")),
+        }
+    }
+    out
+}
+
+// ---------------------------------------------------------------------------
+// normalization
+// ---------------------------------------------------------------------------
+
+/// Every full path of the GameSystemsRoot hierarchical facet.
+///
+/// A hit may belong to several branches (e.g. an army box also being a
+/// "Gaming Rules" entry); each branch is returned as its own path
+/// [system, group, faction, subfaction_or_class, ...].
+pub fn hierarchy_paths(hit: &Value) -> Vec<Vec<String>> {
+    let root = hit.get("GameSystemsRoot").cloned().unwrap_or(Value::Null);
+    let Value::Object(root) = root else {
+        return Vec::new();
+    };
+    let mut levels: Vec<(usize, &Vec<Value>)> = root
+        .iter()
+        .filter_map(|(k, v)| {
+            let lvl = k.strip_prefix("lvl")?.parse::<usize>().ok()?;
+            v.as_array().map(|a| (lvl, a))
+        })
+        .collect();
+    levels.sort_by_key(|(l, _)| *l);
+    let mut paths = Vec::new();
+    for (_, values) in levels {
+        for value in values {
+            let Some(s) = value.as_str() else { continue };
+            paths.push(s.split(" > ").map(|p| p.trim().to_string()).collect());
+        }
+    }
+    paths
+}
+
+/// One Algolia hit -> one or more catalog entries (usually one).
+fn normalize(hit: &Value) -> Value {
+    let name = hit
+        .get("name")
+        .and_then(Value::as_str)
+        .or_else(|| hit.get("objectID").and_then(Value::as_str))
+        .unwrap_or_default()
+        .trim()
+        .to_string();
+    json!({
+        "name": name,
+        "aliases": [],
+        "sku": hit.get("sku"),
+        "slug": hit.get("slug"),
+        "productType": hit.get("productType"),
+        "source": "gwshop",
+        "gw_paths": hierarchy_paths(hit),
+        "collections": hit.get("_collections").cloned().unwrap_or_else(|| json!([])),
+    })
+}
+
+fn entry_from(
+    base: &Value,
+    system: Value,
+    group: Value,
+    faction: Value,
+    subfaction: Value,
+    class: Value,
+) -> Value {
+    json!({
+        "system": system,
+        "group": group,
+        "faction": faction,
+        "subfaction": subfaction,
+        "class": class,
+        "name": base["name"],
+        "aliases": base["aliases"],
+        "sku": base["sku"],
+        "slug": base["slug"],
+        "productType": base["productType"],
+        "source": "gwshop",
+        "collections": base["collections"],
+    })
+}
+
+/// Index dump -> catalog document in the sorter's schema shape.
+///
+/// The hierarchy layer (system/group/faction/subfaction) is expanded from
+/// gw_paths; name stays per product so the fuzzy matcher works on the
+/// product name, exactly like data/library_catalog.json entries.
+pub fn build_catalog(hits: &[Value]) -> Value {
+    let mut entries: Vec<Value> = Vec::new();
+    let mut seen: std::collections::HashSet<(String, Vec<String>)> =
+        std::collections::HashSet::new();
+    for hit in hits {
+        let base = normalize(hit);
+        // python: paths = base["gw_paths"] (precomputed by normalize)
+        let paths: Vec<Vec<String>> = base["gw_paths"]
+            .as_array()
+            .map(|arr| {
+                arr.iter()
+                    .map(|p| {
+                        p.as_array()
+                            .map(|parts| {
+                                parts
+                                    .iter()
+                                    .filter_map(Value::as_str)
+                                    .map(str::to_string)
+                                    .collect()
+                            })
+                            .unwrap_or_default()
+                    })
+                    .collect()
+            })
+            .unwrap_or_default();
+        if paths.is_empty() {
+            let key = (
+                hit.get("objectID")
+                    .and_then(Value::as_str)
+                    .unwrap_or_default()
+                    .to_string(),
+                Vec::new(),
+            );
+            if !seen.insert(key) {
+                continue;
+            }
+            entries.push(entry_from(
+                &base,
+                Value::Null,
+                Value::Null,
+                Value::Null,
+                Value::Null,
+                Value::Null,
+            ));
+            continue;
+        }
+        // primary branch = longest path (most specific), e.g.
+        // [40k, Space Marines, Raven Guard, Walker].
+        // python `max(paths, key=len)` keeps the FIRST longest path on ties;
+        // Rust's `max_by_key` would keep the LAST, so select explicitly to
+        // stay faithful to the original (products faceted under several
+        // branches, e.g. a generic kit listed per chapter, depend on this).
+        let primary = paths.iter().fold(Vec::new(), |best, p| {
+            if p.len() > best.len() {
+                p.clone()
+            } else {
+                best
+            }
+        });
+        let at = |i: usize| -> Value {
+            primary
+                .get(i)
+                .map(|s| Value::String(s.clone()))
+                .unwrap_or(Value::Null)
+        };
+        let system = at(0);
+        let group = at(1);
+        let faction = at(2);
+        // The LAST level is the class/category layer (GW uses shop
+        // categories there: Battleline, Walker, Vehicle, "Rules &
+        // Accessories", product lines like "Cadian"). Deeper sources
+        // (miniset style) may carry a real subfaction before it.
+        let (subfaction, klass) = if primary.len() >= 5 {
+            (at(3), at(4))
+        } else if primary.len() == 4 {
+            (Value::Null, at(3))
+        } else {
+            (Value::Null, Value::Null)
+        };
+        let key = (
+            hit.get("objectID")
+                .and_then(Value::as_str)
+                .unwrap_or_default()
+                .to_string(),
+            primary.clone(),
+        );
+        if !seen.insert(key) {
+            continue;
+        }
+        entries.push(entry_from(&base, system, group, faction, subfaction, klass));
+    }
+    json!({
+        "comment": (
+            "Generated by dlphelper (Rust) from the Games Workshop webshop \
+             (Algolia index, deduplicated by product). Source-agnostic \
+             catalog schema: system/group/faction/subfaction/class/name/aliases."
+        ),
+        "source": "gwshop",
+        "generated_entries": entries.len(),
+        "entries": entries,
+    })
+}
+
+/// CLI body: dump the GW webshop catalog via Algolia.
+pub fn run_cli(out: &Path, index: &str, raw: Option<&Path>) -> anyhow::Result<()> {
+    let (app_id, api_key) = load_credentials()?;
+    let client = GwShopClient::new(&app_id, &api_key);
+    let hits = client.browse(index)?;
+    log::info!("{len} raw hits from {index}", len = hits.len());
+
+    if let Some(raw_path) = raw {
+        util::save_sorted(raw_path, &hits)
+            .with_context(|| format!("writing raw hits to {}", raw_path.display()))?;
+        log::info!("raw hits written to {}", raw_path.display());
+    }
+
+    let catalog = build_catalog(&hits);
+    std::fs::write(out, serde_json::to_string_pretty(&catalog)?.as_bytes())
+        .with_context(|| format!("writing catalog to {}", out.display()))?;
+    log::info!(
+        "{n} catalog entries written to {p}",
+        n = catalog["generated_entries"],
+        p = out.display()
+    );
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn urlencoding_matches_python_quote_safe_empty() {
+        // requests.utils.quote(s, safe='') percent-encodes ' ' and '"'
+        assert_eq!(urlencoding("a b"), "a%20b");
+        assert_eq!(urlencoding("a\"b"), "a%22b");
+        assert_eq!(urlencoding("abc-1.2~_"), "abc-1.2~_");
+        assert_eq!(
+            urlencoding("GameSystemsRoot.lvl0:\"Warhammer 40,000\""),
+            "GameSystemsRoot.lvl0%3A%22Warhammer%2040%2C000%22"
+        );
+    }
+
+    #[test]
+    fn hierarchy_paths_orders_levels() {
+        // python returns EVERY lvl as its own path (each value split on " > ")
+        let hit = json!({
+            "GameSystemsRoot": {
+                "lvl2": ["Warhammer 40,000 > Aeldari > Rangers"],
+                "lvl0": ["Warhammer 40,000"],
+                "lvl1": ["Warhammer 40,000 > Aeldari"],
+            }
+        });
+        assert_eq!(
+            hierarchy_paths(&hit),
+            vec![
+                vec!["Warhammer 40,000".to_string()],
+                vec!["Warhammer 40,000".to_string(), "Aeldari".to_string()],
+                vec![
+                    "Warhammer 40,000".to_string(),
+                    "Aeldari".to_string(),
+                    "Rangers".to_string()
+                ],
+            ]
+        );
+    }
+
+    #[test]
+    fn build_catalog_normalizes_primary_branch() {
+        let hit = json!({
+            "objectID": "obj1",
+            "name": "  Rangers  ",
+            "sku": "S1",
+            "GameSystemsRoot": {
+                "lvl0": ["Warhammer 40,000"],
+                "lvl1": ["Warhammer 40,000 > Aeldari"],
+                "lvl2": ["Warhammer 40,000 > Aeldari > Rangers"],
+                "lvl3": ["Warhammer 40,000 > Aeldari > Rangers > Battleline"],
+            }
+        });
+        let cat = build_catalog(&[hit]);
+        assert_eq!(cat["generated_entries"], 1);
+        let e = &cat["entries"][0];
+        assert_eq!(e["name"], "Rangers");
+        // primary = LONGEST path [40k, Aeldari, Rangers, Battleline]:
+        // system=40k, group=Aeldari (lvl1 value is a full path split? no -
+        // build_catalog indexes primary directly: lvl0=system, lvl1=group, ...)
+        assert_eq!(e["system"], "Warhammer 40,000");
+        assert_eq!(e["group"], "Aeldari");
+        assert_eq!(e["faction"], "Rangers");
+        assert_eq!(e["subfaction"], Value::Null);
+        assert_eq!(e["class"], "Battleline");
+        assert_eq!(e["source"], "gwshop");
+    }
+
+    #[test]
+    fn build_catalog_primary_branch_prefers_first_longest_tie() {
+        // python max(paths, key=len) keeps the first longest path; the class
+        // layer must come from that branch, not the last tied one.
+        let hit = json!({
+            "objectID": "obj3",
+            "name": "Stomper Tribe",
+            "sku": "S3",
+            "GameSystemsRoot": {
+                "lvl0": ["Age of Sigmar"],
+                "lvl1": ["Age of Sigmar > Grand Alliance Destruction"],
+                "lvl2": ["Age of Sigmar > Grand Alliance Destruction > Sons of Behemat"],
+                "lvl3": [
+                    "Age of Sigmar > Grand Alliance Destruction > Sons of Behemat > Gargant",
+                    "Age of Sigmar > Grand Alliance Destruction > Sons of Behemat > Monster",
+                ],
+            }
+        });
+        let cat = build_catalog(&[hit]);
+        let e = &cat["entries"][0];
+        assert_eq!(e["faction"], "Sons of Behemat");
+        assert_eq!(e["class"], "Gargant");
+    }
+
+    #[test]
+    fn build_catalog_dedups_and_handles_pathless() {
+        let hit = json!({"objectID": "obj2", "name": "Paint"});
+        let cat = build_catalog(&[hit.clone(), hit]);
+        assert_eq!(cat["generated_entries"], 1);
+        let e = &cat["entries"][0];
+        assert_eq!(e["system"], Value::Null);
+        assert_eq!(e["name"], "Paint");
+    }
+
+    #[test]
+    fn credentials_error_message() {
+        // No env, no file: error message matches the python SystemExit text.
+        // NOTE: env manipulation is unsafe in recent Rust editions and
+        // racy across parallel tests; only assert the message shape when
+        // the vars are absent from the start.
+        if std::env::var("ALGOLIA_APP_ID")
+            .map(|v| v.is_empty())
+            .unwrap_or(true)
+            && std::env::var("ALGOLIA_API_KEY")
+                .map(|v| v.is_empty())
+                .unwrap_or(true)
+            && !PathBuf::from(env!("CARGO_MANIFEST_DIR"))
+                .join("algolia.json")
+                .exists()
+        {
+            let err = load_credentials().unwrap_err().to_string();
+            assert!(err.contains("Algolia credentials missing"), "{err}");
+        }
+    }
+}

src/main.rs

@@ -0,0 +1,626 @@
+//! dlphelper CLI — Rust rewrite of the python entry points:
+//!
+//! - `dlphelper dlp ...`          <- dlp.py
+//! - `dlphelper catalogue ...`    <- catalogue.py
+//! - `dlphelper sorter ...`       <- sorter.py
+//! - `dlphelper extract ...`      <- tools/extract_all.py
+//! - `dlphelper gwshop ...`       <- dlphelper/gwshop.py
+//!
+//! Secrets rule: the Discord token is resolved via --token / $DISCORD_TOKEN /
+//! hidden prompt and only ever handed to the DCE container via env.
+
+use std::io::IsTerminal;
+use std::path::{Path, PathBuf};
+use std::process::exit;
+
+use anyhow::Context as _;
+use clap::{Parser, Subcommand};
+
+mod catalogue;
+mod difflib;
+mod discord;
+mod extract;
+mod globalstore;
+mod gofile;
+mod gwshop;
+mod manifest;
+mod orchestrate;
+mod sorter;
+mod state;
+mod util;
+mod warhammer_data;
+mod zero807;
+
+#[derive(Parser)]
+#[command(
+    name = "dlphelper",
+    version,
+    about = "Collect, catalogue and sort Warhammer downloads (Rust rewrite)."
+)]
+struct Cli {
+    #[command(subcommand)]
+    cmd: Cmd,
+}
+
+#[derive(Subcommand)]
+enum Cmd {
+    /// Collect downloads from Discord channels (gofile + 0807).
+    Dlp(DlpArgs),
+    /// Classify the collection into system / faction / class.
+    Catalogue(CatalogueArgs),
+    /// Fuzzysort collection files into the library (plan/apply).
+    Sorter(SorterArgs),
+    /// Extract archives into per-source directories.
+    Extract(ExtractArgs),
+    /// Dump the GW webshop catalog via Algolia.
+    Gwshop(GwshopArgs),
+}
+
+#[derive(clap::Args)]
+struct DlpArgs {
+    /// Discord channel or thread id
+    channel_id: Option<String>,
+    /// directory for downloads and state
+    out_dir: Option<PathBuf>,
+    /// process a subscription list instead of one channel
+    /// (entries: channel_id, out_dir, optional name)
+    #[arg(long, value_name = "FILE")]
+    subs: Option<PathBuf>,
+    /// Discord token (else $DISCORD_TOKEN or prompt)
+    #[arg(long)]
+    token: Option<String>,
+    /// skip the Discord export; read URLs from this file (e.g. failed_urls.txt)
+    #[arg(long, value_name = "FILE")]
+    urls_file: Option<PathBuf>,
+    /// password for password-protected gofile links
+    #[arg(long)]
+    password: Option<String>,
+    /// attempts per URL (default 3)
+    #[arg(long, default_value_t = 3)]
+    retries: u32,
+    /// base seconds between retries, multiplied per attempt (default 5)
+    #[arg(long, default_value_t = 5)]
+    retry_delay: u64,
+    /// consecutive failures before a cooldown pause (default 3)
+    #[arg(long, default_value_t = 3)]
+    cooldown_failures: u32,
+    /// length of the cooldown pause (default 60)
+    #[arg(long, default_value_t = 60)]
+    cooldown_seconds: u64,
+    /// process only one host class
+    #[arg(long, value_parser = ["gofile", "0807"])]
+    only: Option<String>,
+    /// only URLs posted at/after this point: e.g. 5d, 12h, 2026-10-04
+    #[arg(long, value_name = "WHEN")]
+    since: Option<String>,
+    /// download at most N pending URLs per host class (test runs)
+    #[arg(long, value_name = "N")]
+    limit: Option<usize>,
+    /// randomize which URLs a --limit picks
+    #[arg(long)]
+    shuffle: bool,
+    /// record sha256 of each downloaded file
+    #[arg(long)]
+    hash: bool,
+    /// unzip downloaded archives into per-archive folders
+    #[arg(long)]
+    extract: bool,
+    /// re-export the Discord channel even if export.json exists
+    /// (subs mode does this by default)
+    #[arg(long)]
+    fresh: bool,
+    /// subs mode: reuse cached exports instead of re-exporting
+    #[arg(long)]
+    no_fresh: bool,
+    /// export/extract/classify only, download nothing
+    #[arg(long)]
+    dry_run: bool,
+    /// cross-run URL store FILE (default: $XDG_DATA_HOME/dlphelper/urls.json
+    /// or ~/.local/share/dlphelper/urls.json)
+    #[arg(long, value_name = "FILE")]
+    global_state: Option<PathBuf>,
+    /// disable the cross-run URL store for this run
+    #[arg(long)]
+    no_global: bool,
+    /// mark every URL in the file as already downloaded, then exit
+    /// (manual import of old download lists)
+    #[arg(long, value_name = "FILE")]
+    assume_done: Option<PathBuf>,
+}
+
+#[derive(clap::Args)]
+struct CatalogueArgs {
+    /// a dlp.py run dir (uses its manifest.jsonl and downloads/)
+    run_dir: Option<PathBuf>,
+    /// additional directory to scan for archive files (repeatable)
+    #[arg(long, value_name = "DIR")]
+    scan: Vec<PathBuf>,
+    /// explicit manifest.jsonl (default: <run_dir>/manifest.jsonl)
+    #[arg(long, value_name = "FILE")]
+    manifest: Option<PathBuf>,
+    /// manual classification overrides (JSON)
+    #[arg(long, value_name = "FILE")]
+    overrides: Option<PathBuf>,
+    #[arg(long, value_name = "FILE")]
+    out_md: Option<PathBuf>,
+    #[arg(long, value_name = "FILE")]
+    out_jsonl: Option<PathBuf>,
+    /// move files into the library structure (default: preview)
+    #[arg(long)]
+    apply: bool,
+    /// library root for --apply (required with --apply)
+    #[arg(long, value_name = "DIR")]
+    library: Option<PathBuf>,
+}
+
+#[derive(clap::Args)]
+struct SorterArgs {
+    /// a directory to sort or a single file
+    source: PathBuf,
+    /// library root the files will be linked into
+    #[arg(long, value_name = "DIR")]
+    library: PathBuf,
+    /// catalog FILE (default: data/gwshop_catalog.json, relative to the cwd)
+    #[arg(long, value_name = "FILE")]
+    catalog: Option<PathBuf>,
+    /// fuzzy match threshold (default 0.62)
+    #[arg(long, default_value_t = sorter::DEFAULT_THRESHOLD)]
+    threshold: f64,
+    /// create the targets (hardlink, fallback copy)
+    #[arg(long)]
+    apply: bool,
+    /// audit FILE (default: <library>/../_logs/sorter_applied.jsonl)
+    #[arg(long, value_name = "FILE")]
+    audit: Option<PathBuf>,
+    /// extract_manifest.jsonl from extract: sort whole extraction
+    /// directories as units
+    #[arg(long, value_name = "FILE")]
+    provenance: Option<PathBuf>,
+    /// print the plan as JSON instead of text
+    #[arg(long)]
+    json: bool,
+    /// never place into this game system (default-dropped: Middle-Earth)
+    #[arg(long, value_name = "NAME")]
+    drop_system: Vec<String>,
+    /// re-enable a default-dropped game system
+    #[arg(long, value_name = "NAME")]
+    keep_system: Vec<String>,
+}
+
+#[derive(clap::Args)]
+struct ExtractArgs {
+    /// archives or directories containing archives
+    sources: Vec<PathBuf>,
+    /// directory to extract into
+    #[arg(long, required = true, value_name = "DIR")]
+    out: PathBuf,
+}
+
+#[derive(clap::Args)]
+struct GwshopArgs {
+    /// output catalog JSON path
+    out: PathBuf,
+    /// Algolia index name
+    #[arg(long, default_value = gwshop::DEFAULT_INDEX)]
+    index: String,
+    /// also write the raw Algolia hits here
+    #[arg(long, value_name = "FILE")]
+    raw: Option<PathBuf>,
+}
+
+/// argparse parser.error equivalent: message to stderr, exit code 2.
+fn parser_error(msg: &str) -> ! {
+    eprintln!("error: {msg}");
+    exit(2);
+}
+
+/// python resolve_token: --token, then $DISCORD_TOKEN, then hidden prompt.
+fn resolve_token(cli_value: Option<&str>) -> anyhow::Result<String> {
+    if let Some(t) = cli_value {
+        return Ok(t.to_string());
+    }
+    if let Ok(t) = std::env::var("DISCORD_TOKEN")
+        && !t.is_empty()
+    {
+        return Ok(t);
+    }
+    log::info!("No Discord token given ($DISCORD_TOKEN); prompting (input hidden).");
+    if !std::io::stdin().is_terminal() {
+        anyhow::bail!(
+            "No Discord token available and stdin is not a terminal; \
+                      use --token or $DISCORD_TOKEN."
+        );
+    }
+    let token = rpassword::prompt_password("Discord token: ").context("reading the token")?;
+    if token.is_empty() {
+        anyhow::bail!("No token provided, aborting.");
+    }
+    Ok(token)
+}
+
+/// The global store for this run, honoring --no-global / --global-state.
+fn make_gstore(args: &DlpArgs) -> globalstore::GlobalStore {
+    if args.no_global {
+        globalstore::GlobalStore::disabled()
+    } else if let Some(p) = &args.global_state {
+        globalstore::GlobalStore::load(p)
+    } else {
+        match globalstore::default_global_path() {
+            Some(p) => globalstore::GlobalStore::load(p),
+            None => globalstore::GlobalStore::disabled(), // no home dir: like python gstore=None
+        }
+    }
+}
+
+fn default_catalog_path() -> PathBuf {
+    PathBuf::from("data/gwshop_catalog.json")
+}
+
+fn cmd_dlp(args: &DlpArgs) -> anyhow::Result<()> {
+    if let Some(assume) = &args.assume_done {
+        if args.channel_id.is_some() || args.subs.is_some() {
+            parser_error("--assume-done cannot be combined with a channel or --subs");
+        }
+        let mut gstore = make_gstore(args);
+        orchestrate::mark_assume_done(assume, &mut gstore)?;
+        return Ok(());
+    }
+
+    if let Some(subs_path) = &args.subs {
+        if args.channel_id.is_some() || args.out_dir.is_some() || args.urls_file.is_some() {
+            parser_error("--subs cannot be combined with channel_id/out_dir/--urls-file");
+        }
+        let subs = orchestrate::load_subscriptions(subs_path)?;
+        let token = resolve_token(args.token.as_deref())?;
+        let mut gstore = make_gstore(args);
+        let mut total_failed = 0u64;
+        for (channel_id, out_dir, name) in &subs {
+            log::info!(
+                "=== subscription: {} ===",
+                name.clone().unwrap_or_else(|| channel_id.clone())
+            );
+            let opts = orchestrate::RunOpts::from(args);
+            // subs mode re-exports by default (python force_export=not --no-fresh)
+            total_failed += orchestrate::process_channel(
+                channel_id,
+                Path::new(out_dir),
+                &opts,
+                Some(&token),
+                Some(&mut gstore),
+                None,
+                !args.no_fresh,
+            )?;
+        }
+        if total_failed > 0 {
+            log::warn!("{total_failed} URL(s) failed in total across subscriptions.");
+            exit(1);
+        }
+        return Ok(());
+    }
+
+    let (Some(channel_id), Some(out_dir)) = (&args.channel_id, &args.out_dir) else {
+        parser_error("channel_id and out_dir are required (or use --subs/--assume-done)");
+    };
+    // A token is only needed when we will export a channel (--urls-file skips that)
+    let token = if args.urls_file.is_some() {
+        None
+    } else {
+        Some(resolve_token(args.token.as_deref())?)
+    };
+    let opts = orchestrate::RunOpts::from(args);
+    let mut gstore = make_gstore(args);
+    let failures = orchestrate::process_channel(
+        channel_id,
+        out_dir,
+        &opts,
+        token.as_deref(),
+        Some(&mut gstore),
+        args.urls_file.as_deref(),
+        false,
+    )?;
+    if failures > 0 {
+        exit(1);
+    }
+    Ok(())
+}
+
+impl From<&DlpArgs> for orchestrate::RunOpts {
+    fn from(a: &DlpArgs) -> Self {
+        orchestrate::RunOpts {
+            retries: a.retries,
+            retry_delay: a.retry_delay,
+            cooldown_failures: a.cooldown_failures,
+            cooldown_seconds: a.cooldown_seconds,
+            hash: a.hash,
+            extract: a.extract,
+            password: a.password.clone(),
+            limit: a.limit,
+            shuffle: a.shuffle,
+            only: a.only.clone(),
+            since: a.since.clone(),
+            dry_run: a.dry_run,
+            fresh: a.fresh,
+        }
+    }
+}
+
+fn cmd_catalogue(args: &CatalogueArgs) -> anyhow::Result<()> {
+    let run_dir = args.run_dir.as_deref();
+    let manifests: Vec<PathBuf> = if let Some(m) = &args.manifest {
+        vec![m.clone()]
+    } else {
+        run_dir
+            .map(|d| vec![d.join("manifest.jsonl")])
+            .unwrap_or_default()
+    };
+    let mut scan_dirs = args.scan.clone();
+    if let Some(rd) = run_dir
+        && args.manifest.is_none()
+    {
+        // run dir is scanned as a whole (covers files not yet in the manifest)
+        scan_dirs.push(rd.to_path_buf());
+    }
+
+    if manifests.is_empty() && scan_dirs.is_empty() {
+        parser_error("give a run_dir or at least one --scan directory");
+    }
+
+    let out_base: PathBuf = run_dir
+        .map(Path::to_path_buf)
+        .unwrap_or_else(|| std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")));
+    let out_md = args
+        .out_md
+        .clone()
+        .unwrap_or_else(|| out_base.join("catalogue.md"));
+    let out_jsonl = args
+        .out_jsonl
+        .clone()
+        .unwrap_or_else(|| out_base.join("catalogue.jsonl"));
+
+    let mut entries =
+        catalogue::build_catalogue(&manifests, &scan_dirs, args.overrides.as_deref())?;
+    catalogue::write_reports(&entries, &out_md, &out_jsonl)?;
+
+    let classified = entries
+        .iter()
+        .filter(|e| e.faction != catalogue::UNCLASSIFIED_FACTION)
+        .count();
+    log::info!(
+        "Catalogued {} files ({} classified) -> {}",
+        entries.len(),
+        classified,
+        out_md.display()
+    );
+
+    // compact per-faction summary to the console
+    let mut summary: std::collections::BTreeMap<(String, String), (u64, u64)> =
+        std::collections::BTreeMap::new();
+    for e in &entries {
+        let entry = summary
+            .entry((e.system.clone(), e.faction.clone()))
+            .or_insert((0, 0));
+        entry.0 += 1;
+        entry.1 += e.size;
+    }
+    for ((system, faction), (n, size)) in &summary {
+        log::info!(
+            "  {:<18} {:<32} {:>4} files  {}",
+            system,
+            faction,
+            n,
+            catalogue::format_size(*size)
+        );
+    }
+
+    if args.apply {
+        let Some(library) = &args.library else {
+            parser_error("--apply needs --library DIR");
+        };
+        let Some(_run_dir) = run_dir else {
+            parser_error("--apply needs a run_dir (files are moved out of it)");
+        };
+        let moved = catalogue::apply_moves(
+            &mut entries,
+            library,
+            &library.join("catalogue_moves.jsonl"),
+        )?;
+        log::info!(
+            "Moved {} files into {} (audit: catalogue_moves.jsonl)",
+            moved,
+            library.display()
+        );
+        // rewrite reports with updated paths
+        catalogue::write_reports(&entries, &out_md, &out_jsonl)?;
+    }
+    Ok(())
+}
+
+fn load_provenance(path: &Path) -> anyhow::Result<sorter::Provenance> {
+    use std::io::BufRead;
+    let mut prov = sorter::Provenance::new();
+    let fh = std::fs::File::open(path).with_context(|| format!("reading {}", path.display()))?;
+    for line in std::io::BufReader::new(fh).lines() {
+        let rec: serde_json::Value = serde_json::from_str(&line?)?;
+        let dest = rec["dest"]
+            .as_str()
+            .context("provenance row lacks 'dest'")?;
+        let unit = rec["unit"]
+            .as_str()
+            .context("provenance row lacks 'unit'")?;
+        prov.insert(dest.to_string(), unit.to_string());
+    }
+    Ok(prov)
+}
+
+fn cmd_sorter(args: &SorterArgs) -> anyhow::Result<()> {
+    let dropped = sorter::dropped_systems_default(&args.keep_system, &args.drop_system);
+    if !args.keep_system.is_empty() || !args.drop_system.is_empty() {
+        log::info!(
+            "System filter adjusted: dropping {}",
+            if dropped.is_empty() {
+                "nothing".to_string()
+            } else {
+                let mut s = dropped.clone();
+                s.sort();
+                s.join(", ")
+            }
+        );
+    }
+
+    let source = &args.source;
+    let files: Vec<PathBuf> = if source.is_file() {
+        vec![source.clone()]
+    } else {
+        // hidden trees like .broken/ must never reach the plan
+        let mut out = Vec::new();
+        collect_visible(source, &mut out)?;
+        out.sort();
+        out
+    };
+    let catalog_path = args.catalog.clone().unwrap_or_else(default_catalog_path);
+    let catalog = sorter::load_catalog(&catalog_path, &dropped)?;
+    if !dropped.is_empty() {
+        let mut s = dropped.clone();
+        s.sort();
+        log::info!("Dropped systems: {}", s.join(", "));
+    }
+    // Bookkeeping must never land inside the library itself (it may be a
+    // seeded torrent whose file set is frozen): default to a _logs folder
+    // beside it. An explicit --audit overrides this.
+    let audit = args.audit.clone().unwrap_or_else(|| {
+        args.library
+            .parent()
+            .unwrap_or(Path::new("."))
+            .join("_logs")
+            .join("sorter_applied.jsonl")
+    });
+    log::info!("Audit file: {}", audit.display());
+
+    let provenance = match &args.provenance {
+        Some(p) => {
+            let prov = load_provenance(p)?;
+            log::info!("Provenance: {} extraction directories", prov.len());
+            Some(prov)
+        }
+        None => None,
+    };
+
+    let mut items = sorter::sort_plan(
+        &files,
+        &args.library,
+        &catalog,
+        args.threshold,
+        provenance.as_ref(),
+        Some(source),
+    );
+
+    if args.json {
+        let rows: Vec<serde_json::Value> = items
+            .iter()
+            .map(|i| {
+                serde_json::json!({
+                    "source": i.path.display().to_string(),
+                    "name": i.name,
+                    "size": i.size,
+                    "system": i.system,
+                    "group": i.group,
+                    "faction": i.faction,
+                    "subfaction": i.subfaction,
+                    "class": i.klass,
+                    "unit": i.unit,
+                    "score": (i.match_score * 1000.0).round_ties_even() / 1000.0,
+                    "kind": i.match_kind,
+                    "inner_count": i.inner_names.len(),
+                    "target": i.target
+                        .as_ref()
+                        .map(|t| t.display().to_string())
+                        .unwrap_or_else(|| "None".into()),
+                })
+            })
+            .collect();
+        println!("{}", serde_json::to_string_pretty(&rows)?);
+    } else {
+        println!("{}", sorter::format_plan(&items));
+    }
+
+    let n_catalog = items.iter().filter(|i| i.match_kind == "catalog").count();
+    let n_implicit = items.iter().filter(|i| i.match_kind == "implicit").count();
+    let n_none = items.iter().filter(|i| i.match_kind == "none").count();
+    log::info!(
+        "Plan: {} files -> {} catalog matches [+] / {} implicit [~] / {} unclassifiable [?]",
+        items.len(),
+        n_catalog,
+        n_implicit,
+        n_none
+    );
+
+    if args.apply {
+        let (linked, copied, skipped) = sorter::apply_plan(&mut items, &audit)?;
+        log::info!(
+            "Applied: {} hardlinked, {} copied, {} skipped (audit: {})",
+            linked,
+            copied,
+            skipped,
+            audit.display()
+        );
+    }
+    Ok(())
+}
+
+/// Recursively collect files under `dir`, skipping any path whose file name
+/// starts with '.' (python rglob filter).
+fn collect_visible(dir: &Path, out: &mut Vec<PathBuf>) -> std::io::Result<()> {
+    for entry in std::fs::read_dir(dir)? {
+        let entry = entry?;
+        let path = entry.path();
+        let name = entry.file_name().to_string_lossy().into_owned();
+        if name.starts_with('.') {
+            continue;
+        }
+        if path.is_dir() {
+            collect_visible(&path, out)?;
+        } else {
+            out.push(path);
+        }
+    }
+    Ok(())
+}
+
+fn cmd_extract(args: &ExtractArgs) -> anyhow::Result<()> {
+    let ex = extract::Extractor {
+        out_root: args.out.clone(),
+    };
+    ex.run(&args.sources)
+}
+
+fn cmd_gwshop(args: &GwshopArgs) -> anyhow::Result<()> {
+    gwshop::run_cli(&args.out, &args.index, args.raw.as_deref())
+}
+
+fn main() {
+    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info"))
+        .format(|buf, record| {
+            use std::io::Write as _;
+            writeln!(
+                buf,
+                "{} {:<7} {}",
+                chrono::Local::now().format("%H:%M:%S"),
+                record.level(),
+                record.args()
+            )
+        })
+        .init();
+
+    let cli = Cli::parse();
+    let res = match &cli.cmd {
+        Cmd::Dlp(args) => cmd_dlp(args),
+        Cmd::Catalogue(args) => cmd_catalogue(args),
+        Cmd::Sorter(args) => cmd_sorter(args),
+        Cmd::Extract(args) => cmd_extract(args),
+        Cmd::Gwshop(args) => cmd_gwshop(args),
+    };
+    if let Err(err) = res {
+        log::error!("{err:#}");
+        exit(1);
+    }
+}

src/manifest.rs

@@ -0,0 +1,101 @@
+//! manifest.jsonl — one compact JSON line per downloaded file — Rust port of
+//! `append_manifest` from `dlphelper/state.py`.
+//!
+//! Row layout (all attribution fields optional except channel/url):
+//! `{"channel", "url", "host", "message_id", "timestamp", "author", "path",
+//! "size", ...}` — unknown extra fields are preserved on round-trip.
+
+use std::collections::BTreeMap;
+use std::fs::OpenOptions;
+use std::io::Write;
+use std::path::Path;
+
+use anyhow::Context as _;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ManifestRow {
+    pub channel: String,
+    pub url: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub host: Option<String>,
+    /// Message ids from Discord exports may be numeric or string.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub message_id: Option<Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub timestamp: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub author: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub path: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub size: Option<u64>,
+    /// Any additional fields merged in by the caller.
+    #[serde(flatten, default)]
+    pub extra: BTreeMap<String, Value>,
+}
+
+/// Append one compact JSON line (UTF-8, like python `ensure_ascii=False`).
+pub fn append_manifest(manifest_path: impl AsRef<Path>, row: &ManifestRow) -> anyhow::Result<()> {
+    let path = manifest_path.as_ref();
+    if let Some(parent) = path.parent() {
+        std::fs::create_dir_all(parent)
+            .with_context(|| format!("creating {}", parent.display()))?;
+    }
+    let mut fh = OpenOptions::new()
+        .create(true)
+        .append(true)
+        .open(path)
+        .with_context(|| format!("opening {}", path.display()))?;
+    fh.write_all(serde_json::to_string(row)?.as_bytes())?;
+    fh.write_all(b"\n")?;
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::fs;
+
+    fn row(path: &str) -> ManifestRow {
+        ManifestRow {
+            channel: "123".into(),
+            url: "https://gofile.io/d/abc".into(),
+            host: Some("gofile".into()),
+            message_id: Some(Value::from(987654)),
+            timestamp: Some("2026-10-09T13:00:00".into()),
+            author: Some("tester".into()),
+            path: Some(path.into()),
+            size: Some(4567),
+            extra: BTreeMap::new(),
+        }
+    }
+
+    #[test]
+    fn appends_jsonl_and_creates_parents() {
+        let dir = tempfile::tempdir().unwrap();
+        let manifest = dir.path().join("a").join("b").join("manifest.jsonl");
+        append_manifest(&manifest, &row("/out/x.7z")).unwrap();
+        append_manifest(&manifest, &row("/out/y.7z")).unwrap();
+
+        let text = fs::read_to_string(&manifest).unwrap();
+        let lines: Vec<&str> = text.lines().collect();
+        assert_eq!(lines.len(), 2);
+        let second: ManifestRow = serde_json::from_str(lines[1]).unwrap();
+        assert_eq!(second.path.as_deref(), Some("/out/y.7z"));
+        assert_eq!(second.channel, "123");
+        assert_eq!(second.message_id, Some(Value::from(987654)));
+    }
+
+    #[test]
+    fn utf8_is_not_ascii_escaped() {
+        let dir = tempfile::tempdir().unwrap();
+        let manifest = dir.path().join("manifest.jsonl");
+        let mut r = row("/out/x.7z");
+        r.author = Some("Zähni".into());
+        append_manifest(&manifest, &r).unwrap();
+        let text = fs::read_to_string(&manifest).unwrap();
+        assert!(text.contains("Zähni"));
+    }
+}

src/state.rs

@@ -0,0 +1,245 @@
+//! Per-run persistent state (state.json) — Rust port of
+//! `dlphelper/state.py`. Makes runs resumable and failures
+//! trackable: re-runs skip URLs with status "ok", failed URLs are retried.
+//!
+//! Schema contract with the python version:
+//!
+//! ```json
+//! { "<url>": { "status": "ok" | "failed" | "error" | ...,
+//!              "attempts": 3, "error": "...", "updated": "2026-10-09T13:00:00",
+//!              "files": [{"path": "...", "size": 123}] } }
+//! ```
+//!
+//! Unknown fields written by other tools survive load+save (serde flatten),
+//! so both versions can consume each other's files.
+
+use std::collections::BTreeMap;
+use std::fs;
+use std::path::PathBuf;
+
+use log::warn;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+use crate::util::{now_local_iso, save_sorted};
+
+/// One downloaded file as recorded in `state.json`.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct FileEntry {
+    pub path: String,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub size: Option<u64>,
+    /// Unknown extra fields, preserved for schema compatibility.
+    #[serde(flatten, default)]
+    pub extra: BTreeMap<String, Value>,
+}
+
+/// Per-URL record in state.json.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct Record {
+    /// "pending" | "ok" | "failed" | "error" — open set, like the python version.
+    #[serde(default)]
+    pub status: String,
+    #[serde(default)]
+    pub attempts: u64,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub error: Option<String>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub files: Vec<FileEntry>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub updated: Option<String>,
+    #[serde(flatten, default)]
+    pub extra: BTreeMap<String, Value>,
+}
+
+impl Record {
+    pub fn is_done(&self) -> bool {
+        self.status == "ok"
+    }
+}
+
+/// JSON-backed state store for one run directory.
+pub struct State {
+    pub path: PathBuf,
+    pub records: BTreeMap<String, Record>,
+}
+
+impl State {
+    /// Load state.json; a corrupt or unreadable file is moved to `*.json.bak`
+    /// and we start fresh (same behavior as the python version).
+    pub fn load(path: impl Into<PathBuf>) -> Self {
+        let path = path.into();
+        if path.exists() {
+            let parsed = fs::read_to_string(&path)
+                .ok()
+                .and_then(|text| serde_json::from_str::<BTreeMap<String, Record>>(&text).ok());
+            if let Some(records) = parsed {
+                return Self { path, records };
+            }
+            let backup = path.with_extension("json.bak");
+            warn!(
+                "Corrupt state file {} ; moving to {} and starting fresh",
+                path.display(),
+                backup.display()
+            );
+            let _ = fs::rename(&path, &backup);
+        }
+        Self {
+            path,
+            records: BTreeMap::new(),
+        }
+    }
+
+    pub fn is_done(&self, url: &str) -> bool {
+        self.records.get(url).is_some_and(Record::is_done)
+    }
+
+    /// The URLs from `urls` whose status is not "ok".
+    #[allow(dead_code)] // python `pending`; kept as a store capability
+    pub fn pending<'a>(&self, urls: &'a [String]) -> Vec<&'a str> {
+        urls.iter()
+            .map(String::as_str)
+            .filter(|u| !self.is_done(u))
+            .collect()
+    }
+
+    /// All URLs whose record is not "ok".
+    pub fn failures(&self) -> Vec<String> {
+        self.records
+            .iter()
+            .filter(|(_, r)| !r.is_done())
+            .map(|(u, _)| u.clone())
+            .collect()
+    }
+
+    /// Create-or-get the record for `url`, bump `attempts`, stamp `updated`,
+    /// apply `f`, then save — mirrors python `mark(url, **fields)`.
+    ///
+    /// Parity note: the python version skips `None` fields, so a value set by
+    /// an earlier mark (e.g. an old error message) is *not* cleared by a later
+    /// mark that doesn't mention it. The closure decides what to set; nothing
+    /// is cleared implicitly.
+    pub fn mark(&mut self, url: &str, f: impl FnOnce(&mut Record)) -> anyhow::Result<&Record> {
+        let fresh = !self.records.contains_key(url);
+        let rec = self.records.entry(url.to_string()).or_default();
+        if fresh && rec.status.is_empty() {
+            rec.status = "pending".to_string();
+        }
+        f(rec);
+        rec.attempts += 1;
+        rec.updated = Some(now_local_iso());
+        self.save()?;
+        Ok(self.records.get(url).expect("entry just inserted"))
+    }
+
+    /// Like [`mark`](Self::mark), but a save failure is logged instead of
+    /// propagated: recording one URL's outcome must never abort the whole run
+    /// (AGENTS.md §6.5). The in-memory record is still updated, so a later
+    /// successful save persists it.
+    pub fn mark_logged(&mut self, url: &str, f: impl FnOnce(&mut Record)) {
+        if let Err(e) = self.mark(url, f) {
+            log::error!("Could not persist state for {url}: {e}");
+        }
+    }
+
+    /// Atomic save: write `*.json.tmp`, then rename over state.json.
+    pub fn save(&self) -> anyhow::Result<()> {
+        save_sorted(&self.path, &self.records)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::fs;
+
+    #[test]
+    fn mark_creates_record_and_increments_attempts() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut st = State::load(dir.path().join("state.json"));
+        st.mark("http://x/1", |r| r.status = "ok".into()).unwrap();
+        let rec = &st.records["http://x/1"];
+        assert_eq!(rec.status, "ok");
+        assert_eq!(rec.attempts, 1);
+        assert_eq!(rec.updated.as_deref().map(str::len), Some(19));
+        assert!(st.is_done("http://x/1"));
+        assert!(st.failures().is_empty());
+    }
+
+    #[test]
+    fn failed_then_ok_keeps_old_error_like_python() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut st = State::load(dir.path().join("state.json"));
+        st.mark("u", |r| {
+            r.status = "failed".into();
+            r.error = Some("boom".into());
+        })
+        .unwrap();
+        // python: mark(url, status="ok", error=None) does NOT clear the error
+        st.mark("u", |r| r.status = "ok".into()).unwrap();
+        let rec = &st.records["u"];
+        assert_eq!(rec.error.as_deref(), Some("boom"));
+        assert_eq!(rec.attempts, 2);
+        assert_eq!(st.failures(), Vec::<String>::new());
+    }
+
+    #[test]
+    fn pending_and_failures() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut st = State::load(dir.path().join("state.json"));
+        st.mark("a", |r| r.status = "ok".into()).unwrap();
+        st.mark("b", |r| r.status = "failed".into()).unwrap();
+        st.mark("c", |_r| {}).unwrap();
+        let urls = vec!["a".into(), "b".into(), "c".into()];
+        assert_eq!(st.pending(&urls), vec!["b", "c"]);
+        assert_eq!(st.failures(), vec!["b".to_string(), "c".to_string()]);
+    }
+
+    #[test]
+    fn corrupt_file_is_backed_up_and_restarted() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("state.json");
+        fs::write(&path, "not json{").unwrap();
+        let st = State::load(&path);
+        assert!(st.records.is_empty());
+        assert!(dir.path().join("state.json.bak").exists());
+        assert!(!path.exists());
+    }
+
+    #[test]
+    fn unknown_fields_survive_roundtrip() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("state.json");
+        fs::write(
+            &path,
+            r#"{"http://x/1": {"status": "ok", "attempts": 3, "custom": 42}}"#,
+        )
+        .unwrap();
+        let st = State::load(&path);
+        assert!(st.is_done("http://x/1"));
+        st.save().unwrap();
+        let v: serde_json::Value =
+            serde_json::from_str(&fs::read_to_string(&path).unwrap()).unwrap();
+        assert_eq!(v["http://x/1"]["custom"], 42);
+        assert_eq!(v["http://x/1"]["attempts"], 3);
+    }
+
+    #[test]
+    fn files_are_recorded_and_persisted() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("state.json");
+        let mut st = State::load(&path);
+        st.mark("u", |r| {
+            r.status = "ok".into();
+            r.files.push(FileEntry {
+                path: "/out/x.7z".into(),
+                size: Some(1234),
+                extra: BTreeMap::new(),
+            });
+        })
+        .unwrap();
+        let reloaded = State::load(&path);
+        assert_eq!(reloaded.records["u"].files.len(), 1);
+        assert_eq!(reloaded.records["u"].files[0].size, Some(1234));
+    }
+}

src/util.rs

@@ -0,0 +1,62 @@
+//! Small shared helpers: timestamps and the atomic sorted-JSON save used by
+//! both state stores (mirrors `json.dumps(..., indent=2, sort_keys=True)` +
+//! tmp-file rename from the python version).
+
+use std::fs;
+use std::path::Path;
+
+use chrono::Local;
+use serde::Serialize;
+
+/// Local time in the exact `"%Y-%m-%dT%H:%M:%S"` format the python version writes.
+pub fn now_local_iso() -> String {
+    Local::now().format("%Y-%m-%dT%H:%M:%S").to_string()
+}
+
+/// Write `value` pretty-printed, then atomically rename over `path`.
+/// The caller must keep keys in a `BTreeMap` so the output is key-sorted,
+/// like the python version's `sort_keys=True`.
+pub fn save_sorted(path: &Path, value: &impl Serialize) -> anyhow::Result<()> {
+    if let Some(parent) = path.parent() {
+        fs::create_dir_all(parent)?;
+    }
+    let tmp = path.with_extension("json.tmp");
+    fs::write(&tmp, serde_json::to_string_pretty(value)?)?;
+    fs::rename(&tmp, path)?;
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::collections::BTreeMap;
+
+    #[test]
+    fn timestamp_format_matches_python() {
+        let t = now_local_iso();
+        assert_eq!(t.len(), 19, "expected YYYY-MM-DDTHH:MM:SS, got {t}");
+        let bytes = t.as_bytes();
+        for (i, b) in bytes.iter().enumerate() {
+            let is_sep = matches!(i, 4 | 7 | 10 | 13 | 16);
+            if is_sep {
+                assert!(!b.is_ascii_digit(), "expected separator at {i} in {t}");
+            } else {
+                assert!(b.is_ascii_digit(), "expected digit at {i} in {t}");
+            }
+        }
+    }
+
+    #[test]
+    fn save_sorted_is_atomic_and_sorted() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("sub").join("x.json");
+        let mut map = BTreeMap::new();
+        map.insert("b", 2);
+        map.insert("a", 1);
+        save_sorted(&path, &map).unwrap();
+        let text = fs::read_to_string(&path).unwrap();
+        assert!(text.find("\"a\"").unwrap() < text.find("\"b\"").unwrap());
+        // tmp file was renamed away
+        assert!(!dir.path().join("sub").join("x.json.tmp").exists());
+    }
+}

src/warhammer_data.rs

@@ -0,0 +1,140 @@
+//! Taxonomy: GW-named systems/factions, GW shop categories as classes,
+//! GROUP_OF_FACTION map — Rust port of `dlphelper/warhammer_data.py`.
+//!
+//! The tables are NOT hand-transcribed: they were dumped from the original
+//! python module into `data/warhammer_data.json` (the committed source of
+//! truth) and are parsed here, so the data stayed byte-identical across the
+//! port.
+//!
+//! Python-dict ordering is preserved exactly (insertion order; a duplicate
+//! key keeps its first insertion position while the *last* value wins),
+//! because the catalogue's "most specific keyword" rule breaks ties by
+//! first occurrence among equal-length keywords.
+
+use std::sync::OnceLock;
+
+use anyhow::Context as _;
+use indexmap::IndexMap;
+use serde::Deserialize;
+
+/// Insertion-ordered pair list with python-dict semantics for building.
+fn dict_insert<V>(vec: &mut Vec<(String, V)>, key: String, value: V) {
+    if let Some(slot) = vec.iter_mut().find(|(k, _)| *k == key) {
+        slot.1 = value;
+    } else {
+        vec.push((key, value));
+    }
+}
+
+/// system -> (faction, keyword aliases), python dict order.
+pub type Systems = Vec<(String, Vec<(String, Vec<String>)>)>;
+
+#[derive(Deserialize)]
+struct Raw {
+    systems: IndexMap<String, IndexMap<String, Vec<String>>>,
+    classes: Vec<(String, Vec<String>)>,
+    group_of_faction: IndexMap<String, String>,
+}
+
+pub struct Data {
+    /// Parsed SYSTEMS table (kept for tests/inspection; matching uses
+    /// `faction_aliases` below).
+    #[allow(dead_code)]
+    pub systems: Systems,
+    /// system -> (alias, faction), insertion order of first occurrence,
+    /// value = last faction seen (python dict semantics).
+    pub faction_aliases: Vec<(String, Vec<(String, String)>)>,
+    /// keyword -> class, insertion order of first occurrence.
+    pub unit_classes: Vec<(String, String)>,
+    pub group_of_faction: IndexMap<String, String>,
+}
+
+impl Data {
+    pub fn global() -> &'static Data {
+        static DATA: OnceLock<Data> = OnceLock::new();
+        DATA.get_or_init(|| Data::load().expect("embedded warhammer_data.json is valid"))
+    }
+
+    /// Parse the embedded JSON dump of the python taxonomy module.
+    fn load() -> anyhow::Result<Data> {
+        let raw: Raw = serde_json::from_str(include_str!("../data/warhammer_data.json"))
+            .context("parsing embedded warhammer_data.json")?;
+
+        // SYSTEMS in dict order
+        let mut systems: Systems = Vec::new();
+        for (system, factions) in &raw.systems {
+            let mut factions_vec: Vec<(String, Vec<String>)> = Vec::new();
+            for (faction, aliases) in factions {
+                dict_insert(&mut factions_vec, faction.clone(), aliases.clone());
+            }
+            dict_insert(&mut systems, system.clone(), factions_vec);
+        }
+
+        // FACTION_ALIASES = {system: {alias: faction ...}} in dict order
+        let mut faction_aliases: Vec<(String, Vec<(String, String)>)> = Vec::new();
+        for (system, factions) in &systems {
+            let mut aliases_vec: Vec<(String, String)> = Vec::new();
+            for (faction, aliases) in factions {
+                for alias in aliases {
+                    dict_insert(&mut aliases_vec, alias.clone(), faction.clone());
+                }
+            }
+            dict_insert(&mut faction_aliases, system.clone(), aliases_vec);
+        }
+
+        // UNIT_CLASSES = {kw: klass for klass, kws in CLASSES for kw in kws}
+        let mut unit_classes: Vec<(String, String)> = Vec::new();
+        for (klass, keywords) in &raw.classes {
+            for kw in keywords {
+                dict_insert(&mut unit_classes, kw.clone(), klass.clone());
+            }
+        }
+
+        Ok(Data {
+            systems,
+            faction_aliases,
+            unit_classes,
+            group_of_faction: raw.group_of_faction,
+        })
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn tables_load_and_are_complete() {
+        let d = Data::global();
+        assert!(d.systems.iter().any(|(s, _)| s == "Warhammer 40,000"));
+        assert!(d.systems.iter().any(|(s, _)| s == "Middle-Earth"));
+        let f40k = &d
+            .systems
+            .iter()
+            .find(|(s, _)| s == "Warhammer 40,000")
+            .unwrap()
+            .1;
+        assert!(f40k.iter().any(|(f, _)| f == "Space Wolves"));
+        // "magnus" is listed in both Monster and Hero -> last (Hero) wins
+        let magnus = d
+            .unit_classes
+            .iter()
+            .find(|(kw, _)| kw == "magnus")
+            .map(|(_, c)| c.as_str());
+        assert_eq!(magnus, Some("Hero"));
+        // duplicate alias inside a system keeps first position, last value
+        let aliases = &d
+            .faction_aliases
+            .iter()
+            .find(|(s, _)| s == "Warhammer 40,000")
+            .unwrap()
+            .1;
+        let idx = aliases.iter().position(|(a, _)| a == "skull lord").unwrap();
+        assert_eq!(aliases[idx].1, "World Eaters");
+        // position kept from FIRST insertion (Chaos Daemons block): it sits
+        // between 'slaanesh' and 'shalaxi helbane' (verified against python)
+        assert_eq!(aliases[idx - 1].0, "slaanesh");
+        assert_eq!(aliases[idx + 1].0, "shalaxi helbane");
+        assert_eq!(d.group_of_faction["Space Wolves"], "Space Marines");
+    }
+}

src/zero807.rs

@@ -0,0 +1,519 @@
+//! curl-based downloader for 0807.st direct links — Rust port of
+//! `dlphelper/zero807.py`.
+//!
+//! 0807.st serves the files directly but rejects requests that don't look
+//! like a browser (requires a User-Agent and the Referer of the site page),
+//! and it rejects multi-connection/range abuse patterns differently than
+//! curl — which is why curl is used instead of aria2c here (see python
+//! module docstring: curl worked, aria2c's default request got 403).
+//!
+//! Downloads go to a `.part` file first and are renamed on success, so an
+//! interrupted download is resumable (`--continue-at -`) and partial files
+//! are never mistaken for finished ones.
+
+use std::fs;
+use std::io::Read;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+pub const USER_AGENT: &str = "Mozilla/5.0";
+pub const REFERER: &str = "https://0807.st/";
+
+/// Error type mirroring the python exception classes. `NotFound` (dead link,
+/// HTTP 404/410) is a final state the orchestrator never retries.
+#[allow(dead_code)] // used by the orchestrator, not yet wired up
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Zero807Error {
+    /// python `Zero807Error` (base): transient/unclassified failure.
+    Error(String),
+    /// python `Zero807NotFound`: dead link, never retry.
+    NotFound(String),
+}
+
+impl std::fmt::Display for Zero807Error {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Zero807Error::Error(m) | Zero807Error::NotFound(m) => write!(f, "{m}"),
+        }
+    }
+}
+
+impl std::error::Error for Zero807Error {}
+
+/// Derive the filename from the URL path (e.g. /6saCNie.zip -> 6saCNie.zip).
+///
+/// Matches python `Path(urlparse(url).path).name`: the query string is not
+/// part of the path, a trailing slash still yields the last real segment
+/// (`.../a/b/` → `b`), and an empty result is an error.
+pub fn target_name(url: &str) -> Result<String, Zero807Error> {
+    let path = match url::Url::parse(url) {
+        Ok(u) => u.path().to_string(),
+        // python urlparse is lenient about a missing scheme but still strips
+        // query/fragment; mirror that.
+        Err(_) => url.split(['?', '#']).next().unwrap_or(url).to_string(),
+    };
+    let name = Path::new(&path)
+        .file_name()
+        .map(|n| n.to_string_lossy().into_owned())
+        .unwrap_or_default();
+    if name.is_empty() {
+        return Err(Zero807Error::Error(format!(
+            "cannot derive a filename from URL: {url}"
+        )));
+    }
+    Ok(name)
+}
+
+/// Download one URL with curl; returns (final_path, size_bytes).
+///
+/// Defaults match python: `timeout_seconds = 7200`, `resume = true`.
+/// Use `download_with` for explicit parameters.
+pub fn download(url: &str, out_dir: &Path) -> Result<(PathBuf, u64), Zero807Error> {
+    download_with(url, out_dir, 7200, true)
+}
+
+/// Same with explicit timeout/resume parameters (python keyword args).
+pub fn download_with(
+    url: &str,
+    out_dir: &Path,
+    timeout_seconds: u64,
+    resume: bool,
+) -> Result<(PathBuf, u64), Zero807Error> {
+    let name = target_name(url)?;
+    let dest = out_dir.join(&name);
+    let size = download_to(url, &dest, timeout_seconds, resume)?;
+    Ok((dest, size))
+}
+
+/// Download `url` to an explicit `dest` (`.part` + rename, resume, skip if
+/// already present). `download_with` derives `dest` from the URL; the folder
+/// downloader derives it from the page's file name.
+pub fn download_to(
+    url: &str,
+    dest: &Path,
+    timeout_seconds: u64,
+    resume: bool,
+) -> Result<u64, Zero807Error> {
+    let name = dest
+        .file_name()
+        .map(|n| n.to_string_lossy().into_owned())
+        .unwrap_or_default();
+    if let Some(parent) = dest.parent() {
+        fs::create_dir_all(parent)
+            .map_err(|e| Zero807Error::Error(format!("cannot create {}: {e}", parent.display())))?;
+    }
+    let tmp = dest.with_file_name(format!("{name}.part"));
+
+    if dest.exists() {
+        let size = fs::metadata(dest).map(|m| m.len()).unwrap_or(0);
+        if size > 0 {
+            log::info!("Skip existing: {}", dest.display());
+            return Ok(size);
+        }
+    }
+
+    let mut cmd = Command::new("curl");
+    cmd.args([
+        "-fSL",
+        "--connect-timeout",
+        "20",
+        "--max-time",
+        &timeout_seconds.to_string(),
+        "--retry",
+        "2",
+        "--retry-delay",
+        "5",
+        "-A",
+        USER_AGENT,
+        "-e",
+        REFERER,
+        "-o",
+    ]);
+    cmd.arg(&tmp);
+    if resume {
+        cmd.arg("--continue-at").arg("-");
+    }
+    cmd.arg(url);
+
+    log::info!("Downloading {} -> {}", url, name);
+    let proc = cmd.output().map_err(|e| {
+        // python: subprocess raises FileNotFoundError, which the
+        // orchestrator's catch-all would record; wrap it here.
+        Zero807Error::Error(format!("failed to run curl: {e}"))
+    })?;
+
+    if proc.status.code() != Some(0) {
+        let returncode = proc.status.code().unwrap_or(-1);
+        let stderr = String::from_utf8_lossy(&proc.stderr);
+        let tail: Vec<&str> = stderr
+            .trim()
+            .lines()
+            .filter(|line| !line.is_empty())
+            .collect();
+        let start = tail.len().saturating_sub(3);
+        let stderr = tail[start..].join(" | ");
+        // -f makes curl exit 22 on any HTTP error; dead links fail for good
+        if stderr.contains("404") || stderr.contains("410") {
+            return Err(Zero807Error::NotFound(format!(
+                "dead link (HTTP 404/410): {stderr}"
+            )));
+        }
+        return Err(Zero807Error::Error(format!(
+            "curl exit code {returncode}: {stderr}"
+        )));
+    }
+
+    let size = fs::metadata(&tmp).map(|m| m.len()).unwrap_or(0);
+    if size == 0 {
+        let _ = fs::remove_file(&tmp); // python unlink(missing_ok=True)
+        return Err(Zero807Error::Error(format!("empty file for {name}")));
+    }
+    if dest
+        .extension()
+        .map(|e| e.to_string_lossy().to_lowercase())
+        .as_deref()
+        == Some("zip")
+    {
+        // python reads the whole file just to peek at 2 bytes; reading only
+        // the first two bytes is behavior-identical and cheap.
+        let mut first = [0u8; 2];
+        let n = fs::File::open(&tmp)
+            .and_then(|mut fh| fh.read(&mut first))
+            .unwrap_or(0);
+        if first[..n] != *b"PK" {
+            log::warn!("{name} does not look like a ZIP file (first bytes are not PK)");
+        }
+    }
+    // os.replace semantics: rename over an existing destination.
+    fs::rename(&tmp, dest)
+        .map_err(|e| Zero807Error::Error(format!("cannot move {name} into place: {e}")))?;
+    Ok(size)
+}
+
+/// A file listed on a 0807.st folder page (`/f/<id>`).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FolderEntry {
+    /// file id used in the direct-download URL (`https://0807.st/<id>?dl=1`)
+    pub id: String,
+    /// display file name from the page (`<span class="alname">`)
+    pub name: String,
+    pub size: u64,
+    pub mime: String,
+}
+
+/// The `<id>` of a 0807.st folder page URL (`/f/<id>`), if it is one.
+/// A direct file link (`/x.zip`) or the folder export (`/f/<id>.zip`) is not.
+pub fn folder_page_id(url: &str) -> Option<String> {
+    let path = match url::Url::parse(url) {
+        Ok(u) => u.path().to_string(),
+        Err(_) => url.split(['?', '#']).next().unwrap_or(url).to_string(),
+    };
+    let rest = path.trim_end_matches('/').strip_prefix("/f/")?;
+    if rest.is_empty() || rest.contains('/') || rest.contains('.') {
+        return None;
+    }
+    Some(rest.to_string())
+}
+
+/// True when `url` points at a 0807.st folder page.
+pub fn is_folder_page(url: &str) -> bool {
+    folder_page_id(url).is_some()
+}
+
+/// Parse the file list from a 0807.st folder page. Each entry is an
+/// `<a class="alitem" ... data-id="..." data-size="..." data-mime="...">`
+/// whose name sits in `<span class="alname">...</span>`. Pure string scanning
+/// (no HTML-parser dependency).
+pub fn parse_folder_page(html: &str) -> Vec<FolderEntry> {
+    const MARK: &str = "<a class=\"alitem\"";
+    let mut out = Vec::new();
+    let mut rest = html;
+    while let Some(pos) = rest.find(MARK) {
+        rest = &rest[pos + MARK.len()..];
+        let end = rest.find("</a>").unwrap_or(rest.len());
+        let seg = &rest[..end];
+        if let Some(id) = attr(seg, "data-id") {
+            let size = attr(seg, "data-size")
+                .and_then(|s| s.parse::<u64>().ok())
+                .unwrap_or(0);
+            let mime = attr(seg, "data-mime").unwrap_or_default();
+            let name = span_text(seg, "alname").unwrap_or_else(|| id.clone());
+            out.push(FolderEntry {
+                id,
+                name,
+                size,
+                mime,
+            });
+        }
+        rest = &rest[end..];
+    }
+    out
+}
+
+/// Value of `key="..."` inside a tag fragment.
+fn attr(seg: &str, key: &str) -> Option<String> {
+    let needle = format!("{key}=\"");
+    let start = seg.find(&needle)? + needle.len();
+    let end = seg[start..].find('"')? + start;
+    Some(seg[start..end].to_string())
+}
+
+/// Text of `<span class="class">...</span>` (HTML entities decoded).
+fn span_text(seg: &str, class: &str) -> Option<String> {
+    let needle = format!("<span class=\"{class}\">");
+    let start = seg.find(&needle)? + needle.len();
+    let end = seg[start..].find("</span>")? + start;
+    Some(html_unescape(&seg[start..end]))
+}
+
+/// Decode the handful of HTML entities that can appear in file names.
+fn html_unescape(s: &str) -> String {
+    s.replace("&amp;", "&")
+        .replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", "\"")
+        .replace("&#39;", "'")
+        .replace("&#x27;", "'")
+}
+
+/// A safe single path segment for a file name taken from a page.
+fn sanitize_filename(name: &str) -> String {
+    let cleaned: String = name
+        .chars()
+        .map(|c| {
+            if c == '/' || c == '\\' || (c as u32) < 0x20 {
+                '_'
+            } else {
+                c
+            }
+        })
+        .collect();
+    let cleaned = cleaned.trim().trim_start_matches('.').to_string();
+    if cleaned.is_empty() {
+        "unnamed".into()
+    } else {
+        cleaned
+    }
+}
+
+/// Fetch a folder page's HTML via curl (browser-like headers, like the file
+/// downloads). Writes to a temp file because curl is used throughout.
+fn fetch_folder_html(url: &str, timeout_seconds: u64) -> Result<String, Zero807Error> {
+    let tmp =
+        std::env::temp_dir().join(format!("dlphelper-0807-folder-{}.html", std::process::id()));
+    let mut cmd = Command::new("curl");
+    cmd.args([
+        "-fSL",
+        "--connect-timeout",
+        "20",
+        "--max-time",
+        &timeout_seconds.to_string(),
+        "-A",
+        USER_AGENT,
+        "-e",
+        REFERER,
+        "-o",
+    ]);
+    cmd.arg(&tmp);
+    cmd.arg(url);
+    let proc = cmd
+        .output()
+        .map_err(|e| Zero807Error::Error(format!("failed to run curl: {e}")))?;
+    if proc.status.code() != Some(0) {
+        let stderr = String::from_utf8_lossy(&proc.stderr);
+        let tail: Vec<&str> = stderr.trim().lines().filter(|l| !l.is_empty()).collect();
+        let start = tail.len().saturating_sub(3);
+        let msg = tail[start..].join(" | ");
+        let _ = fs::remove_file(&tmp);
+        if msg.contains("404") || msg.contains("410") {
+            return Err(Zero807Error::NotFound(format!(
+                "dead folder page (HTTP 404/410): {msg}"
+            )));
+        }
+        return Err(Zero807Error::Error(format!("curl exit for page: {msg}")));
+    }
+    let text = fs::read_to_string(&tmp)
+        .map_err(|e| Zero807Error::Error(format!("reading folder page: {e}")))?;
+    let _ = fs::remove_file(&tmp);
+    Ok(text)
+}
+
+/// Download every file on a 0807.st folder page into `out_dir/<page-id>/`.
+///
+/// Returns `(folder, [(path, size), ...])`. Each file is fetched from
+/// `https://0807.st/<data-id>?dl=1` (a 302 to the CDN). Already-present files
+/// are skipped, so the caller's retry loop only fetches what is missing; a
+/// partial failure is reported as an error so that retry happens.
+pub fn download_folder(
+    url: &str,
+    out_dir: &Path,
+    timeout_seconds: u64,
+    resume: bool,
+) -> Result<(PathBuf, Vec<(PathBuf, u64)>), Zero807Error> {
+    let id = folder_page_id(url)
+        .ok_or_else(|| Zero807Error::Error(format!("not a folder page: {url}")))?;
+    let html = fetch_folder_html(url, timeout_seconds)?;
+    let entries = parse_folder_page(&html);
+    if entries.is_empty() {
+        return Err(Zero807Error::Error(format!(
+            "folder page {url} lists no downloadable files"
+        )));
+    }
+    let folder = out_dir.join(&id);
+    fs::create_dir_all(&folder)
+        .map_err(|e| Zero807Error::Error(format!("cannot create {}: {e}", folder.display())))?;
+    log::info!("Folder page {url}: {} files", entries.len());
+    let mut files: Vec<(PathBuf, u64)> = Vec::new();
+    let mut failed: Vec<String> = Vec::new();
+    for e in &entries {
+        let name = sanitize_filename(&e.name);
+        let dest = folder.join(&name);
+        let direct = format!("https://0807.st/{}?dl=1", e.id);
+        match download_to(&direct, &dest, timeout_seconds, resume) {
+            Ok(size) => {
+                log::info!("OK folder file {name} ({})", human_size(size));
+                files.push((dest, size));
+            }
+            Err(err) => {
+                log::error!("Folder file {name} failed: {err}");
+                failed.push(format!("{name}: {err}"));
+            }
+        }
+    }
+    if !failed.is_empty() {
+        return Err(Zero807Error::Error(format!(
+            "{}/{} folder files failed: {}",
+            failed.len(),
+            entries.len(),
+            failed.join("; ")
+        )));
+    }
+    Ok((folder, files))
+}
+
+/// Compact human-readable size (kept local so this module stays self-contained).
+fn human_size(bytes: u64) -> String {
+    const UNITS: [&str; 5] = ["B", "KiB", "MiB", "GiB", "TiB"];
+    let mut v = bytes as f64;
+    let mut u = 0;
+    while v >= 1024.0 && u < UNITS.len() - 1 {
+        v /= 1024.0;
+        u += 1;
+    }
+    if u == 0 {
+        format!("{bytes} B")
+    } else {
+        format!("{v:.1} {}", UNITS[u])
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn target_name_from_path_last_segment() {
+        assert_eq!(
+            target_name("https://0807.st/6saCNie.zip").unwrap(),
+            "6saCNie.zip"
+        );
+        assert_eq!(
+            target_name("https://0807.st/a/b/filename.bin").unwrap(),
+            "filename.bin"
+        );
+    }
+
+    #[test]
+    fn target_name_drops_query_and_fragment() {
+        // python takes Path(urlparse(url).path).name: query/fragment dropped
+        assert_eq!(
+            target_name("https://0807.st/file.zip?x=1").unwrap(),
+            "file.zip"
+        );
+        assert_eq!(
+            target_name("https://0807.st/file.zip?x=1#frag").unwrap(),
+            "file.zip"
+        );
+        // schemeless URLs parse like python urlparse (query still stripped)
+        assert_eq!(target_name("0807.st/file.zip?x=1").unwrap(), "file.zip");
+    }
+
+    #[test]
+    fn target_name_trailing_slash_matches_pathlib() {
+        // pathlib: Path("/a/b/").name == "b" (trailing slash ignored)
+        assert_eq!(target_name("https://0807.st/a/b/").unwrap(), "b");
+        assert_eq!(
+            target_name("https://0807.st/").unwrap_err().to_string(),
+            "cannot derive a filename from URL: https://0807.st/"
+        );
+        assert_eq!(
+            target_name("https://0807.st").unwrap_err().to_string(),
+            "cannot derive a filename from URL: https://0807.st"
+        );
+    }
+
+    #[test]
+    fn error_display_is_the_message() {
+        assert_eq!(
+            Zero807Error::NotFound("dead link (HTTP 404/410): 404".into()).to_string(),
+            "dead link (HTTP 404/410): 404"
+        );
+        assert_eq!(Zero807Error::Error("boom".into()).to_string(), "boom");
+    }
+
+    #[test]
+    fn folder_page_detection() {
+        assert_eq!(
+            folder_page_id("https://0807.st/f/Df8A5sn3").as_deref(),
+            Some("Df8A5sn3")
+        );
+        assert_eq!(
+            folder_page_id("https://0807.st/f/MM5AWQ9R/").as_deref(),
+            Some("MM5AWQ9R")
+        );
+        assert!(is_folder_page("https://0807.st/f/Df8A5sn3"));
+        // a direct file link is not a folder page
+        assert!(!is_folder_page("https://0807.st/6saCNie.zip"));
+        // the folder export (.zip) is not a folder page either
+        assert!(!is_folder_page("https://0807.st/f/Df8A5sn3.zip"));
+        // deeper paths are not folder pages
+        assert!(!is_folder_page("https://0807.st/f/a/b"));
+        assert!(!is_folder_page("https://0807.st/"));
+    }
+
+    #[test]
+    fn parse_folder_page_reads_alitem_entries() {
+        let html = r#"<div class="allist">
+            <a class="alitem" href="/p/9z5SsgV" data-id="9z5SsgV" data-rel="" data-size="1397446559" data-mime="application/zip"><span class="althumbwrap"><span class="alicon"><svg></svg></span></span><span class="alinfo"><span class="alname">13. Terrain.zip</span><span class="alsize">1.3 GB</span></span></a>
+            <a class="alitem" href="/p/wNuEQcp" data-id="wNuEQcp" data-rel="" data-size="2295545985" data-mime="application/zip"><span class="alname">Orks &amp; Goblins.zip</span></a>
+        </div>"#;
+        let got = parse_folder_page(html);
+        assert_eq!(got.len(), 2);
+        assert_eq!(got[0].id, "9z5SsgV");
+        assert_eq!(got[0].name, "13. Terrain.zip");
+        assert_eq!(got[0].size, 1397446559);
+        assert_eq!(got[0].mime, "application/zip");
+        assert_eq!(got[1].id, "wNuEQcp");
+        assert_eq!(got[1].name, "Orks & Goblins.zip"); // entity decoded
+        assert_eq!(got[1].size, 2295545985);
+    }
+
+    #[test]
+    fn parse_folder_page_ignores_non_alitem_anchors() {
+        let html = r#"<a href="/" class="x">home</a><a class="alitem" href="/p/abc" data-id="abc" data-size="5" data-mime="text/plain"><span class="alname">a.txt</span></a>"#;
+        let got = parse_folder_page(html);
+        assert_eq!(got.len(), 1);
+        assert_eq!(got[0].id, "abc");
+        assert_eq!(got[0].name, "a.txt");
+    }
+
+    #[test]
+    fn sanitize_filename_blocks_traversal() {
+        assert_eq!(sanitize_filename("13. Terrain.zip"), "13. Terrain.zip");
+        let s = sanitize_filename("../../etc/passwd");
+        assert!(!s.contains('/') && !s.contains('\\'));
+        assert!(!s.starts_with('.'));
+        assert_eq!(sanitize_filename("a\\b"), "a_b");
+        assert_eq!(sanitize_filename("   "), "unnamed");
+    }
+}

tests/test_catalogue.py

@@ -1,62 +0,0 @@
-#!/usr/bin/env python3
-"""Offline classifier test for the catalogue (no network, no deps).
-
-Run with: python3 tests/test_catalogue.py
-"""
-
-from __future__ import annotations
-
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dlphelper.catalogue import classify  # noqa: E402
-
-CASES = [
-    # (filename, expected system, expected faction, expected class or None-any)
-    # GW naming: chapters are factions, classes are GW unit types.
-    ("Thunderwolf Cavalry.7z", "Warhammer 40,000", "Space Wolves", "Mounted"),
-    ("Wulfen.zip", "Warhammer 40,000", "Space Wolves", "Infantry"),
-    ("Keeper of Secrets & Shalaxi Helbane.7z", "Warhammer 40,000", "Chaos Daemons", "Hero"),
-    ("Assorted Drones.zip", "Warhammer 40,000", "T'au Empire", None),
-    ("Commander Shadowsun.zip", "Warhammer 40,000", "T'au Empire", "Hero"),
-    ("Devilfish.zip", "Warhammer 40,000", "T'au Empire", "Vehicle"),
-    ("Battle Sister Scanz.zip", "Warhammer 40,000", "Adepta Sororitas", None),
-    ("Knights.zip", None, None, None),  # ambiguous keyword alone: unclassified
-    ("Khorne Demon Enginez and skull lordz.zip", "Warhammer 40,000", "Chaos Daemons", None),
-    ("Cultists.zip", "Warhammer 40,000", "Chaos Space Marines", "Battleline"),
-    ("Da Troopz.zip", "Warhammer 40,000", "Orks", None),
-    ("Seraphon.zip", "Age of Sigmar", "Seraphon", None),
-    ("Leviathan Pattern Dreadnought.zip", "The Horus Heresy", "Space Marine Legions", "Walker"),
-    ("Poxwalkers.zip", "Warhammer 40,000", "Death Guard", "Infantry"),
-    ("Kabalite Warriors.zip", "Warhammer 40,000", "Drukhari", "Battleline"),
-    ("Canoptek Wraiths.zip", "Warhammer 40,000", "Necrons", "Infantry"),
-    ("random thing with no keywords.zip", None, None, None),
-]
-
-
-def main() -> None:
-    failures = 0
-    for name, exp_system, exp_faction, exp_class in CASES:
-        system, faction, klass = classify(name)
-        problems = []
-        if exp_system and system != exp_system:
-            problems.append(f"system={system!r} != {exp_system!r}")
-        if exp_faction and faction != exp_faction:
-            problems.append(f"faction={faction!r} != {exp_faction!r}")
-        if exp_class and klass != exp_class:
-            problems.append(f"class={klass!r} != {exp_class!r}")
-        if problems:
-            failures += 1
-            print(f"FAIL {name!r}: {', '.join(problems)}")
-        else:
-            print(f"ok   {name!r} -> {system} / {faction} / {klass}")
-    if failures:
-        print(f"\n{failures} failures")
-        sys.exit(1)
-    print("\ntest_catalogue: OK")
-
-
-if __name__ == "__main__":
-    main()

tests/test_extract.py

@@ -1,68 +0,0 @@
-#!/usr/bin/env python3
-"""Offline extraction test with a synthetic export.json (no network, no deps).
-
-Run with: python3 tests/test_extract.py
-"""
-
-from __future__ import annotations
-
-import base64
-import json
-import sys
-import tempfile
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dlphelper.discord import classify, extract_urls  # noqa: E402
-
-
-def b64(s: str) -> str:
-    return base64.b64encode(s.encode()).decode()
-
-
-def main() -> None:
-    messages = [
-        {
-            "id": "1",
-            "timestamp": "2026-01-01T00:00:00",
-            "author": {"name": "poster"},
-            "content": f"chaos stuff: {b64('https://gofile.io/d/abc123')}",
-        },
-        {
-            "id": "2",
-            "timestamp": "2026-01-02T00:00:00",
-            "author": {"name": "poster"},
-            "content": (
-                f"0807 zip: {b64('https://0807.st/6saCNie.zip')}\n"
-                f"dupe of 1: {b64('https://gofile.io/d/abc123')}\n"
-                f"plain: https://0807.st/plainLink.zip and "
-                f"https://discord.com/channels/1/2 (unsupported)\n"
-                f"junk: {b64('https://pixeldrain.com/u/xyz')} "
-                "and not-our-domain base64: aHR0cHM6Ly9leGFtcGxlLm9yZw=="
-            ),
-        },
-    ]
-
-    with tempfile.TemporaryDirectory() as tmp:
-        export = Path(tmp) / "export.json"
-        export.write_text(json.dumps({"messages": messages}), encoding="utf-8")
-        found = extract_urls(export)
-
-    urls = {f.url for f in found}
-    expected = {"https://gofile.io/d/abc123", "https://0807.st/6saCNie.zip",
-                "https://0807.st/plainLink.zip"}
-    assert urls == expected, f"unexpected urls: {sorted(urls)}"
-
-    by_url = {f.url: f for f in found}
-    g = by_url["https://gofile.io/d/abc123"]
-    assert g.host == "gofile" and g.message_id == "1" and g.author == "poster"
-    assert classify("https://0807.st/x.zip") == "0807"
-    assert classify("https://gofile.io/d/x") == "gofile"
-    assert classify("https://example.org/x") is None
-
-    print("test_extract: OK")
-
-
-if __name__ == "__main__":
-    main()

tools/debug_gofile.py

@@ -1,68 +0,0 @@
-#!/usr/bin/env python3
-"""One-off diagnostic: inspect the raw gofile API response for one content id.
-
-Prints everything except the account token. Creates exactly one guest account.
-"""
-
-import hashlib
-import json
-import sys
-import time
-
-import requests
-
-CID = sys.argv[1]
-
-USER_AGENT = "Mozilla/5.0"
-LOCALE = "en-US"
-TOKEN_SECRET = "12af056dacea0b"
-WINDOW_SEC = 14400
-COMMON = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip", "Accept": "*/*",
-          "Connection": "keep-alive"}
-
-resp = requests.post("https://api.gofile.io/accounts", headers=COMMON, timeout=15)
-data = resp.json()
-assert data.get("status") == "ok", data
-token = data["data"]["token"]
-
-window = int(time.time()) // WINDOW_SEC
-seed = f"{USER_AGENT}::{LOCALE}::{token}::{window}::{TOKEN_SECRET}"
-wtoken = hashlib.sha256(seed.encode()).hexdigest()
-headers = {
-    **COMMON,
-    "Authorization": f"Bearer {token}",
-    "X-Website-Token": wtoken,
-    "X-BL": LOCALE,
-}
-
-r = requests.get(
-    f"https://api.gofile.io/contents/{CID}?cache=true&sortField=createTime&sortDirection=1",
-    headers=headers, timeout=30,
-)
-print("HTTP", r.status_code)
-body = r.json()
-print("api status:", body.get("status"))
-
-
-def scrub(node):
-    """Remove anything token-like from the tree before printing."""
-    if isinstance(node, dict):
-        return {
-            k: ("<redacted>" if k in ("token", "accountToken") else scrub(v))
-            for k, v in node.items()
-        }
-    if isinstance(node, list):
-        return [scrub(x) for x in node]
-    return node
-
-
-d = body.get("data") or {}
-print("data keys:", sorted(d.keys()))
-print("type:", d.get("type"))
-children = d.get("children") or {}
-print("children count:", len(children))
-for cid, child in list(children.items())[:5]:
-    print("  child:", cid, "type:", child.get("type"), "name:", child.get("name"),
-          "has link:", "link" in child)
-print()
-print(json.dumps(scrub(body), indent=2)[:3000])

tools/gw_browse_test.py

@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-"""One-shot browse of the GW Algolia index -> gwshop_raw.json (gitignored)."""
-import json
-import logging
-import pathlib
-import sys
-
-sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))
-
-import dlphelper.gwshop as gw
-
-logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-
-app, key = gw.load_credentials()
-client = gw.GwShopClient(app, key)
-hits = client.browse("prod-lazarus-product-en-ww", hits_per_page=100)
-print("total hits", len(hits))
-pathlib.Path("gwshop_raw.json").write_text(
-    json.dumps(hits, ensure_ascii=False), encoding="utf-8"
-)
-print("written gwshop_raw.json")

tools/gw_build_catalog.py

@@ -1,42 +0,0 @@
-#!/usr/bin/env python3
-"""gwshop_raw.json -> data/gwshop_catalog.json + quick shape report."""
-import json
-import logging
-import pathlib
-import sys
-from collections import Counter
-
-sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))
-
-import dlphelper.gwshop as gw
-
-logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-
-hits = json.loads(pathlib.Path("gwshop_raw.json").read_text(encoding="utf-8"))
-print("raw hits:", len(hits))
-unique = len({h.get("objectID") for h in hits})
-print("unique objectIDs:", unique)
-
-catalog = gw.build_catalog(hits)
-pathlib.Path("data/gwshop_catalog.json").write_text(
-    json.dumps(catalog, ensure_ascii=False, indent=1), encoding="utf-8"
-)
-print("entries:", catalog["generated_entries"])
-
-entries = catalog["entries"]
-print("no system:", sum(1 for e in entries if e["system"] is None))
-print("by system:", Counter(e["system"] for e in entries).most_common())
-print("40k groups:", Counter(
-    e["group"] for e in entries if e["system"] == "Warhammer 40,000"
-).most_common())
-print("lvl depth:", Counter(
-    len((e.get("gw_paths") or [[]])[0]) for e in entries
-).most_common())
-print("classes (lvl>=4):", Counter(
-    e["class"] for e in entries if e["class"]
-).most_common(12))
-sample = next(
-    e for e in entries
-    if e["name"].startswith("Codex: Adeptus Custodes")
-)
-print(json.dumps(sample, indent=1, ensure_ascii=False)[:900])

tools/gw_inspect.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python3
-"""Inspect gwshop catalog shape: Space Marines chapters, AoS, dedupe stats."""
-import json
-import pathlib
-from collections import Counter
-
-cat = json.loads(pathlib.Path("data/gwshop_catalog.json").read_text(encoding="utf-8"))
-entries = cat["entries"]
-
-def show(pred, label, n=6):
-    print(f"--- {label}")
-    shown = 0
-    for e in entries:
-        if pred(e):
-            p = " > ".join(x for x in (e["system"], e["group"], e["faction"], e["subfaction"], e["class"]) if x)
-            print(f"  {e['name'][:60]!r}  ->  {p}")
-            shown += 1
-            if shown >= n:
-                break
-
-show(lambda e: e["group"] == "Space Marines", "Space Marines group samples")
-show(lambda e: e["group"] == "Armies of the Imperium" and e["subfaction"], "Imperium with subfaction")
-show(lambda e: e["system"] == "Age of Sigmar", "Age of Sigmar samples", 5)
-show(lambda e: e["system"] == "The Old World", "Old World samples", 5)
-show(lambda e: e["system"] == "Other Games", "Other Games samples", 5)
-show(lambda e: e["group"] == "Unit Type", "Unit Type branch", 6)
-
-names = Counter(e["name"].casefold() for e in entries)
-dups = {n: c for n, c in names.items() if c > 1}
-print("duplicate names:", len(dups), "example:", list(dups.items())[:3])
-
-# faction-level value census for 40k
-print("--- 40k factions census")
-print(Counter(e["faction"] for e in entries if e["system"] == "Warhammer 40,000").most_common(40))

tools/new_repo.py

@@ -1,108 +0,0 @@
-#!/usr/bin/env python3
-"""Create or list repositories on the Gitea instance at git.dieterunterstrich.de.
-
-Credentials are read from ~/.git-credentials (the line for that host); the
-secret is never printed. Useful when you need a new repo without touching the
-web UI.
-
-Usage:
-    python3 tools/new_repo.py list
-    python3 tools/new_repo.py create <name> [--private] [--desc "..."]
-"""
-
-from __future__ import annotations
-
-import argparse
-import base64
-import json
-import sys
-from pathlib import Path
-from urllib.request import Request, urlopen
-
-HOST = "git.dieterunterstrich.de"
-API = f"https://{HOST}/api/v1"
-CREDENTIALS_FILE = Path.home() / ".git-credentials"
-
-
-def load_credentials() -> tuple[str, str]:
-    """Return (username, password) for the Gitea host from ~/.git-credentials."""
-    for line in CREDENTIALS_FILE.read_text().splitlines():
-        line = line.strip()
-        if not line.startswith("https://") or HOST not in line:
-            continue
-        # https://user:pass@host
-        rest = line.removeprefix("https://")
-        userinfo = rest.split("@", 1)[0]
-        user, _, password = userinfo.partition(":")
-        return user, password
-    raise SystemExit(f"No credentials for {HOST} in {CREDENTIALS_FILE}")
-
-
-def api(method: str, path: str, body: dict | None = None) -> tuple[int, dict]:
-    """Call the Gitea API; returns (status_code, parsed_json)."""
-    user, password = load_credentials()
-    auth = base64.b64encode(f"{user}:{password}".encode()).decode()
-    data = json.dumps(body).encode() if body is not None else None
-    req = Request(
-        f"{API}{path}",
-        data=data,
-        method=method,
-        headers={
-            "Authorization": f"Basic {auth}",
-            "Content-Type": "application/json",
-        },
-    )
-    try:
-        resp = urlopen(req, timeout=30)
-        return resp.status, json.loads(resp.read() or b"{}")
-    except Exception as exc:  # noqa: BLE001
-        # HTTPError carries the response body, useful for API errors
-        code = getattr(exc, "code", None)
-        if code is not None:
-            try:
-                return code, json.loads(exc.read() or b"{}")
-            except Exception:  # noqa: BLE001
-                return code, {"message": str(exc)}
-        raise
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(description="Gitea repo helper")
-    sub = parser.add_subparsers(dest="command", required=True)
-
-    sub.add_parser("list")
-
-    p_create = sub.add_parser("create")
-    p_create.add_argument("name")
-    p_create.add_argument("--desc", default=None)
-    p_create.add_argument("--public", action="store_true")
-
-    args = parser.parse_args()
-
-    if args.command == "list":
-        status, repos = api("GET", "/user/repos?limit=100")
-        print(f"HTTP {status}")
-        for repo in repos:
-            print(f"  {repo['full_name']}  (private={repo['private']})")
-        return
-
-    status, result = api(
-        "POST",
-        "/user/repos",
-        {
-            "name": args.name,
-            "description": args.desc or "",
-            "private": not args.public,
-            "auto_init": False,
-            "default_branch": "main",
-        },
-    )
-    if status in (201, 200):
-        print(f"Created: {result['full_name']} -> {result['clone_url']}")
-    else:
-        print(f"Failed (HTTP {status}): {result.get('message', result)}")
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()

tools/plan_diff.py

@@ -1,26 +0,0 @@
-#!/usr/bin/env python3
-"""Diff old vs new sorter plans by source path -> target."""
-import json
-import sys
-
-changed = 0
-same = 0
-examples = []
-for n in (1, 2, 3):
-    old = {p["source"]: p for p in json.load(open(f"/tmp/plan{n}.json"))}
-    new = {p["source"]: p for p in json.load(open(f"/tmp/plan{n}b.json"))}
-    for src, np in new.items():
-        op = old.get(src)
-        if not op:
-            continue
-        if op["target"] != np["target"] or op["kind"] != np["kind"]:
-            changed += 1
-            if len(examples) < 40:
-                examples.append((n, op, np))
-        else:
-            same += 1
-print(f"same: {same}  changed: {changed}")
-for n, op, np in examples:
-    print(f"\n[{n}] {op['name']}  (score {op['score']} -> {np['score']})")
-    print(f"  old: {op['target']}")
-    print(f"  new: {np['target']}")

tools/plan_summary.py

@@ -1,22 +0,0 @@
-#!/usr/bin/env python3
-"""Summarize a sorter plan JSON: match kinds, top targets, oddities."""
-import json
-import sys
-from collections import Counter
-from pathlib import Path
-
-plan = json.loads(Path(sys.argv[1]).read_text())
-print(f"files: {len(plan)}")
-print("kinds:", Counter(p["kind"] for p in plan))
-print("actions:", Counter(p.get("action", "?") for p in plan))
-# system breakdown
-print("systems:", Counter(p["system"] for p in plan).most_common(10))
-# top-level targets
-roots = Counter(str(Path(p["target"]).relative_to(
-    Path(p["target"]).parents[3])) if len(Path(p["target"]).parents) > 3
-    else p["target"] for p in plan)
-for root, n in roots.most_common(15):
-    print(f"  {n:5d}  {root}")
-# anything with empty target?
-empties = [p for p in plan if not p.get("target")]
-print("empty targets:", len(empties))