refactor: flatten the crate to the repo root and drop the python version

The Rust rewrite has replaced the original python implementation, so the transition layout is no longer needed:

- move version-rust/{Cargo.toml,Cargo.lock,README.md,.gitignore,src,data} to the repo root; the crate is now the repo.
- move docs/SORTING.md and data/gwshop_catalog.json to the root.
- remove version-python/ entirely, plus the dev-only version-rust/tools/dump_warhammer_data.py (its only job was regenerating data/warhammer_data.json from python).
- rewrite AGENTS.md and README.md for the single-project layout; strip all version-python/version-rust references from code comments.
- fix the gwshop credential lookup: algolia.json now sits at CARGO_MANIFEST_DIR (crate root == repo root), not its parent.
- restore the credentials/run-output ignores in .gitignore (algolia.json, .env, *.token, downloads/, state.json, manifest.jsonl, ...).
- docs/SORTING.md: retarget the extraction/sorter/dlp examples at the dlphelper binary instead of the deleted python scripts.

cargo test: 86/86 green.

.gitignore

@@ -1,12 +1,17 @@
-# Python
+/target
-__pycache__/
+
-*.pyc
+# Algolia credentials (search-only key, still never commit)
-.venv/
+algolia.json
-*.egg-info/
+gwshop_raw.json
+
+# Never commit credentials
+*.token
+.env
 
 # Run outputs (if OUT_DIR points into the repo)
 thread/
 downloads/
+extracted/
 export.json
 export.html
 state.json
@@ -14,11 +19,3 @@ state.json.tmp
 failed_urls.txt
 manifest.jsonl
 *.log
-
-# Never commit credentials
-*.token
-.env
-
-# Algolia credentials (search-only key, still never commit)
-algolia.json
-gwshop_raw.json

AGENTS.md

@@ -1,26 +1,252 @@
-# AGENTS.md — Warhammer_DLP_Helper (transition layout)
+# AGENTS.md — Warhammer_DLP_Helper
 
-This repo is mid-transition. **The `main` branch still carries the original
+Guidance for any AI agent (or human) working in this repository. Read this
-flat python layout** — read `version-python/AGENTS.md` there (and on this
+fully before making changes.
-branch at `version-python/AGENTS.md`) for the full project rules: secrets,
-failure policy, idempotency, rate limiting, testing. Those rules still apply.
 
-## Layout on this branch (`version-rust`)
+## 1. What this project is
 
-- `version-python/` — the existing, working python tooling, moved here with
+The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
-  `git mv` (history preserved). All rules and path references in its
+official Games Workshop assets) that are shared in Discord channels. Links in
-  `AGENTS.md` resolve relative to `version-python/`.
+those channels point to two hosters and are often **base64-encoded** inside
-- `version-rust/` — the Rust rewrite (cargo project) that will eventually
+message content (obfuscated by the posting bot).
-  replace the python version. See `version-rust/README.md` for the port plan
-  and crate choices.
 
-## Transition rules
+The end goal is a **torrent-first platform** (think radarr/lidarr, but for
+proxy files). This repo is the Rust implementation of the current stage: one
+binary, `dlphelper`, that
 
-1. Python fixes go into `version-python/` on `main` (flat layout there) and
+1. exports a Discord channel/thread (DiscordChatExporter via docker),
-   are merged into `version-python/` here; never fork the logic in two
+2. extracts base64-encoded and plain download links,
-   places.
+3. picks the right downloader per host automatically,
-2. The Rust version must stay compatible with the run artifacts
+4. downloads everything with per-file tracking, retries and resume,
-   (`state.json`, `manifest.jsonl`, `~/.local/share/dlphelper/urls.json`) —
+5. extracts the archives into a plain-file tree,
-   schemas are the contract.
+6. sorts the files into the library by system/group/faction/class/unit,
-3. `data/gwshop_catalog.json` (committed snapshot) is shared; the sorter in
+7. records a manifest that will later feed the torrent stage.
-   both versions must be able to read the same catalog schema.
+
+A webapp is the next stage. Deliberately **not** in scope (yet): `.torrent`
+creation, tracker upload, deseeeding, rename-into-library conventions. Design
+decisions should not close the door on these — especially `manifest.jsonl`
+and the `--hash` option exist for dedupe/cross-seeding later.
+
+This repository was ported from a Python implementation, which the Rust
+version has now replaced (the ported modules keep `Rust port of …` provenance
+notes in their headers).
+
+## 2. Repository & environment
+
+- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
+  **private** repo, branch `main`. Push with the credentials already stored
+  via git's credential store (`~/.git-credentials`); never paste them into
+  commands or files.
+- Machine: Linux, Rust (edition 2024), docker is available. The repo lives on
+  **NFS**, where rustc SIGBUSes — always build/test with an explicit target
+  directory:
+
+      CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
+
+- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
+  upstream reading material only.
+
+## 3. Architecture
+
+```
+Cargo.toml                  crate `dlphelper` (edition 2024)
+src/main.rs                 CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
+src/orchestrate.rs          dlp pipeline: export -> extract URLs -> per-host download
+src/discord.rs              DCE docker export (JSON) + base64/plain URL extraction
+src/gofile.rs               GoFile API client (one guest account per run)
+src/zero807.rs              curl downloader for 0807.st (direct links + /f/<id> folder pages)
+src/state.rs                state.json store
+src/globalstore.rs          cross-run URL store (~/.local/share/dlphelper/urls.json)
+src/manifest.rs             manifest.jsonl appender
+src/extract.rs              recursive archive extraction into a plain-file tree
+src/catalogue.rs            scanner + keyword classifier + report writer
+src/sorter.rs               library sorter: inspect/classify/fuzzy-match, plan + apply
+src/difflib.rs              faithful CPython SequenceMatcher port (bit-identical ratios)
+src/warhammer_data.rs       taxonomy tables (parsed from data/warhammer_data.json)
+src/gwshop.rs               GW webshop catalog dump via Algolia
+src/util.rs                 small shared helpers
+data/gwshop_catalog.json    committed catalog snapshot (sorter default)
+data/warhammer_data.json    committed taxonomy tables
+docs/SORTING.md             the library/sorting contract (change it before the code)
+```
+
+Data flow:
+
+```mermaid
+graph LR
+    A[channel id + token] --> B[DCE docker export -> export.json]
+    B --> C[decode base64 in message content]
+    C --> D{classify host}
+    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
+    D -->|0807.st| F[curl with UA + Referer]
+    D -->|other| G[recorded as failed: unsupported host]
+    E --> H[downloads/gofile/...]
+    F --> I[downloads/0807/...]
+    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
+    J --> K[globalstore: urls.json, marks URL done forever]
+    J --> L[extract -> plain-file tree]
+    L --> M[sorter -> library, hardlinks + audit]
+```
+
+Key facts about the hosters (learned the hard way, do not re-learn):
+
+- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
+  Creation is aggressively rate-limited — create **at most one account per
+  run**. Content listing needs `Authorization: Bearer <token>`, an
+  `X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
+  see `src/gofile.rs`), and downloads need `Cookie: accountToken=<token>` plus
+  Referer/Origin headers.
+- **0807.st**: serves files directly but 403s anything that doesn't look like
+  a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
+  default multi-connection request does not. curl is the tool here.
+  Folder pages (`https://0807.st/f/<id>`) are expanded: the page is parsed and
+  each listed file is fetched from `https://0807.st/<id>?dl=1` into
+  `downloads/0807/<page-id>/`. The folder `.zip` export is refused by the site
+  ("folder too large"), so per-file download is the only route in.
+- Encoded links in the channels start as base64 strings matching
+  `aHR0cHM6Ly[89]...`.
+
+## 4. Usage (what "done" looks like)
+
+    export DISCORD_TOKEN=...
+    dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash
+
+`--since 5d` (or `12h`, `2026-10-04`) limits a run to recent messages — handy
+for incremental updates. Re-runs are safe: `state.json` and the global store
+skip everything already marked ok.
+
+Retry only failures:
+
+    dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt
+
+Several channels at once (`subscriptions.json`: list of `{name, channel_id,
+out_dir}`):
+
+    dlphelper dlp --subs subscriptions.json
+
+Subs mode re-exports every channel by default (that is how new links get
+picked up) and relies on `state.json` + the global store to only download what
+is actually new. Use a cron/systemd timer for watch-like behavior.
+
+Extract and sort:
+
+    dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
+    dlphelper sorter <out>/extracted --library <library> --apply
+
+Manual import of old URL lists (known fully downloaded, never re-fetch):
+
+    dlphelper dlp --assume-done old_urls.txt
+
+## 5. Run artifacts (all in OUT_DIR, gitignored)
+
+| File | Purpose |
+|------|---------|
+| `export.json` | DCE export; cached — `--fresh` re-exports |
+| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
+| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
+| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
+| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
+| `extracted/extract_manifest.jsonl` | one line per extraction `{archive, dest, unit}` (feeds `sorter --provenance`) |
+| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. `--global-state` moves it, `--no-global` disables it for a run. |
+
+Sorter bookkeeping (audit + low-confidence doc) is written to
+`<library>/../_logs/` by default — never inside the library (see §7).
+
+## 6. Rules
+
+### Secrets and tokens
+1. Never print, log, commit or echo a Discord token or GoFile account token.
+   Not even partial. Strip them from any output you paste into chats.
+2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
+   It is handed to the DCE container via the `TOKEN` env var, never as a
+   command-line argument.
+3. `~/.git-credentials` may only be read by tooling that needs it; never copy
+   credentials into scripts, tests, or fixtures.
+4. Nothing downloaded and no export/state/manifest file is ever committed.
+
+### Failure policy (the core requirement)
+5. A failure in one URL or file must **never** abort the run. Record it in
+   `state.json` and continue. The per-URL loops in `src/orchestrate.rs`
+   therefore have catch-alls — keep them.
+6. Cooldowns are the only permitted "stall": after N consecutive failures,
+   pause, then continue. Circuit-breakers that exit the process need an
+   explicit justification in the commit message.
+7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
+   Password-required gofile links are not retried (it can't help); dead links
+   (HTTP 404/410) are never retried.
+
+### Idempotency
+8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json`
+   **or** the global store), skip existing non-empty files, write partial
+   downloads to `*.part` and rename on success.
+9. `state.json` is the source of truth for resume; keep its schema backward
+   compatible (unknown keys preserved, missing keys defaulted). Same for the
+   global store schema.
+10. `--assume-done` is a deliberate, manual import step for old URL lists.
+    Never invoke it automatically, and never let the tool mark URLs done
+    without a verified successful download.
+
+### Rate limiting / network etiquette
+11. One gofile guest account per run; `ensure_token()` backs off linearly
+    (20s × attempt). Do not raise the guest-account frequency "to make tests
+    pass". Testing account creation more than a couple times per hour is a
+    bug, not a test strategy.
+12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
+    re-verifying the 403 behavior.
+
+### Code style
+13. Rust 2024, idiomatic. Use `log` (not `println!`) in library modules;
+    prints are fine only for CLI output (plan/dry-run).
+14. Keep new dependencies justified — the crate already covers HTTP
+    (`reqwest`), archives (`sevenz-rust2`, `zip`, `unrar`), fuzzy matching
+    (the in-tree `difflib` port, **not** an external crate) and CLI (`clap`).
+    The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
+15. `src/difflib.rs` must stay bit-identical to CPython `difflib` — the sorter
+    thresholds depend on exact ratios; do not "simplify" it.
+
+### Validation
+16. Minimum validation for any change: `CARGO_TARGET_DIR=/var/tmp/dlphelper-target
+    cargo test` (the offline suite must stay green). Document what you could
+    *not* test (live gofile/0807/Discord paths need real links and tokens).
+17. Full live runs need a Discord token from the user — ask for it, never ask
+    for it to be pasted into a file or committed.
+
+### Git
+18. Branch `main`, descriptive commit messages, no force-push, no history
+    rewriting on the shared remote.
+
+## 7. Sorter & library structure
+
+`docs/SORTING.md` is the working agreement for the library hierarchy and the
+sorting pipeline — **change it first, then the code**. Summary:
+
+- Hierarchy: `<library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>`.
+  Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
+- Matching rules are fixed: the outer file name wins when it scores ≥
+  threshold; inner archive names only vote as fallback. A fuzzy match never
+  jumps into another game's section (keyword systems constrain the catalog
+  pool; cross-system adoption needs `CROSS_SYSTEM_MIN = 0.75` and token
+  containment). Deepest-matching-dir wins (junk dir names are never
+  candidates). Weak hits ≥ `WEAK_MIN = 0.50` are accepted but flagged
+  low-confidence. Tokenless normal adoption needs `+0.08`
+  (`TOKENLESS_BONUS`).
+- `DROPPED_SYSTEMS` (default `{Middle-Earth}`) filters catalog entries at load
+  and counts keyword hits as unclassified; `--drop-system`/`--keep-system`
+  adjust. The committed catalog snapshot keeps all systems.
+- Catalog source: the GW webshop via Algolia (`src/gwshop.rs`). The app-id +
+  public **search-only** key live in the gitignored `algolia.json` (or
+  `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY`); never commit any other Algolia key.
+  `data/gwshop_catalog.json` is the committed snapshot and the sorter default.
+- Bookkeeping (`sorter_applied.jsonl`, `sorter_low_confidence.txt`) is written
+  to `<library>/../_logs/` by default — never inside the library, which may be
+  a seeded torrent whose file set is frozen.
+
+## 8. Testing notes
+
+- Tests live inline (`#[cfg(test)]`) in each module and run with `cargo test`.
+  Keep them dependency-free and offline — they must never hit
+  gofile/0807/Discord.
+- `tempfile` tests stage under `/var/tmp` when available, because `/tmp` is a
+  small tmpfs that trips the extractor's free-space guard (`MIN_FREE_BYTES`).
+- To exercise downloads without Discord: build a `--urls-file` from known URLs
+  and run with a scratch OUT_DIR.

Cargo.toml

@@ -2,7 +2,7 @@
 name = "dlphelper"
 version = "0.1.0"
 edition = "2024"
-description = "Torrent-first platform for Warhammer proxy models: Discord channel export, link extraction, per-host downloaders, manifest — Rust rewrite of version-python."
+description = "Torrent-first platform for Warhammer proxy models: Discord channel export, link extraction, per-host downloaders, manifest — Rust rewrite of the original Python tool."
 
 [dependencies]
 anyhow = "1"

README.md

@@ -1,21 +1,19 @@
-# version-rust
+# Warhammer DLP Helper
 
-Rust rewrite of `version-python` (same repo, `main` branch still carries the
+Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts):
-flat python layout). Goal: one concise binary + webapp, eventually replacing
+export a Discord channel, extract the (often base64-encoded) download links,
-the python tooling.
+download per host, extract the archives, sort the files into the library, and
+record a manifest that will feed the torrent stage. One concise Rust binary +
+webapp (the webapp is the next stage).
 
-## Why this structure
+Ported from the original Python implementation, which this version replaced.
-
+The `Rust port of …` provenance notes in the module headers name the Python
-- `main` stays untouched (flat python layout) until the rewrite reaches parity.
+sources it was derived from; `AGENTS.md` holds the project rules and
-- This branch re-homes the python code into `version-python/` (via `git mv`,
+`docs/SORTING.md` the library/sorting contract.
-  history preserved) and starts `version-rust/` beside it.
-- `version-python/AGENTS.md` still documents the python stage — its rules
-  (secrets, failure policy, idempotency, rate limiting) apply to the rewrite
-  too and are the contract the Rust version must keep.
 
 ## Crate choices (as ported)
 
-| Concern | Python today | Rust |
+| Concern | Original Python | Rust |
 |---|---|---|
 | 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
 | zip extraction | 7z CLI | `zip` crate |
@@ -30,8 +28,7 @@ the python tooling.
 
 ## Port status
 
-All python entry points are ported; live network paths are tested offline
+All original Python entry points are ported and live-validated (see below).
-only (see "Untested live paths").
 
 | Rust module | Python source | Tests |
 |---|---|---|
@@ -75,13 +72,12 @@ Notes:
 
 - The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
   only ever passed to the DCE container via env, never argv.
-- `sorter --catalog` defaults to `version-python/data/gwshop_catalog.json`
+- `sorter --catalog` defaults to `data/gwshop_catalog.json`, relative to the
-  (falling back to `data/gwshop_catalog.json`), both relative to the current
+  current working directory — pass `--catalog` when running elsewhere.
-  working directory — pass `--catalog` when running elsewhere.
 - `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
   (bookkeeping stays out of the library, which may be a seeded torrent).
-- The sorter implements the rules in `version-python/docs/SORTING.md` — that
+- The sorter implements the rules in `docs/SORTING.md` — that document is the
-  document is still the working agreement; change it first, then both codes.
+  working agreement; change it first, then the code.
 - Retry loops, cooldowns and the "one gofile guest account per run" rule are
   ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
 - Ctrl-C: state.json is saved after every URL; re-run to resume (the python

docs/SORTING.md

@@ -7,8 +7,7 @@ sorting-related code; change it here first, then in code.
 ## Target hierarchy (Games Workshop webshop-aligned)
 
 The hierarchy follows the official GW webshop taxonomy (dumped from the
-shop's Algolia index by `dlphelper/gwshop.py` into
+shop's Algolia index by `dlphelper gwshop` into `data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
-`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
 hierarchy treated as sub-factions are full GW factions, and the class layer
 holds GW shop categories (unit types / product lines) instead of the retired
 40k force-organization slots:
@@ -36,12 +35,11 @@ Layer rules:
 
 - Files without a faction but with a game land in `<game>/_unsorted/`.
 - Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
-- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
+- The keyword tables in `data/warhammer_data.json` (GW naming) provide the
-  system/faction/class signal; a catalog match then overrides the hierarchy
+  initial system/faction/class signal; a catalog match then overrides the
-  from the catalog entry, which is authoritative.
+  hierarchy from the catalog entry, which is authoritative.
-- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
+- The catalog is pluggable via `dlphelper sorter --catalog`; every entry
-  still usable via `sorter.py --catalog`; its sub-faction entries map into
+  uses the same schema. For GW-sourced entries `subfaction` stays empty.
-  the same schema. For GW-sourced entries `subfaction` stays empty.
 
 ## Canonical names
 
@@ -63,15 +61,15 @@ graph LR
 ```
 
 1. **Inspect** — read the file name; for archives, also read the *inner*
-   file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
+   file list (zip via the `zip` crate, 7z via `sevenz-rust2` — no external
-   no Python dependency). Inner names improve classification for hash-named
+   binary). Inner names improve classification for hash-named archives and
-   archives and are recorded in the report.
+   are recorded in the report.
 2. **Classify** — game/faction/class from the keyword tables
-   (`warhammer_data.py`), same as `catalogue.py`.
+   (`data/warhammer_data.json`).
-3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
+3. **Match** — fuzzy-match the (inner) name against the catalog
-   entries within the same faction; best sequence-match above
+   (`data/gwshop_catalog.json`) entries within the same faction; best
-   `--threshold` (default 0.62) wins. The catalog provides canonical names,
+   sequence-match above `--threshold` (default 0.62) wins. The catalog
-   sub-factions and classes.
+   provides canonical names, sub-factions and classes.
 4. **Plan** — default output: a dry-run table `source → target` plus
    `sorter_plan.jsonl`. Nothing is touched.
 5. **Apply** — create the target and **hardlink** the file there
@@ -104,12 +102,12 @@ confidence doc always lands next to the audit file.
   unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
   until the link-based flow is proven on a small batch.
 
-## Extraction stage (`tools/extract_all.py`)
+## Extraction stage (`dlphelper extract`)
 
-That endgame stage now exists as a standalone tool (the sorter stays
+That endgame stage now exists as a standalone command (the sorter stays
 plan/apply-only):
 
-    python3 tools/extract_all.py <src_root>... --out <extracted_root>
+    dlphelper extract <src_root>... --out <extracted_root>
 
 - Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
   into a folder named after the archive, preserving relative structure;
@@ -127,7 +125,7 @@ plan/apply-only):
   `The Twin Lance/` it belongs to that unit) and placed together, keeping
   the archive's inner directory structure:
 
-      python3 sorter.py <extracted_root> --library <lib> \
+      dlphelper sorter <extracted_root> --library <lib> \
           --provenance <extracted_root>/extract_manifest.jsonl --apply
 
 Operational notes from the first full run (2026-10-06, sources
@@ -141,17 +139,16 @@ already-present duplicates, 0 copied):
   link, compare `st_nlink` (and content/size) — never inodes. Server-side
   links are real: a fresh `os.link` bumps nlink (2→3 in the checked case).
 - Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2
-  ("Cannot open output file"); `extract_all.py` then extracts to a
+  ("Cannot open output file"); the extractor then extracts to a `/var/tmp`
-  `/var/tmp` temp dir, repairs the names and moves the result over.
+  temp dir, repairs the names and moves the result over.
 - `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not
   extracted.
-- Files under hidden trees (`.broken/`) are never planned; retries of the
+- Files under hidden trees (`.broken/`) are never planned; failed archives
-  truncated/HTML dead-link downloads are listed in
+  (HTML-page fake zips, password archives, disk full) are quarantined there
-  `<extracted>/.broken/RETRY_URLS.txt` (all 0807.st; re-run
+  for inspection and the run continues.
-  `python3 dlp.py <channel> <out> --urls-file ...` for them).
 - **Non-warhammer systems are dropped** (2026-10-07, user decision:
-  "drop anything that isn't really warhammer"). The sorter default
+  ("drop anything that isn't really warhammer"). The sorter default
-  (`DROPPED_SYSTEMS` in `dlphelper/sorter.py`) is `{Middle-Earth}`:
+  (`DROPPED_SYSTEMS` in `src/sorter.rs`) is `{Middle-Earth}`:
   catalog entries of dropped systems are filtered at load and keyword
   hits for them count as unclassified, so nothing is ever placed into
   those sections - the files stay in `_unsorted`. `--drop-system NAME`
@@ -168,7 +165,7 @@ already-present duplicates, 0 copied):
   unit's class layer always wins (no keyword class survives next to a
   precise unit match).
 
-## Catalog file (`data/library_catalog.json`)
+## Catalog file (`data/gwshop_catalog.json`)
 
 ```json
 {

src/catalogue.rs

@@ -1,5 +1,5 @@
 //! Catalogue: classify collected files into game system / faction / class —
-//! Rust port of `version-python/dlphelper/catalogue.py`.
+//! Rust port of `dlphelper/catalogue.py`.
 //!
 //! The catalogue is the bridge between "a pile of downloads" and the later
 //! library stage: it reads manifest.jsonl (with full source attribution) and/or
@@ -745,7 +745,7 @@ pub fn build_catalogue(
 }
 
 // ---------------------------------------------------------------------------
-// Tests (offline port of version-python/tests/test_catalogue.py)
+// Tests (offline port of tests/test_catalogue.py)
 // ---------------------------------------------------------------------------
 
 #[cfg(test)]

src/discord.rs

@@ -1,5 +1,5 @@
 //! DCE docker export (JSON) + base64/plain URL extraction — Rust port of
-//! `version-python/dlphelper/discord.py`.
+//! `dlphelper/discord.py`.
 //!
 //! The channels this tool targets encode their download links as base64 inside
 //! message content, and occasionally as plain links. We export the channel as
@@ -324,7 +324,7 @@ mod tests {
         STANDARD.encode(s.as_bytes())
     }
 
-    /// The two-message fixture from version-python/tests/test_extract.py.
+    /// The two-message fixture from tests/test_extract.py.
     fn fixture_messages() -> Value {
         json!([
             {

src/extract.rs

@@ -1,5 +1,5 @@
 //! Recursive archive extraction into a plain-file tree — Rust port of
-//! `version-python/tools/extract_all.py` (extract_manifest.jsonl feeds the
+//! `tools/extract_all.py` (extract_manifest.jsonl feeds the
 //! sorter's provenance mode; failures go to .broken/; the run never stops).
 //!
 //! Extraction backends: `sevenz-rust2` for .7z, `zip` for .zip, `unrar` for

src/globalstore.rs

@@ -1,5 +1,5 @@
 //! Cross-run, cross-channel record of completed URLs — Rust port of
-//! `version-python/dlphelper/globalstore.py`.
+//! `dlphelper/globalstore.py`.
 //!
 //! state.json only knows about one OUT_DIR; this store (default
 //! `~/.local/share/dlphelper/urls.json`) records every successfully

src/gofile.rs

@@ -1,5 +1,5 @@
 //! GoFile API client (one guest account per run) — Rust port of
-//! `version-python/dlphelper/gofile.py`.
+//! `dlphelper/gofile.py`.
 //!
 //! The API interaction mirrors Lysagxra/GoFileDownloader (same headers,
 //! website-token scheme and endpoints), with the same two deliberate

src/gwshop.rs

@@ -1,5 +1,5 @@
 //! GW webshop catalog dump via Algolia — Rust port of
-//! `version-python/dlphelper/gwshop.py`.
+//! `dlphelper/gwshop.py`.
 //!
 //! The warhammer.com shop is behind an AWS WAF challenge for non-browser
 //! clients, but its product search runs on Algolia and the app-id plus the
@@ -42,15 +42,11 @@ pub fn load_credentials() -> anyhow::Result<(String, String)> {
     if !app_id.is_empty() && !api_key.is_empty() {
         return Ok((app_id, api_key));
     }
-    // python: algolia.json next to the repo root; the Rust crate root is
+    // algolia.json lives at the repo root, which is also the crate root, so
-    // version-rust/, so the repo root is its parent. CARGO_MANIFEST_DIR is
+    // CARGO_MANIFEST_DIR (baked at build time) points straight at it. Fall
-    // baked at build time - use the executable's location upward search as
+    // back to the current working directory for runs from elsewhere.
-    // fallback? Keep it simple and deterministic: repo root via
-    // CARGO_MANIFEST_DIR/../algolia.json, checked at runtime.
     let candidates = [
-        PathBuf::from(env!("CARGO_MANIFEST_DIR"))
+        Some(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("algolia.json")),
-            .parent()
-            .map(|p| p.join("algolia.json")),
         Some(PathBuf::from("algolia.json")),
     ];
     for candidate in candidates.into_iter().flatten() {
@@ -623,8 +619,6 @@ mod tests {
                 .map(|v| v.is_empty())
                 .unwrap_or(true)
             && !PathBuf::from(env!("CARGO_MANIFEST_DIR"))
-                .parent()
-                .unwrap()
                 .join("algolia.json")
                 .exists()
         {

src/main.rs

@@ -1,10 +1,10 @@
 //! dlphelper CLI — Rust rewrite of the python entry points:
 //!
-//! - `dlphelper dlp ...`          <- version-python/dlp.py
+//! - `dlphelper dlp ...`          <- dlp.py
-//! - `dlphelper catalogue ...`    <- version-python/catalogue.py
+//! - `dlphelper catalogue ...`    <- catalogue.py
-//! - `dlphelper sorter ...`       <- version-python/sorter.py
+//! - `dlphelper sorter ...`       <- sorter.py
-//! - `dlphelper extract ...`      <- version-python/tools/extract_all.py
+//! - `dlphelper extract ...`      <- tools/extract_all.py
-//! - `dlphelper gwshop ...`       <- version-python/dlphelper/gwshop.py
+//! - `dlphelper gwshop ...`       <- dlphelper/gwshop.py
 //!
 //! Secrets rule: the Discord token is resolved via --token / $DISCORD_TOKEN /
 //! hidden prompt and only ever handed to the DCE container via env.
@@ -160,8 +160,7 @@ struct SorterArgs {
     /// library root the files will be linked into
     #[arg(long, value_name = "DIR")]
     library: PathBuf,
-    /// catalog FILE (default: version-python/data/gwshop_catalog.json,
+    /// catalog FILE (default: data/gwshop_catalog.json, relative to the cwd)
-    /// falling back to data/gwshop_catalog.json, both relative to the cwd)
     #[arg(long, value_name = "FILE")]
     catalog: Option<PathBuf>,
     /// fuzzy match threshold (default 0.62)
@@ -254,15 +253,7 @@ fn make_gstore(args: &DlpArgs) -> globalstore::GlobalStore {
 }
 
 fn default_catalog_path() -> PathBuf {
-    let v = PathBuf::from("version-python/data/gwshop_catalog.json");
+    PathBuf::from("data/gwshop_catalog.json")
-    if v.exists() {
-        return v;
-    }
-    let alt = PathBuf::from("data/gwshop_catalog.json");
-    if alt.exists() {
-        return alt;
-    }
-    v // surfaces a clear error in load_catalog
 }
 
 fn cmd_dlp(args: &DlpArgs) -> anyhow::Result<()> {

src/manifest.rs

@@ -1,5 +1,5 @@
 //! manifest.jsonl — one compact JSON line per downloaded file — Rust port of
-//! `append_manifest` from `version-python/dlphelper/state.py`.
+//! `append_manifest` from `dlphelper/state.py`.
 //!
 //! Row layout (all attribution fields optional except channel/url):
 //! `{"channel", "url", "host", "message_id", "timestamp", "author", "path",

src/orchestrate.rs

@@ -1,5 +1,5 @@
 //! Orchestration: per-channel pipeline, gofile/0807 loops, subscriptions,
-//! --assume-done — Rust port of the logic in `version-python/dlp.py`.
+//! --assume-done — Rust port of the logic in `dlp.py`.
 //!
 //! Failure policy (the core requirement): a failure in one URL must never
 //! abort the run. Catch, record in state.json, continue. Cooldown pauses

src/sorter.rs

@@ -1,8 +1,8 @@
 //! Library sorter: inspect/classify/fuzzy-match against the catalog, plan +
 //! hardlink apply, optional provenance unit mode — Rust port of
-//! `version-python/dlphelper/sorter.py`.
+//! `dlphelper/sorter.py`.
 //!
-//! Matching rules are fixed (see version-python/AGENTS.md §7 "Sorter matching
+//! Matching rules are fixed (see AGENTS.md §7 "Sorter matching
 //! rules"); they are ported exactly, including thresholds (CROSS_SYSTEM_MIN
 //! 0.75, WEAK_MIN 0.50, TOKENLESS_BONUS 0.08), the generic-token stoplist,
 //! junk-dir stoplist, deepest-dir re-homing and the weak tier. Fuzzy scores
@@ -1732,7 +1732,7 @@ pub fn format_plan(items: &[SortItem]) -> String {
 mod tests {
     use super::*;
 
-    /// Same fixture as version-python/tests/test_sorter_units.py.
+    /// Same fixture as tests/test_sorter_units.py.
     fn test_catalog_json() -> String {
         serde_json::json!({
             "entries": [

src/state.rs

@@ -1,5 +1,5 @@
 //! Per-run persistent state (state.json) — Rust port of
-//! `version-python/dlphelper/state.py`. Makes runs resumable and failures
+//! `dlphelper/state.py`. Makes runs resumable and failures
 //! trackable: re-runs skip URLs with status "ok", failed URLs are retried.
 //!
 //! Schema contract with the python version:

src/warhammer_data.rs

@@ -1,10 +1,10 @@
 //! Taxonomy: GW-named systems/factions, GW shop categories as classes,
-//! GROUP_OF_FACTION map — Rust port of `version-python/dlphelper/warhammer_data.py`.
+//! GROUP_OF_FACTION map — Rust port of `dlphelper/warhammer_data.py`.
 //!
-//! The tables are NOT hand-transcribed: they are dumped from the python
+//! The tables are NOT hand-transcribed: they were dumped from the original
-//! module into `data/warhammer_data.json` (regenerate with
+//! python module into `data/warhammer_data.json` (the committed source of
-//! `version-rust/tools/dump_warhammer_data.py`) and parsed here, so both
+//! truth) and are parsed here, so the data stayed byte-identical across the
-//! versions work from byte-identical data.
+//! port.
 //!
 //! Python-dict ordering is preserved exactly (insertion order; a duplicate
 //! key keeps its first insertion position while the *last* value wins),

src/zero807.rs

@@ -1,5 +1,5 @@
 //! curl-based downloader for 0807.st direct links — Rust port of
-//! `version-python/dlphelper/zero807.py`.
+//! `dlphelper/zero807.py`.
 //!
 //! 0807.st serves the files directly but rejects requests that don't look
 //! like a browser (requires a User-Agent and the Referer of the site page),

version-python/AGENTS.md

@@ -1,310 +0,0 @@
-# AGENTS.md — Warhammer_DLP_Helper
-
-Guidance for any AI agent (or human) working in this repository. Read this
-fully before making changes.
-
-## 1. What this project is
-
-The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
-official Games Workshop assets) that are shared in Discord channels. Links in
-those channels point to two hosters and are often **base64-encoded** inside
-message content (obfuscated by the posting bot).
-
-The end goal is a **torrent-first platform** (think radarr/lidarr, but for
-proxy files). This repo is the current stage of that journey: a tool that
-
-1. exports a Discord channel/thread (DiscordChatExporter via docker),
-2. extracts base64-encoded and plain download links,
-3. picks the right downloader per host automatically,
-4. downloads everything with per-file tracking, retries and resume,
-5. records a manifest that will later feed the torrent stage.
-
-Deliberately **not** in scope (yet): .torrent creation, tracker upload,
-deseeding, rename-into-library conventions. Design decisions should not close
-the door on these — especially `manifest.jsonl` and the `--hash` option exist
-for dedupe/cross-seeding later.
-
-## 2. Repository & environment
-
-- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
-  **private** repo, branch `main`. Push with the credentials already stored
-  via git's credential store (`~/.git-credentials`); never paste them into
-  commands or files.
-- The server is Gitea 1.27. To create *another* repo, use
-  `python3 tools/new_repo.py list|create <name>` (reads `~/.git-credentials`
-  itself, never prints secrets). Never delete or overwrite existing repos.
-- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only
-  runtime dependency (see `requirements.txt`), docker is available.
-- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
-  upstream reading material only.
-
-## 3. Architecture
-
-```
-dlp.py                     CLI orchestrator (entry point)
-catalogue.py               CLI: classify the collection into system/faction/class
-sorter.py                  CLI: fuzzysort files into the library (plan/apply,
-                           hardlinks, default catalog data/gwshop_catalog.json)
-dlphelper/
-  discord.py               DCE docker export (JSON) + base64/plain URL extraction
-  gofile.py                GoFile API client (one guest account per run)
-  zero807.py               curl-based downloader for 0807.st direct links
-  state.py                 state.json store + manifest.jsonl appender
-  globalstore.py           cross-run URL store (~/.local/share/dlphelper/urls.json)
-  catalogue.py             scanner + keyword classifier + report writer
-  warhammer_data.py        taxonomy: GW-named systems/factions, GW shop
-                           categories as classes, GROUP_OF_FACTION map
-  gwshop.py                GW webshop catalog dump via Algolia (see §7.3)
-  sorter.py                library sorter: inspect/classify/fuzzy-match
-                           against the catalog, plan + hardlink apply,
-                           optional --provenance unit mode for extracted trees
-parts/grab.sh              legacy groundstone script (kept, not used by dlp.py)
-tools/new_repo.py          Gitea repo helper (reads ~/.git-credentials itself)
-tools/debug_gofile.py      inspect raw gofile API response for one content id
-tools/extract_all.py       recursive archive extraction into a plain-file tree
-                           (extract_manifest.jsonl feeds the sorter's
-                           --provenance unit mode; failures go to .broken/;
-                           7z-direct-to-CIFS failures fall back to a
-                           /var/tmp extraction with cp437 name repair;
-                           AppleDouble/__MACOSX junk is dropped)
-tests/test_extract.py      offline extraction test (plain python3, no deps)
-tests/test_catalogue.py    offline classifier test
-```
-
-Data flow:
-
-```mermaid
-graph LR
-    A[channel id + token] --> B[DCE docker export -> export.json]
-    B --> C[decode base64 in message content]
-    C --> D{classify host}
-    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
-    D -->|0807.st| F[curl with UA + Referer]
-    D -->|other| G[recorded as failed: unsupported host]
-    E --> H[downloads/gofile/...]
-    F --> I[downloads/0807/...]
-    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
-    J --> K[globalstore: urls.json, marks URL done forever]
-```
-
-Key facts about the hosters (learned the hard way, do not re-learn):
-
-- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
-  Creation is aggressively rate-limited — create **at most one account per
-  run**. Content listing needs `Authorization: Bearer <token>`, an
-  `X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
-  see `gofile.py`), and downloads need `Cookie: accountToken=<token>` plus
-  Referer/Origin headers.
-- **0807.st**: serves files directly but 403s anything that doesn't look like
-  a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
-  default multi-connection request does not. curl is the tool here.
-- Encoded links in the channels start as base64 strings matching
-  `aHR0cHM6Ly[89]...` — keep the extraction regex compatible with
-  `parts/grab.sh`.
-
-## 4. Usage (what "done" looks like)
-
-    export DISCORD_TOKEN=...
-    python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>
-
-Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt`
-(or just re-run — state.json skips everything already marked ok).
-
-Several channels at once (`subscriptions.json`: list of {name, channel_id,
-out_dir}):
-
-    python3 dlp.py --subs subscriptions.json
-
-Subs mode re-exports every channel by default (that is how new links get
-picked up) and relies on state.json + the global store to only download
-what is actually new. Use a cron/systemd timer for watch-like behavior.
-
-Manual import of the pre-dlp.py era (URLs you know are fully downloaded,
-never re-fetch them again):
-
-    python3 dlp.py --assume-done old_urls.txt
-
-## 5. Run artifacts (all in OUT_DIR, gitignored)
-
-| File | Purpose |
-|------|---------|
-| `export.json` | DCE export; cached — `--fresh` re-exports |
-| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
-| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
-| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
-| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
-| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. |
-
-## 6. Rules
-
-### Secrets and tokens
-1. Never print, log, commit or echo a Discord token or GoFile account token.
-   Not even partial. Strip them from any output you paste into chats.
-2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
-   It is handed to the DCE container via the `TOKEN` env var, never as a
-   command-line argument.
-3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy
-   credentials into scripts, tests, or fixtures.
-4. Nothing downloaded and no export/state/manifest file is ever committed.
-
-### Failure policy (the core requirement)
-5. A failure in one URL or file must **never** abort the run. Catch, record in
-   `state.json`, continue. The loops in `dlp.py` therefore have catch-alls —
-   keep them.
-6. Cooldowns are the only permitted "stall": after N consecutive failures,
-   pause, then continue. Circuit-breakers that exit the process need a
-   explicit justification in the PR/commit message.
-7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
-   Password-required gofile links are not retried (it can't help).
-
-### Idempotency
-8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or**
-   the global store), skip existing non-empty files, write partial downloads
-   to `*.part` and rename on success.
-9. `state.json` is the source of truth for resume; keep its schema backward
-   compatible when touching `state.py` (unknown keys are preserved, missing
-   keys defaulted). Same for the global store schema in `globalstore.py`.
-10. `--assume-done` is a deliberate, manual import step for old URL lists.
-    Never invoke it automatically, and never let the tool mark URLs done
-    without a verified successful download.
-
-### Rate limiting / network etiquette
-11. One gofile guest account per run; `ensure_token()` backs off linearly
-    (20s × attempt). Do not raise the guest-account frequency "to make tests
-    pass". Testing account creation more than a couple times per hour is a
-    bug, not a test strategy.
-12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
-    re-verifying the 403 behavior.
-
-### Code style
-13. Python 3.12+, type hints on public functions, `logging` not `print` in
-    library code (prints are fine only for `--dry-run` output).
-14. Stay on stdlib + `requests`. New dependencies need a real justification
-    and belong in `requirements.txt`.
-15. `dlphelper/` modules must stay import-safe (no side effects at import).
-
-### Validation
-15. Minimum validation for any change: `python3 -m py_compile dlp.py
-    catalogue.py dlphelper/*.py`, plus the offline tests (`python3
-    tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
-    modules changed. Document what you could *not* test (e.g. live
-    gofile/0807 downloads need real links and tokens).
-17. Full live runs need a Discord token from the user — ask for it, never
-    ask for it to be pasted into a file or committed.
-
-### Git
-18. Branch `main`, descriptive commit messages, no force-push, no history
-    rewriting on the shared remote.
-19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
-    fine, deletion/renaming of existing repos requires the user.
-
-## 7. Roadmap (future stages, keep compatible)
-
-1. **Dedupe**: hash-based (sha256) duplicate detection across threads via
-   `manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already
-   exists via the global store; content-level (same file, different link)
-   comes with hashing.
-2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
-   a watch dir, cross-seeding between direct-download and torrent copies.
-3. **Library organization (started)**: `catalogue.py` classifies every file
-   into system / faction / class and can move files into
-   `<library>/<System>/<Group>/<Faction>/<Class>/` with `--apply` (audited in
-   `catalogue_moves.jsonl`). The taxonomy lives in
-   `dlphelper/warhammer_data.py`; it is keyword-based and grows with the
-   collection - unclassified files are reported, never guessed. Manual fixes
-   go into an overrides JSON (`--overrides`). The library hierarchy follows
-   the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
-   working agreement for the sorting pipeline.
-
-   **Sorter matching rules (fixed 2026-10-05 after the first full run):**
-   the outer file name wins when it scores >= threshold; inner archive
-   names only vote as fallback (hash-named gofile archives need that). A
-   fuzzy match never jumps into another game's section: keyword-identified
-   systems constrain the catalog pool, and a file with neither system nor
-   faction keyword only adopts a catalog unit on a strong hit
-   (`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`.
-   Token-containment boosts exist for short names vs long catalog names
-   ("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token
-   stoplist.
-
-   **Refined 2026-10-07 (dir re-homing + weak tier + non-warhammer drop):**
-   - **Deepest-matching-dir wins**: ancestor directory names (inside the
-     archive in provenance mode, real dirs in manual trees) compete with
-     the pack/file name; the best accepted match re-homes the whole
-     subtree. Junk dir names (files/images/HEADS/pure numbers/lone generic
-     words) are never candidates. Fixes multi-unit sculptor packs (one
-     "Retro Elvez" archive ≈ 20 Eldar units no longer all land in one
-     keyword class).
-   - **Weak tier**: when nothing reaches the threshold, a hit >= `WEAK_MIN`
-     (0.50) is accepted but flagged low confidence - `[w]` in the plan,
-     `confidence: low` in the audit, listed in
-     `sorter_low_confidence.txt` in the log dir (see below). Weak and tokenless adoptions
-     require a shared non-generic token; tokenless normal adoption needs
-     +0.08 (`TOKENLESS_BONUS`); cross-system adoption additionally needs
-     token containment (`_token_contained`), because char-ratio lies
-     ("aeldari rangers" vs "dwarf rangers" = 0.786).
-   - **Non-warhammer systems dropped**: `DROPPED_SYSTEMS` (default
-     `{Middle-Earth}`) - catalog entries filtered at load, keyword hits
-     count as unclassified; files stay in `_unsorted`. `--drop-system`/
-     `--keep-system` adjust. The catalog snapshot keeps all systems.
-   - **Probe votes on full inner paths** (2026-10-07, after run 5): pack
-     probes clean the whole relative path, so inner dir names vote
-     ("Catachan/Jungle Fighters/" resolves Astra Militarum). No
-     single-token containment boost for inner votes (a root file named
-     "lieutenant" once claimed ~7000 files into one Iron Hands unit);
-     full-catalog and different-faction adoptions need token
-     containment (`_contained_somewhere`).
-
-   - **Bookkeeping location (fixed 2026-10-08)**: the audit and the
-     low-confidence doc are written to `<library>/../_logs/` by default
-     (`--audit` overrides) - never inside the library itself, because the
-     library may be a seeded torrent whose file set is frozen.
-
-   The catalog deliberately covers all GW systems (needed for
-   catalog-agnostic use) - wrong-system placement is prevented by matching
-   discipline and the drop filter, not by shrinking the catalog.
-
-   **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
-   working, verified 2026-10-04 — how it was obtained, do not re-derive):**
-   scraping warhammer.com directly is blocked — the shop HTML and all
-   search/listing endpoints (`/en-WW/shop/*`,
-   `/app/resources/catalog/search/*`, `/app/resources/catalog/categories`)
-   sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any
-   non-browser client, regardless of User-Agent; the sitemap declared in
-   robots.txt does not exist; deep paths under the allowed
-   `/app/resources/catalog/product/*` reach a separate commerce backend
-   (plain `404 Not found`) but the resource shape is unknown. What works:
-   the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and
-   its app-id plus the **public search-only API key** ship with the browser
-   page; the user captured both from the browser devtools ("Copy as cURL"
-   on one search request) and they are stored in the gitignored
-   `algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the
-   key is search-only, never commit any other Algolia key. The full
-   product index `prod-lazarus-product-en-ww` (~3500 products, plus one
-   sibling index per locale) is dumped with plain `requests`: the
-   public search-only key cannot use Algolia's `/browse` (403) and normal
-   pagination is capped at 1000 cumulative hits, so `gwshop.py`
-   **partitions the index on the `GameSystemsRoot` hierarchical facet** —
-   each branch is fetched with an explicit filter, recursing deeper only
-   when a branch exceeds one page. (Live count 2026-10-04: 4183 records,
-   of which 3491 are reachable via the facet; the ~692 left out are paints,
-   Black Library books, White Dwarf issues, brushes, bases and gifts that
-   carry no `GameSystemsRoot` path — no miniature kits among them.) Records are normalized into the
-   source-agnostic catalog schema (system/group/faction/subfaction/class
-   = the facet path, name/sku/productType attached) into
-   `data/gwshop_catalog.json`, which is the sorter's default catalog and
-   is **committed as a snapshot**; refresh with
-   `python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog
-   sources stay pluggable and source-agnostic (miniset.net — reachable
-   but stale, manual JSON overrides, `data/library_catalog.json`).
-4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
-   primitive version; a real watch mode (per-subscription schedules, tear-off
-   notifications, discord pings on new links) builds on it.
-
-## 8. Testing notes
-
-- `tests/` currently holds a synthetic `export.json` test for extraction
-  (`tests/test_extract.py`, runnable with plain `python3`). Keep tests
-  dependency-free and offline — they must never hit gofile/0807/Discord.
-- To exercise downloads without Discord: build a `--urls-file` from known
-  URLs and run with a scratch OUT_DIR.

version-python/README.md

@@ -1,111 +0,0 @@
-# Warhammer DLP Helper
-
-A small "radarr for Warhammer proxy files" tool: point it at a Discord
-channel/thread and an output directory, and it exports the channel, extracts
-the encoded download links, and downloads everything - automatically choosing
-the right downloader per host:
-
-| Host        | Downloader | Notes |
-|-------------|------------|-------|
-| gofile.io   | built-in GoFile API client | one guest account per run (rate-limit friendly) |
-| 0807.st     | curl | needs browser-like headers (`-A`, `-e https://0807.st/`) |
-
-## Install
-
-    pip install -r requirements.txt   # just `requests`
-    docker pull tyrrrz/discordchatexporter:stable
-
-## Usage
-
-    export DISCORD_TOKEN="..."
-    python3 dlp.py <channel_or_thread_id> /path/to/output
-
-The token can also be passed with `--token` (or you'll be prompted, input
-hidden). Useful options:
-
-    --urls-file failed_urls.txt   retry a previous failure list, no re-export
-    --password <pw>               for password-protected gofile links
-    --only gofile|0807            process only one host
-    --extract                     unzip archives into per-archive folders
-    --hash                        record sha256 per file in the manifest
-    --dry-run                     just show what would be downloaded
-    --fresh                       force a fresh channel export
-
-## Several channels / watching for updates
-
-Put your channels into `subscriptions.json`:
-
-```json
-{
-  "subscriptions": [
-    {"name": "some-eldar",
-     "channel_id": "1455993943471820842",
-     "out_dir": "/big/storage/warhammer/some-eldar"}
-  ]
-}
-```
-
-then run (e.g. from a daily cron/systemd timer):
-
-    python3 dlp.py --subs subscriptions.json
-
-Subs mode re-exports each channel to pick up new links; `state.json` and the
-global store make sure only genuinely new URLs are downloaded.
-
-## Cataloguing the collection
-
-`catalogue.py` classifies every downloaded file into game system, faction and
-force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
-Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
-taxonomy in `dlphelper/warhammer_data.py`:
-
-    # preview a dlp.py run dir (manifest + downloads)
-    python3 catalogue.py /path/to/run_dir
-
-    # include the manual-era directories
-    python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
-
-    # catalogue arbitrary dirs only
-    python3 catalogue.py --scan /big/storage/manual
-
-    # actually reorganize into a library tree (audited, only classified files)
-    python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
-
-Outputs `catalogue.md` (readable) and `catalogue.jsonl` (machine, with source
-attribution per file). Unclassified files are listed, never guessed; fix them
-with an overrides JSON (`--overrides overrides.json`).
-
-## Already downloaded the old way?
-
-`--assume-done` imports a plain URL list into the global store so those links
-are never fetched again (no verification — use deliberately):
-
-    python3 dlp.py --assume-done old_urls.txt
-
-## Output layout
-
-    OUT_DIR/
-      export.json           Discord channel export (cached between runs)
-      state.json            per-URL status; re-runs skip everything marked "ok"
-      failed_urls.txt       URLs that failed after all retries
-      manifest.jsonl        one line per downloaded file (url, source message, path, size)
-      downloads/
-        gofile/<content>/<folders>/<files>
-        0807/<id>.zip
-| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
-| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
-
-Runs are resumable and idempotent: interrupted downloads use `.part` files and
-finished ones are skipped on the next run. Individual failures never stop the
-run; after 3 consecutive failures it pauses for a cooldown (rate-limit
-protection) and continues.
-
-## Why not use Lysagxra/GoFileDownloader directly?
-
-That tool creates a new GoFile guest account **per URL** and calls
-`sys.exit(1)` on the first failure, which is exactly why runs died after ~6
-downloads. This project reimplements its (working) API scheme - headers,
-`X-Website-Token`, download links with the `accountToken` cookie - but creates
-one guest account per run, records failures per file, and keeps going.
-
-`parts/grab.sh` is the original groundstone script, kept for reference.

version-python/catalogue.py

@@ -1,118 +0,0 @@
-#!/usr/bin/env python3
-"""Catalogue CLI: classify the collection into system / faction / class.
-
-Examples:
-    # catalogue the dlp.py run dir (manifest + downloads)
-    python3 catalogue.py /path/to/run_dir
-
-    # additionally scan manual-era directories
-    python3 catalogue.py /path/to/run_dir --scan /big/storage/manual/Aeldari
-
-    # catalogue ONLY manual dirs, without any run dir
-    python3 catalogue.py --scan /big/storage/manual --out-md catalogue.md
-
-    # fix misclassifications (see overrides format below)
-    python3 catalogue.py <dir> --overrides overrides.json
-
-    # actually move files into <library>/<System>/<Faction>/<Class>/
-    python3 catalogue.py <dir> --apply --library /big/storage/warhammer_library
-
-overrides.json format (keys are file names or full paths, case-insensitive):
-    {
-      "some file.zip": {"faction": "Aeldari", "class": "Heavy Support",
-                        "system": "Warhammer 40,000"}
-    }
-
-Outputs (default into the first given dir, or cwd):
-    catalogue.md    human-readable, grouped system > faction > class
-    catalogue.jsonl one JSON line per file, with source attribution
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-import sys
-from pathlib import Path
-
-from dlphelper.catalogue import (
-    build_catalogue,
-    format_size,
-    apply_moves,
-    write_reports,
-)
-
-log = logging.getLogger("catalogue")
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Classify the Warhammer collection into factions and classes."
-    )
-    parser.add_argument("run_dir", nargs="?", type=Path,
-                        help="a dlp.py run dir (uses its manifest.jsonl and downloads/)")
-    parser.add_argument("--scan", action="append", default=[], type=Path, metavar="DIR",
-                        help="additional directory to scan for archive files (repeatable)")
-    parser.add_argument("--manifest", type=Path, default=None, metavar="FILE",
-                        help="explicit manifest.jsonl (default: <run_dir>/manifest.jsonl)")
-    parser.add_argument("--overrides", type=Path, default=None, metavar="FILE",
-                        help="manual classification overrides (JSON)")
-    parser.add_argument("--out-md", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--out-jsonl", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--apply", action="store_true",
-                        help="move files into the library structure (default: preview)")
-    parser.add_argument("--library", type=Path, default=None, metavar="DIR",
-                        help="library root for --apply (required with --apply)")
-    args = parser.parse_args()
-
-    logging.basicConfig(level=logging.INFO,
-                        format="%(asctime)s %(levelname)-7s %(message)s",
-                        datefmt="%H:%M:%S")
-
-    run_dir = args.run_dir
-    manifests = [args.manifest] if args.manifest else (
-        [run_dir / "manifest.jsonl"] if run_dir else []
-    )
-    scan_dirs = list(args.scan)
-    if run_dir and not args.manifest:
-        # run dir is scanned as a whole (covers files not yet in the manifest)
-        scan_dirs.append(run_dir)
-
-    if not manifests and not scan_dirs:
-        parser.error("give a run_dir or at least one --scan directory")
-
-    out_base = run_dir if run_dir else Path.cwd()
-    out_md = args.out_md or out_base / "catalogue.md"
-    out_jsonl = args.out_jsonl or out_base / "catalogue.jsonl"
-
-    entries = build_catalogue(manifests, scan_dirs, args.overrides)
-    write_reports(entries, out_md, out_jsonl)
-
-    classified = sum(1 for e in entries if e.faction != "Unclassified")
-    log.info("Catalogued %d files (%d classified) -> %s", len(entries), classified, out_md)
-
-    # compact per-faction summary to the console
-    summary: dict[tuple[str, str], list[int]] = {}
-    for e in entries:
-        key = (e.system, e.faction)
-        n, size = summary.get(key, (0, 0))
-        summary[key] = [n + 1, size + e.size]
-    for (system, faction), (n, size) in sorted(summary.items()):
-        log.info("  %-18s %-32s %4d files  %s", system, faction, n, format_size(size))
-
-    if args.apply:
-        if not args.library:
-            parser.error("--apply needs --library DIR")
-        if not args.run_dir:
-            parser.error("--apply needs a run_dir (files are moved out of it)")
-        moved = apply_moves(entries, args.library, args.library / "catalogue_moves.jsonl")
-        log.info("Moved %d files into %s (audit: catalogue_moves.jsonl)", moved, args.library)
-        # rewrite reports with updated paths
-        write_reports(entries, out_md, out_jsonl)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        sys.exit(130)

version-python/data/library_catalog.json

@@ -1,140 +0,0 @@
-{
-  "comment": "Canonical library catalog. Fields: system/group/faction/subfaction/class/name/aliases. Names and aliases are matched case-insensitively; subfaction and class may be null. Aligned with miniset.net faction names.",
-  "entries": [
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": "Space Wolves",
-      "class": "Fast Attack",
-      "name": "Thunderwolf Cavalry",
-      "aliases": ["thunder wolf", "thunderwolves", "thunder wolf cavalry"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": "Space Wolves",
-      "class": "Elites",
-      "name": "Wulfen",
-      "aliases": ["wulfen dreadnought"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of Chaos",
-      "faction": "Chaos Daemons",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Keeper of Secrets",
-      "aliases": ["keeper of secrets and shalaxi helbane", "shalaxi helbane", "keepers of secrets"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "Drukhari",
-      "subfaction": null,
-      "class": null,
-      "name": "The Twin Lance",
-      "aliases": ["twin lance"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Elites",
-      "name": "Terminator Squad",
-      "aliases": ["terminators", "terminator squad"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Elites",
-      "name": "Dreadnought",
-      "aliases": ["dread"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Captain",
-      "aliases": ["captain in gravis armour", "primaris captain"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Heavy Support",
-      "name": "Hellblaster Squad",
-      "aliases": ["hellblasters"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adeptus Astartes",
-      "subfaction": null,
-      "class": "Troops",
-      "name": "Intercessor Squad",
-      "aliases": ["intercessors"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Commander Farsight",
-      "aliases": ["farsight"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "HQ",
-      "name": "Commander Shadowsun",
-      "aliases": ["shadowsun"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "Fast Attack",
-      "name": "Pathfinder Team",
-      "aliases": ["pathfinders"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": "Dedicated Transport",
-      "name": "Devilfish",
-      "aliases": ["devil fish"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Xenos Armies",
-      "faction": "T'au Empire",
-      "subfaction": null,
-      "class": null,
-      "name": "Assorted Drones",
-      "aliases": ["drones"]
-    },
-    {
-      "system": "Warhammer 40,000",
-      "group": "Armies of the Imperium",
-      "faction": "Adepta Sororitas",
-      "subfaction": null,
-      "class": null,
-      "name": "Battle Sisters",
-      "aliases": ["battle sister squad", "battle sisters squad"]
-    }
-  ]
-}

version-python/dlp.py

@@ -1,544 +0,0 @@
-#!/usr/bin/env python3
-"""Warhammer DLP Helper.
-
-Give it a Discord channel/thread id and an output directory; it exports the
-channel, extracts the encoded download links (gofile.io and 0807.st), picks
-the right downloader for each link, and keeps going when individual links
-fail. Everything is recorded in state.json; failures also land in
-failed_urls.txt so a re-run (or --urls-file) retries exactly those.
-
-Already-downloaded URLs are recorded in a global store (default
-~/.local/share/dlphelper/urls.json), so a link that was downloaded for one
-channel is never re-downloaded for another - and re-runs of the same channel
-skip everything finished (see also --assume-done for importing old lists).
-
-Usage:
-    python3 dlp.py <channel_id> <out_dir> [options]
-    python3 dlp.py --subs subscriptions.json [options]
-    python3 dlp.py <channel_id> <out_dir> --urls-file failed_urls.txt
-    python3 dlp.py --assume-done old_urls.txt          # import past downloads
-"""
-
-from __future__ import annotations
-
-import argparse
-import getpass
-import hashlib
-import json
-import logging
-import os
-import random
-import sys
-import time
-import zipfile
-from itertools import chain
-from pathlib import Path
-
-from dlphelper import discord as dlp_discord
-from dlphelper import gofile as dlph_gofile
-from dlphelper import zero807 as dlph_zero807
-from dlphelper.globalstore import GlobalStore, default_global_path
-from dlphelper.state import State, append_manifest
-
-log = logging.getLogger("dlp")
-
-
-def sha256_of(path: Path) -> str:
-    digest = hashlib.sha256()
-    with path.open("rb") as fh:
-        for chunk in iter(lambda: fh.read(1024 * 1024), b""):
-            digest.update(chunk)
-    return digest.hexdigest()
-
-
-def resolve_token(cli_value: str | None) -> str:
-    if cli_value:
-        return cli_value
-    if os.environ.get("DISCORD_TOKEN"):
-        return os.environ["DISCORD_TOKEN"]
-    log.info("No Discord token given ($DISCORD_TOKEN); prompting (input hidden).")
-    token = getpass.getpass("Discord token: ")
-    if not token:
-        raise SystemExit("No token provided, aborting.")
-    return token
-
-
-def read_url_lines(path: Path) -> list[str]:
-    return [
-        line.strip()
-        for line in path.read_text(encoding="utf-8").splitlines()
-        if line.strip() and not line.strip().startswith("#")
-    ]
-
-
-def load_urls_file(path: Path) -> list[dlp_discord.FoundUrl]:
-    """Load a plain URL list (e.g. failed_urls.txt); no attribution available."""
-    found: dict[str, dlp_discord.FoundUrl] = {}
-    for url in read_url_lines(path):
-        host = dlp_discord.classify(url)
-        if host is None:
-            log.warning("Unsupported host in %s: %s", path.name, url)
-            continue
-        found.setdefault(url, dlp_discord.FoundUrl(url=url, host=host))
-    return sorted(found.values(), key=lambda f: f.url)
-
-
-def record_files(
-    state: State,
-    gstore: GlobalStore | None,
-    url: str,
-    files: list[dict],
-    source: dict,
-) -> None:
-    state.mark(url, status="ok", error=None, files=files)
-    if gstore is not None:
-        gstore.mark(url, channel=source.get("channel"), host=source.get("host"),
-                    file_count=len(files))
-    for f in files:
-        append_manifest(Path(source["manifest"]), {
-            "channel": source["channel"],
-            "url": url,
-            "host": source.get("host"),
-            "message_id": source.get("message_id"),
-            "timestamp": source.get("timestamp"),
-            "author": source.get("author"),
-            **f,
-        })
-
-
-def extract_zips(files: list[dict]) -> None:
-    """Optional: unzip each downloaded archive next to itself."""
-    for f in files:
-        path = Path(f["path"])
-        if path.suffix.lower() != ".zip" or not path.exists():
-            continue
-        target = path.with_suffix("")
-        if target.exists() and any(target.iterdir()):
-            log.info("Skip extracting %s (target exists)", path.name)
-            continue
-        log.info("Extracting %s -> %s", path.name, target.name)
-        try:
-            target.mkdir(parents=True, exist_ok=True)
-            with zipfile.ZipFile(path) as zf:
-                zf.extractall(target)
-        except (zipfile.BadZipFile, OSError) as exc:
-            log.warning("Extraction of %s failed: %s", path.name, exc)
-
-
-def run_gofile(
-    urls: list[dlp_discord.FoundUrl],
-    args: argparse.Namespace,
-    state: State,
-    source_by_url: dict[str, dict],
-    is_done,
-    gstore: GlobalStore | None,
-) -> None:
-    if not [f for f in urls if not is_done(f.url)]:
-        return  # nothing pending: no guest account creation, no API calls
-    client = dlph_gofile.GoFileClient(password=args.password)
-    try:
-        client.ensure_token()
-    except dlph_gofile.GoFileError as exc:
-        log.error("%s - marking all pending gofile links as failed.", exc)
-        for f in urls:
-            if is_done(f.url):
-                continue
-            state.mark(f.url, status="failed", error=str(exc), files=[])
-        return
-
-    consecutive_failures = 0
-    for f in urls:
-        if is_done(f.url):
-            log.info("Skip (already downloaded): %s", f.url)
-            continue
-        source = source_by_url[f.url]
-
-        for attempt in range(1, args.retries + 1):
-            try:
-                files_spec = client.list_files(f.url)
-                if not files_spec:
-                    raise dlph_gofile.GoFileEmpty("content exists but contains no files")
-                files: list[dict] = []
-                for relpath, name, link in files_spec:
-                    dest = Path(args.out_dir) / "downloads" / "gofile" / relpath / name
-                    size = client.download_file(link, dest)
-                    entry = {"path": str(dest), "size": size, "name": name}
-                    if args.hash:
-                        entry["sha256"] = sha256_of(dest)
-                    files.append(entry)
-                log.info("OK %s (%d files)", f.url, len(files))
-                record_files(state, gstore, f.url, files, source)
-                if args.extract:
-                    extract_zips(files)
-                consecutive_failures = 0
-                break
-            except dlph_gofile.GoFilePasswordRequired as exc:
-                # Retrying without a password never helps
-                log.error("PASSWORD REQUIRED %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileNotFound as exc:
-                # Dead links never come back - no retry
-                log.error("DEAD LINK %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileEmpty as exc:
-                # Emptied content is stable - no retry
-                log.error("EMPTY CONTENT %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_gofile.GoFileAuthError as exc:
-                # Token may have died mid-run (rate limit); re-auth once per attempt
-                log.warning("Auth error for %s: %s - re-authenticating", f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                client.reset()
-                try:
-                    client.ensure_token()
-                except dlph_gofile.GoFileError as reauth_exc:
-                    log.error("Re-authentication failed: %s", reauth_exc)
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except dlph_gofile.GoFileError as exc:
-                log.warning("Attempt %d/%d failed for %s: %s",
-                            attempt, args.retries, f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except Exception as exc:  # noqa: BLE001 - a bug must not stop the run
-                log.exception("Unexpected error for %s: %s", f.url, exc)
-                state.mark(f.url, error=f"unexpected: {exc}", files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except KeyboardInterrupt:
-                state.mark(f.url, error="interrupted", files=[])
-                raise
-        else:
-            rec = state.records.get(f.url, {})
-            state.mark(f.url, status="failed", error=rec.get("error", "unknown"), files=[])
-            consecutive_failures += 1
-            log.error("FAILED %s after %d attempts", f.url, args.retries)
-            if consecutive_failures >= args.cooldown_failures:
-                log.warning(
-                    "%d failures in a row - cooling down %ds before continuing "
-                    "(rate limit?). The run does not stop.",
-                    consecutive_failures, args.cooldown_seconds,
-                )
-                time.sleep(args.cooldown_seconds)
-
-
-def run_0807(
-    urls: list[dlp_discord.FoundUrl],
-    args: argparse.Namespace,
-    state: State,
-    source_by_url: dict[str, dict],
-    is_done,
-    gstore: GlobalStore | None,
-) -> None:
-    if not [f for f in urls if not is_done(f.url)]:
-        return
-    out_dir = Path(args.out_dir) / "downloads" / "0807"
-    consecutive_failures = 0
-    for f in urls:
-        if is_done(f.url):
-            log.info("Skip (already downloaded): %s", f.url)
-            continue
-        source = source_by_url[f.url]
-
-        for attempt in range(1, args.retries + 1):
-            try:
-                dest, size = dlph_zero807.download(f.url, out_dir)
-                entry: dict = {"path": str(dest), "size": size, "name": dest.name}
-                if args.hash:
-                    entry["sha256"] = sha256_of(dest)
-                log.info("OK %s (%s)", f.url, format_size(size))
-                record_files(state, gstore, f.url, [entry], source)
-                if args.extract:
-                    extract_zips([entry])
-                consecutive_failures = 0
-                break
-            except dlph_zero807.Zero807NotFound as exc:
-                # Dead links never come back - no retry
-                log.error("DEAD LINK %s: %s", f.url, exc)
-                state.mark(f.url, status="failed", error=str(exc), files=[])
-                break
-            except dlph_zero807.Zero807Error as exc:
-                log.warning("Attempt %d/%d failed for %s: %s",
-                            attempt, args.retries, f.url, exc)
-                state.mark(f.url, error=str(exc), files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except Exception as exc:  # noqa: BLE001 - a bug must not stop the run
-                log.exception("Unexpected error for %s: %s", f.url, exc)
-                state.mark(f.url, error=f"unexpected: {exc}", files=[])
-                if attempt < args.retries:
-                    time.sleep(args.retry_delay * attempt)
-            except KeyboardInterrupt:
-                state.mark(f.url, error="interrupted", files=[])
-                raise
-        else:
-            rec = state.records.get(f.url, {})
-            state.mark(f.url, status="failed", error=rec.get("error", "unknown"), files=[])
-            consecutive_failures += 1
-            log.error("FAILED %s after %d attempts", f.url, args.retries)
-            if consecutive_failures >= args.cooldown_failures:
-                log.warning(
-                    "%d failures in a row - cooling down %ds before continuing "
-                    "(HTTP 403 storm?). The run does not stop.",
-                    consecutive_failures, args.cooldown_seconds,
-                )
-                time.sleep(args.cooldown_seconds)
-
-
-def format_size(n: int) -> str:
-    size = float(n)
-    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
-        if size < 1024 or unit == "TiB":
-            return f"{size:.1f} {unit}"
-        size /= 1024
-    return f"{n} B"
-
-
-def write_failed_urls(out_dir: Path, state: State) -> Path:
-    dest = out_dir / "failed_urls.txt"
-    dest.parent.mkdir(parents=True, exist_ok=True)
-    failures = state.failures()
-    dest.write_text("\n".join(failures) + ("\n" if failures else ""), encoding="utf-8")
-    return dest
-
-
-def mark_assume_done(path: Path, gstore: GlobalStore | None) -> int:
-    """Record URLs as already downloaded (no files) so they are never fetched.
-
-    Manual import step for the pre-dlp.py era: point it at an old URL list you
-    know is fully downloaded. There is no verification - garbage in, skipped
-    forever.
-    """
-    if gstore is None:
-        raise SystemExit("--assume-done needs the global store (drop --no-global)")
-    count = 0
-    for url in read_url_lines(path):
-        gstore.mark(url, channel=None, host=dlp_discord.classify(url), file_count=0,
-                    note="assumed done via --assume-done")
-        count += 1
-    log.info("Marked %d URLs as done in the global store (%s)", count, gstore.path)
-    return count
-
-
-def process_channel(
-    channel_id: str,
-    out_dir: Path,
-    args: argparse.Namespace,
-    token: str | None,
-    gstore: GlobalStore | None,
-    force_export: bool = False,
-) -> int:
-    """Run the full pipeline for one channel; returns the number of failures."""
-    state = State(out_dir / "state.json")
-    manifest = out_dir / "manifest.jsonl"
-
-    def is_done(url: str) -> bool:
-        return state.is_done(url) or (gstore is not None and gstore.is_done(url))
-
-    if args.urls_file:
-        found = load_urls_file(args.urls_file)
-    else:
-        if token is None:
-            raise SystemExit("internal error: no token resolved")
-        export_path = dlp_discord.export_channel(
-            channel_id, out_dir, token, force=force_export or args.fresh
-        )
-        found = dlp_discord.extract_urls(export_path)
-
-    if args.only:
-        found = [f for f in found if f.host == args.only]
-
-    buckets: dict[str, list[dlp_discord.FoundUrl]] = {"gofile": [], "0807": []}
-    for f in found:
-        if f.host in buckets:
-            buckets[f.host].append(f)
-        else:  # classify() already filtered, but be defensive
-            log.warning("Skipping unsupported URL: %s", f.url)
-            state.mark(f.url, status="failed", error="unsupported host")
-
-    source_by_url: dict[str, dict] = {
-        f.url: {
-            "channel": channel_id,
-            "host": f.host,
-            "message_id": f.message_id,
-            "timestamp": f.timestamp,
-            "author": f.author,
-            "manifest": str(manifest),
-        }
-        for f in found
-    }
-
-    # --limit applies per host class to the *pending* URLs, so a test run
-    # downloads exactly N new things per host (done ones don't use up the quota)
-    if args.limit is not None:
-        for key, items in buckets.items():
-            pending = [f for f in items if not is_done(f.url)]
-            if args.shuffle:
-                random.shuffle(pending)
-            buckets[key] = pending[: args.limit]
-
-    pending = [f for f in found if not is_done(f.url)]
-    if args.limit is not None:
-        pending = [f for f in chain.from_iterable(buckets.values()) if not is_done(f.url)]
-    log.info(
-        "%s: %d URLs (%d gofile, %d 0807); %d done, %d to download",
-        channel_id, len(found),
-        len(buckets["gofile"]), len(buckets["0807"]),
-        len(found) - len(pending), len(pending),
-    )
-
-    if args.dry_run:
-        selected = {f.url for f in chain.from_iterable(buckets.values())}
-        for f in found:
-            if f.url in selected:
-                flag = "download"
-            elif is_done(f.url):
-                flag = "done"
-            else:
-                flag = "skip"  # cut by --limit / --only
-            print(f"[{f.host:6}] [{flag:8}] {f.url}")
-        return 0
-
-    if buckets["gofile"]:
-        run_gofile(buckets["gofile"], args, state, source_by_url, is_done, gstore)
-    if buckets["0807"]:
-        run_0807(buckets["0807"], args, state, source_by_url, is_done, gstore)
-
-    failed_file = write_failed_urls(out_dir, state)
-
-    ok = [u for u, r in state.records.items() if r.get("status") == "ok"]
-    bad = state.failures()
-    dead = sum(1 for r in state.records.values() if "dead link" in (r.get("error") or ""))
-    total_files = sum(len(r.get("files") or []) for r in state.records.values())
-    total_size = sum(
-        (f_.get("size") or 0) for r in state.records.values() for f_ in (r.get("files") or [])
-    )
-    log.info("%s: %d ok (%d files, %s) / %d failed (%d dead links) / %d seen",
-             out_dir, len(ok), total_files, format_size(total_size),
-             len(bad), dead, len(state.records))
-    if bad:
-        log.info("Failed URLs written to %s - re-run this tool or use "
-                 "python3 dlp.py <id> <out> --urls-file %s to retry them.",
-                 failed_file, failed_file.name)
-    return len(bad)
-
-
-def load_subscriptions(path: Path) -> list[dict]:
-    """Load subscriptions.json: [{"name": ..., "channel_id": ..., "out_dir": ...}]."""
-    data = json.loads(path.read_text(encoding="utf-8"))
-    entries = data.get("subscriptions", []) if isinstance(data, dict) else data
-    if not isinstance(entries, list):
-        raise SystemExit(f"{path}: expected a list or {'{'}\"subscriptions\": [...]{'}'}")
-    subs = []
-    for entry in entries:
-        if not isinstance(entry, dict) or "channel_id" not in entry or "out_dir" not in entry:
-            raise SystemExit(
-                f"{path}: each subscription needs 'channel_id' and 'out_dir' "
-                f"(got: {entry})"
-            )
-        subs.append(entry)
-    return subs
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Collect downloads from Discord channels (gofile + 0807)."
-    )
-    parser.add_argument("channel_id", nargs="?", help="Discord channel or thread id")
-    parser.add_argument("out_dir", nargs="?", help="directory for downloads and state")
-    parser.add_argument("--subs", type=Path, metavar="FILE",
-                        help="process a subscription list instead of one channel "
-                             "(entries: channel_id, out_dir, optional name)")
-    parser.add_argument("--token", help="Discord token (else $DISCORD_TOKEN or prompt)")
-    parser.add_argument("--urls-file", type=Path,
-                        help="skip the Discord export; read URLs from this file "
-                             "(e.g. failed_urls.txt)")
-    parser.add_argument("--password", help="password for password-protected gofile links")
-    parser.add_argument("--retries", type=int, default=3, help="attempts per URL (default 3)")
-    parser.add_argument("--retry-delay", type=int, default=5,
-                        help="base seconds between retries, multiplied per attempt (default 5)")
-    parser.add_argument("--cooldown-failures", type=int, default=3,
-                        help="consecutive failures before a cooldown pause (default 3)")
-    parser.add_argument("--cooldown-seconds", type=int, default=60,
-                        help="length of the cooldown pause (default 60)")
-    parser.add_argument("--only", choices=("gofile", "0807"),
-                        help="process only one host class")
-    parser.add_argument("--limit", type=int, metavar="N",
-                        help="download at most N pending URLs per host class "
-                             "(test runs)")
-    parser.add_argument("--shuffle", action="store_true",
-                        help="randomize which URLs a --limit picks")
-    parser.add_argument("--hash", action="store_true",
-                        help="record sha256 of each downloaded file")
-    parser.add_argument("--extract", action="store_true",
-                        help="unzip downloaded archives into per-archive folders")
-    parser.add_argument("--fresh", action="store_true",
-                        help="re-export the Discord channel even if export.json exists "
-                             "(subs mode does this by default)")
-    parser.add_argument("--no-fresh", action="store_true",
-                        help="subs mode: reuse cached exports instead of re-exporting")
-    parser.add_argument("--dry-run", action="store_true",
-                        help="export/extract/classify only, download nothing")
-    parser.add_argument("--global-state", type=Path, metavar="FILE",
-                        help=f"cross-run URL store (default: {default_global_path()})")
-    parser.add_argument("--no-global", action="store_true",
-                        help="disable the cross-run URL store for this run")
-    parser.add_argument("--assume-done", type=Path, metavar="FILE",
-                        help="mark every URL in the file as already downloaded, "
-                             "then exit (manual import of old download lists)")
-    args = parser.parse_args()
-
-    logging.basicConfig(
-        level=logging.INFO,
-        format="%(asctime)s %(levelname)-7s %(message)s",
-        datefmt="%H:%M:%S",
-    )
-
-    gstore = None
-    if not args.no_global:
-        gstore = GlobalStore(args.global_state or default_global_path())
-
-    if args.assume_done:
-        if args.channel_id or args.subs:
-            parser.error("--assume-done cannot be combined with a channel or --subs")
-        mark_assume_done(args.assume_done, gstore)
-        return
-
-    if args.subs:
-        if args.channel_id or args.out_dir or args.urls_file:
-            parser.error("--subs cannot be combined with channel_id/out_dir/--urls-file")
-        subs = load_subscriptions(args.subs)
-        token = resolve_token(args.token)
-        total_failed = 0
-        for entry in subs:
-            out_dir = Path(entry["out_dir"])
-            log.info("=== subscription: %s ===", entry.get("name") or entry["channel_id"])
-            total_failed += process_channel(
-                entry["channel_id"], out_dir, args, token, gstore,
-                force_export=not args.no_fresh,  # subs mode re-exports by default
-            )
-        if total_failed:
-            log.warning("%d URL(s) failed in total across subscriptions.", total_failed)
-            sys.exit(1)
-        return
-
-    if not args.channel_id or not args.out_dir:
-        parser.error("channel_id and out_dir are required (or use --subs/--assume-done)")
-
-    # A token is only needed when we will export a channel (--urls-file skips that)
-    token = None if args.urls_file else resolve_token(args.token)
-    failures = process_channel(args.channel_id, Path(args.out_dir), args, token, gstore)
-    if failures:
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        log.warning("Interrupted - state.json is saved; re-run to resume.")
-        sys.exit(130)

version-python/dlphelper/__init__.py

@@ -1,3 +0,0 @@
-"""Warhammer DLP Helper - collect and organize files shared in Discord channels."""
-
-__version__ = "0.1.0"

version-python/dlphelper/catalogue.py

@@ -1,305 +0,0 @@
-"""Catalogue: classify collected files into game system / faction / class.
-
-The catalogue is the bridge between "a pile of downloads" and the later
-library stage: it reads manifest.jsonl (with full source attribution) and/or
-scans arbitrary directories for archive files, classifies each file against
-the taxonomy in warhammer_data.py, and writes machine- (catalogue.jsonl) and
-human-readable (catalogue.md) reports.
-
-Classification is keyword-based and deliberately conservative: files that
-cannot be classified land in the "unclassified" bucket so gaps become visible
-instead of producing wrong assignments. A manual overrides file can pin
-individual files.
-
-Nothing is moved by default - use --apply for that (it moves files into
-<library-root>/<System>/<Faction>/<Class>/ and records every move in
-catalogue_moves.jsonl so it can be audited/undone).
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import re
-import shutil
-from dataclasses import dataclass, field
-from pathlib import Path
-
-from dlphelper.warhammer_data import (
-    FACTION_ALIASES,
-    GROUP_OF_FACTION,
-    SYSTEMS,
-    UNIT_CLASSES,
-)
-
-log = logging.getLogger(__name__)
-
-ARCHIVE_SUFFIXES = {".zip", ".7z", ".rar", ".tar", ".gz"}
-UNKNOWN = "Unknown"
-UNCLASSIFIED_FACTION = "Unclassified"
-UNCLASSIFIED_CLASS = "Unclassified"
-
-
-@dataclass
-class CatalogueEntry:
-    """One classified file."""
-
-    path: str
-    name: str
-    size: int
-    system: str = UNKNOWN
-    faction: str = UNCLASSIFIED_FACTION
-    klass: str = UNCLASSIFIED_CLASS
-    source: dict = field(default_factory=dict)  # manifest attribution if known
-
-
-from functools import lru_cache
-
-
-@lru_cache(maxsize=None)
-def _clean_name(name: str) -> str:
-    stem = Path(name).stem
-    stem = re.sub(r"[_\.\+\[\]\(\)]+", " ", stem)
-    stem = re.sub(r"\s+", " ", stem).strip().lower()
-    return stem
-
-
-def _kw_regex(keyword: str) -> str:
-    """Word-boundary regex for a keyword, tolerating a plural 's'."""
-    return rf"(?<![a-z0-9]){re.escape(keyword)}s?(?![a-z0-9])"
-
-
-@lru_cache(maxsize=None)
-def _match_faction(clean: str) -> tuple[str, str] | None:
-    """Return (system, faction) with the most specific keyword hit."""
-    best: tuple[int, str, str] | None = None  # (len, system, faction)
-    for system, aliases in FACTION_ALIASES.items():
-        for keyword, faction in aliases.items():
-            if re.search(_kw_regex(keyword), clean):
-                if best is None or len(keyword) > best[0]:
-                    best = (len(keyword), system, faction)
-    return (best[1], best[2]) if best else None
-
-
-@lru_cache(maxsize=None)
-def _match_class(clean: str) -> str | None:
-    """Return the most specific force-organization class keyword hit."""
-    best: tuple[int, str] | None = None
-    for keyword, klass in UNIT_CLASSES.items():
-        if re.search(_kw_regex(keyword), clean):
-            if best is None or len(keyword) > best[0]:
-                best = (len(keyword), klass)
-    return best[1] if best else None
-
-
-def classify(name: str) -> tuple[str, str, str]:
-    """Classify a file name -> (system, faction, class)."""
-    clean = _clean_name(name)
-    faction_hit = _match_faction(clean)
-    klass = _match_class(clean)
-
-    if faction_hit:
-        system, faction = faction_hit
-        # Non-40k systems: keep their class if we found one, else Unclassified
-        if system != "Warhammer 40,000" and klass is None:
-            return system, faction, UNCLASSIFIED_CLASS
-        return system, faction, klass or UNCLASSIFIED_CLASS
-
-    # No faction match: a class keyword alone cannot attribute a system
-    return UNKNOWN, UNCLASSIFIED_FACTION, UNCLASSIFIED_CLASS
-
-
-def scan_manifest(manifest_path: Path) -> list[CatalogueEntry]:
-    """Entries from a manifest.jsonl (preferred: has source attribution)."""
-    entries: list[CatalogueEntry] = []
-    seen: set[str] = set()
-    if not manifest_path.exists():
-        return entries
-    for line in manifest_path.read_text(encoding="utf-8").splitlines():
-        line = line.strip()
-        if not line:
-            continue
-        try:
-            row = json.loads(line)
-        except ValueError:
-            log.warning("Skipping malformed manifest line: %s", line[:80])
-            continue
-        path = row.get("path") or ""
-        if path in seen:
-            continue
-        seen.add(path)
-        entries.append(CatalogueEntry(
-            path=path,
-            name=row.get("name") or Path(path).name,
-            size=int(row.get("size") or 0),
-            source={
-                "channel": row.get("channel"),
-                "url": row.get("url"),
-                "message_id": row.get("message_id"),
-                "timestamp": row.get("timestamp"),
-                "author": row.get("author"),
-            },
-        ))
-    return entries
-
-
-def scan_directory(directory: Path) -> list[CatalogueEntry]:
-    """Entries from walking a directory tree for archive files (manual era)."""
-    entries: list[CatalogueEntry] = []
-    for path in sorted(directory.rglob("*")):
-        if not path.is_file() or path.suffix.lower() not in ARCHIVE_SUFFIXES:
-            continue
-        try:
-            size = path.stat().st_size
-        except OSError:
-            size = 0
-        entries.append(CatalogueEntry(path=str(path), name=path.name, size=size))
-    return entries
-
-
-def load_overrides(path: Path | None) -> dict[str, dict]:
-    """overrides.json: {"/path/or/filename": {"faction": ..., "class": ..., "system": ...}}"""
-    if path is None or not path.exists():
-        return {}
-    return json.loads(path.read_text(encoding="utf-8"))
-
-
-def apply_overrides(entries: list[CatalogueEntry], overrides: dict[str, dict]) -> int:
-    count = 0
-    by_name = {e.name.lower(): e for e in entries}
-    for key, fix in overrides.items():
-        entry = by_name.get(key.lower())
-        if entry is None:
-            log.warning("Override target not found: %s", key)
-            continue
-        if "system" in fix:
-            entry.system = fix["system"]
-        if "faction" in fix:
-            entry.faction = fix["faction"]
-        if "class" in fix:
-            entry.klass = fix["class"]
-        count += 1
-    return count
-
-
-def format_size(n: int) -> str:
-    size = float(n)
-    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
-        if size < 1024 or unit == "TiB":
-            return f"{size:.1f} {unit}"
-        size /= 1024
-    return f"{n} B"
-
-
-def write_reports(
-    entries: list[CatalogueEntry],
-    out_md: Path,
-    out_jsonl: Path,
-) -> None:
-    # machine report
-    with out_jsonl.open("w", encoding="utf-8") as fh:
-        for e in entries:
-            fh.write(json.dumps({
-                "path": e.path,
-                "name": e.name,
-                "size": e.size,
-                "system": e.system,
-                "faction": e.faction,
-                "class": e.klass,
-                "source": {k: v for k, v in e.source.items() if v},
-            }, ensure_ascii=False) + "\n")
-
-    # human report
-    grouped: dict[str, dict[str, dict[str, list[CatalogueEntry]]]] = {}
-    for e in entries:
-        grouped.setdefault(e.system, {}).setdefault(e.faction, {}).setdefault(
-            e.klass, []).append(e)
-
-    lines: list[str] = ["# Warhammer Collection Catalogue", ""]
-    total_size = sum(e.size for e in entries)
-    classified = sum(1 for e in entries if e.faction != UNCLASSIFIED_FACTION)
-    lines.append(f"{len(entries)} files, {format_size(total_size)} total, "
-                 f"{classified} classified, "
-                 f"{len(entries) - classified} unclassified")
-    lines.append("")
-
-    for system in sorted(grouped):
-        factions = grouped[system]
-        all_sys_files = [e for f in factions.values() for c in f.values() for e in c]
-        sys_size = sum(e.size for e in all_sys_files)
-        lines.append(f"## {system} ({len(all_sys_files)} files, {format_size(sys_size)})")
-        for faction in sorted(factions):
-            classes = factions[faction]
-            fac_size = sum(e.size for c in classes.values() for e in c)
-            lines.append(f"### {faction} ({format_size(fac_size)})")
-            for klass in sorted(classes):
-                files = classes[klass]
-                lines.append(f"- **{klass}** ({len(files)} files, {format_size(sum(e.size for e in files))})")
-                for e in sorted(files, key=lambda x: x.name.lower()):
-                    src = ""
-                    if e.source.get("url"):
-                        src = f" — from {e.source['url']}"
-                    lines.append(f"  - {e.name} ({format_size(e.size)}){src}")
-            lines.append("")
-
-    out_md.write_text("\n".join(lines), encoding="utf-8")
-
-
-def apply_moves(
-    entries: list[CatalogueEntry],
-    library_root: Path,
-    moves_log: Path,
-) -> int:
-    """Move classified files into the library structure. Audited in moves_log."""
-    moves_log.parent.mkdir(parents=True, exist_ok=True)
-    moved = 0
-    with moves_log.open("a", encoding="utf-8") as logfh:
-        for e in entries:
-            if e.faction == UNCLASSIFIED_FACTION:
-                continue  # never move what we cannot classify
-            if e.system == UNKNOWN:
-                continue
-            src = Path(e.path)
-            if not src.exists():
-                log.warning("Cannot move, missing: %s", src)
-                continue
-            dest_dir = (
-                library_root / e.system
-                / GROUP_OF_FACTION.get(e.faction, "Unclassified")
-                / e.faction / e.klass
-            )
-            dest = dest_dir / src.name
-            if dest.exists():
-                log.warning("Skip, destination exists: %s", dest)
-                continue
-            dest_dir.mkdir(parents=True, exist_ok=True)
-            shutil.move(str(src), str(dest))
-            logfh.write(json.dumps({"from": str(src), "to": str(dest)}) + "\n")
-            e.path = str(dest)
-            moved += 1
-    return moved
-
-
-def build_catalogue(
-    manifests: list[Path],
-    scan_dirs: list[Path],
-    overrides_path: Path | None,
-) -> list[CatalogueEntry]:
-    entries: list[CatalogueEntry] = []
-    seen: set[str] = set()
-    for m in manifests:
-        for e in scan_manifest(m):
-            if e.path not in seen:
-                seen.add(e.path)
-                entries.append(e)
-    for d in scan_dirs:
-        for e in scan_directory(d):
-            if e.path not in seen:
-                seen.add(e.path)
-                entries.append(e)
-
-    for e in entries:
-        e.system, e.faction, e.klass = classify(e.name)
-
-    apply_overrides(entries, load_overrides(overrides_path))
-    return entries

version-python/dlphelper/discord.py

@@ -1,160 +0,0 @@
-"""Export a Discord channel/thread with DiscordChatExporter and extract URLs.
-
-The channels this tool targets encode their download links as base64 inside
-message content, and occasionally as plain links. We export the channel as
-JSON (richer metadata than the HTML export: message id, timestamp, author),
-decode every base64 candidate, and keep only URLs for supported hosts.
-"""
-
-from __future__ import annotations
-
-import base64
-import json
-import logging
-import os
-import re
-import subprocess
-from dataclasses import dataclass
-from pathlib import Path
-from urllib.parse import urlparse
-
-log = logging.getLogger(__name__)
-
-DCE_IMAGE = "tyrrrz/discordchatexporter:stable"
-
-# The channels encode links as base64 that always starts with this prefix
-# (https://...); grab.sh established this regex, keep it compatible.
-B64_RE = re.compile(r"aHR0cHM6Ly[89][A-Za-z0-9+/_-]*")
-URL_RE = re.compile(r"https://[^\s<>\"'\)\]\}]+")
-
-SUPPORTED_HOSTS = {
-    "gofile": ("gofile.io", "www.gofile.io"),
-    "0807": ("0807.st", "www.0807.st"),
-}
-
-
-@dataclass(frozen=True)
-class FoundUrl:
-    """One download URL found in the channel, with source attribution."""
-
-    url: str
-    host: str  # "gofile" | "0807" (set after classification)
-    message_id: str | None = None
-    timestamp: str | None = None
-    author: str | None = None
-    encoded: bool = False  # found via base64 decode vs. plain link
-
-
-def export_channel(channel_id: str, out_dir: Path, token: str, force: bool = False) -> Path:
-    """Export the channel to out_dir/export.json using the DCE docker image.
-
-    The token is passed to the container via the DISCORD_TOKEN env var (DCE
-    reads it natively) so it never appears on the docker command line. TOKEN
-    is set as well for DCE versions that only know that name.
-    """
-    dest = out_dir / "export.json"
-    if dest.exists() and not force:
-        log.info("Reusing existing export %s (pass --fresh to re-export)", dest)
-        return dest
-
-    out_dir.mkdir(parents=True, exist_ok=True)
-    env = dict(os.environ)
-    env["DISCORD_TOKEN"] = token
-    env["TOKEN"] = token
-    cmd = [
-        "docker", "run", "--rm",
-        "--user", f"{os.getuid()}:{os.getgid()}",
-        "-v", f"{out_dir.resolve()}:/out",
-        "-e", "DISCORD_TOKEN",
-        "-e", "TOKEN",
-        DCE_IMAGE,
-        "export", "-c", channel_id, "-f", "Json", "-o", "/out/export.json",
-    ]
-    log.info("Exporting channel %s via DiscordChatExporter ...", channel_id)
-    try:
-        subprocess.run(cmd, env=env, check=True)
-    except FileNotFoundError as exc:
-        raise RuntimeError(
-            "docker not found - the Discord export needs the docker image "
-            f"{DCE_IMAGE}"
-        ) from exc
-    except subprocess.CalledProcessError as exc:
-        raise RuntimeError(
-            f"DiscordChatExporter failed with exit code {exc.returncode} "
-            "(check the token and the channel id)"
-        ) from exc
-    if not dest.exists():
-        raise RuntimeError("DiscordChatExporter produced no output file")
-    return dest
-
-
-def classify(url: str) -> str | None:
-    """Return the host key ('gofile' | '0807') for a URL, else None."""
-    try:
-        hostname = (urlparse(url).hostname or "").lower()
-    except ValueError:
-        return None
-    for key, hosts in SUPPORTED_HOSTS.items():
-        if hostname in hosts:
-            return key
-    return None
-
-
-def _decode_b64(candidate: str) -> str | None:
-    s = candidate.replace("_", "/").replace("-", "+")
-    s = s.rstrip("=")
-    s += "=" * ((4 - len(s) % 4) % 4)
-    try:
-        decoded = base64.b64decode(s, validate=False).decode("utf-8", "strict")
-    except Exception:  # noqa: BLE001 - malformed candidate, just skip it
-        return None
-    # One encoded blob can contain the URL plus padding/newlines
-    stripped = decoded.strip()
-    line = stripped.splitlines()[0] if stripped else ""
-    return line or None
-
-
-def extract_urls(export_path: Path) -> list[FoundUrl]:
-    """Extract supported URLs from a DCE JSON export, deduplicated."""
-    data = json.loads(export_path.read_text(encoding="utf-8"))
-    messages = data if isinstance(data, list) else data.get("messages", [])
-
-    found: dict[str, FoundUrl] = {}
-    stats = {"b64": 0, "plain": 0, "unsupported": 0}
-
-    for msg in messages:
-        content = str(msg.get("content") or "")
-        if not content:
-            continue
-        author = (msg.get("author") or {}).get("name")
-        attrib = {
-            "message_id": str(msg.get("id") or "") or None,
-            "timestamp": msg.get("timestamp"),
-            "author": author,
-        }
-
-        for match in B64_RE.finditer(content):
-            stats["b64"] += 1
-            decoded = _decode_b64(match.group(0))
-            if not decoded:
-                continue
-            host = classify(decoded)
-            if host is None:
-                stats["unsupported"] += 1
-                continue
-            found.setdefault(decoded, FoundUrl(url=decoded, host=host, encoded=True, **attrib))
-
-        for match in URL_RE.finditer(content):
-            url = match.group(0).rstrip(".,;)")
-            host = classify(url)
-            if host is None:
-                continue
-            stats["plain"] += 1
-            found.setdefault(url, FoundUrl(url=url, host=host, encoded=False, **attrib))
-
-    log.info(
-        "Extraction: %d base64 candidates, %d plain links, %d unsupported/undecodable, "
-        "%d unique supported URLs",
-        stats["b64"], stats["plain"], stats["unsupported"], len(found),
-    )
-    return sorted(found.values(), key=lambda f: f.url)

version-python/dlphelper/globalstore.py

@@ -1,70 +0,0 @@
-"""Cross-run, cross-channel record of completed URLs.
-
-This is what prevents re-downloading your collection: state.json only knows
-about one OUT_DIR, but the same gofile link posted in a second channel would
-otherwise be downloaded again into that channel's OUT_DIR. The global store
-(default: ~/.local/share/dlphelper/urls.json) records every successfully
-downloaded URL once, forever, regardless of OUT_DIR.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import os
-import time
-from pathlib import Path
-
-log = logging.getLogger(__name__)
-
-
-def default_global_path() -> Path:
-    data_home = os.environ.get("XDG_DATA_HOME") or str(
-        Path.home() / ".local" / "share"
-    )
-    return Path(data_home) / "dlphelper" / "urls.json"
-
-
-class GlobalStore:
-    """JSON file mapping url -> {status, channel, host, files, updated}."""
-
-    def __init__(self, path: Path | None = None):
-        self.path = path
-        self.records: dict[str, dict] = {}
-        if self.path and self.path.exists():
-            try:
-                self.records = json.loads(self.path.read_text(encoding="utf-8"))
-                if not isinstance(self.records, dict):
-                    raise ValueError("global store root must be an object")
-            except (ValueError, OSError) as exc:
-                backup = self.path.with_suffix(".json.bak")
-                log.warning("Corrupt global store %s (%s); backed up to %s",
-                            self.path, exc, backup)
-                self.path.replace(backup)
-                self.records = {}
-
-    def is_done(self, url: str) -> bool:
-        if not self.records:
-            return False
-        rec = self.records.get(url)
-        return bool(rec) and rec.get("status") == "ok"
-
-    def mark(self, url: str, **fields) -> None:
-        if self.path is None:
-            return
-        rec = self.records.setdefault(url, {"status": "ok"})
-        for key, value in fields.items():
-            if value is not None:
-                rec[key] = value
-        rec["status"] = "ok"
-        rec["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
-        self.save()
-
-    def save(self) -> None:
-        if self.path is None:
-            return
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        tmp = self.path.with_suffix(".json.tmp")
-        tmp.write_text(json.dumps(self.records, indent=2, sort_keys=True),
-                       encoding="utf-8")
-        tmp.replace(self.path)

version-python/dlphelper/gofile.py

@@ -1,239 +0,0 @@
-"""Minimal GoFile API client - one guest account per run.
-
-The API interaction mirrors Lysagxra/GoFileDownloader (same headers, website
-token scheme and endpoints), but with two deliberate differences:
-
-1. A single guest account is created per run (GoFileDownloader creates one per
-   URL, which trips Gofile's rate limit after ~6 requests).
-2. Failures raise exceptions instead of sys.exit(), so the orchestrator can
-   track them per file/URL and keep going.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import logging
-import os
-import time
-from pathlib import Path
-from urllib.parse import urlparse
-
-import requests
-
-log = logging.getLogger(__name__)
-
-API = "https://api.gofile.io"
-USER_AGENT = "Mozilla/5.0"
-LOCALE = "en-US"
-TOKEN_SECRET = "12af056dacea0b"  # public website-token seed, see GoFileDownloader  # noqa: S105
-TOKEN_WINDOW_SEC = 14400
-CHUNK_SIZE = 1024 * 1024
-
-COMMON_HEADERS = {
-    "User-Agent": USER_AGENT,
-    "Accept-Encoding": "gzip",
-    "Accept": "*/*",
-    "Connection": "keep-alive",
-}
-EXTENDED_HEADERS = {
-    **COMMON_HEADERS,
-    "Sec-Fetch-Dest": "empty",
-    "Sec-Fetch-Mode": "cors",
-    "Sec-Fetch-Site": "same-site",
-    "Pragma": "no-cache",
-    "Cache-Control": "no-cache",
-}
-
-
-class GoFileError(Exception):
-    """Base error for GoFile operations."""
-
-
-class GoFilePasswordRequired(GoFileError):
-    """The content is password protected and none (or a wrong one) was given."""
-
-
-class GoFileNotFound(GoFileError):
-    """The content/file no longer exists (404 / status notFound).
-
-    Dead links never come back, so the orchestrator must not retry these.
-    """
-
-
-class GoFileEmpty(GoFileError):
-    """The content exists but contains no files (e.g. emptied folder).
-
-    Stable state like a dead link: no retry.
-    """
-
-
-class GoFileAuthError(GoFileError):
-    """The account token was rejected; the client can re-auth once."""
-
-
-def content_id_from_url(url: str) -> str:
-    """Extract the content id from a gofile.io/d/<id> URL."""
-    parts = [p for p in urlparse(url).path.split("/") if p]
-    if len(parts) >= 2 and parts[-2] == "d":
-        return parts[-1]
-    raise GoFileError(f"not a gofile content URL (expected gofile.io/d/<id>): {url}")
-
-
-class GoFileClient:
-    """GoFile API client that reuses one guest account token for the whole run."""
-
-    def __init__(self, password: str | None = None):
-        self._password_hash = (
-            hashlib.sha256(password.encode()).hexdigest() if password else None
-        )
-        self._token: str | None = None
-        self.session = requests.Session()
-
-    # -- authentication -----------------------------------------------------
-
-    def ensure_token(self, max_attempts: int = 8, wait_seconds: int = 20) -> str:
-        """Create a guest account; backs off and retries on rate limiting."""
-        if self._token:
-            return self._token
-
-        last_status = "unknown"
-        for attempt in range(1, max_attempts + 1):
-            try:
-                resp = self.session.post(
-                    f"{API}/accounts", headers=COMMON_HEADERS, timeout=15
-                )
-                data = resp.json()
-            except (ValueError, requests.RequestException) as exc:
-                last_status = f"request error: {exc}"
-                log.warning(
-                    "GoFile account creation attempt %d/%d failed: %s",
-                    attempt, max_attempts, last_status,
-                )
-            else:
-                if isinstance(data, dict) and data.get("status") == "ok":
-                    self._token = data["data"]["token"]
-                    log.info("GoFile guest account created (token reused for whole run)")
-                    return self._token
-                last_status = str(data.get("status", data)[:200] if data else data)
-                log.warning(
-                    "GoFile account creation attempt %d/%d failed: %s",
-                    attempt, max_attempts, last_status,
-                )
-            if attempt < max_attempts:
-                time.sleep(wait_seconds * attempt)
-        raise GoFileError(
-            f"GoFile account creation failed after {max_attempts} attempts "
-            f"(last: {last_status})"
-        )
-
-    def reset(self) -> None:
-        """Drop the current token so the next ensure_token() creates a new one."""
-        self._token = None
-
-    def _website_token(self) -> str:
-        window = int(time.time()) // TOKEN_WINDOW_SEC
-        seed = f"{USER_AGENT}::{LOCALE}::{self._token}::{window}::{TOKEN_SECRET}"
-        return hashlib.sha256(seed.encode()).hexdigest()
-
-    def _auth_headers(self) -> dict:
-        return {
-            **EXTENDED_HEADERS,
-            "Authorization": f"Bearer {self._token}",
-            "X-Website-Token": self._website_token(),
-            "X-BL": LOCALE,
-        }
-
-    # -- listing ------------------------------------------------------------
-
-    def list_files(self, url: str) -> list[tuple[str, str, str]]:
-        """Return (relpath, filename, download_link) for every file under a URL."""
-        cid = content_id_from_url(url)
-        results: list[tuple[str, str, str]] = []
-        self._walk(cid, [], results)
-        # Direct files at the content root get their own folder named after the id
-        normalized = [
-            (relpath if relpath else cid, name, link) for relpath, name, link in results
-        ]
-        return normalized
-
-    def _walk(self, cid: str, prefix: list[str], results: list[tuple[str, str, str]]) -> None:
-        query = "?cache=true&sortField=createTime&sortDirection=1"
-        if self._password_hash:
-            query += f"&password={self._password_hash}"
-        try:
-            resp = self.session.get(
-                f"{API}/contents/{cid}{query}", headers=self._auth_headers(), timeout=30
-            )
-        except requests.RequestException as exc:
-            raise GoFileError(f"content request for {cid} failed: {exc}") from exc
-
-        if resp.status_code == 404:
-            raise GoFileNotFound(f"content {cid} not found (dead link)")
-        if resp.status_code in (401, 403):
-            raise GoFileAuthError(f"content request denied ({resp.status_code}) for {cid}")
-        try:
-            data = resp.json()
-        except ValueError as exc:
-            raise GoFileError(f"non-JSON response for {cid}: {resp.text[:200]}") from exc
-
-        if data.get("status") == "notFound" or "notFound" in str(data.get("status", "")):
-            raise GoFileNotFound(f"content {cid} does not exist anymore (dead link)")
-        if data.get("status") != "ok":
-            raise GoFileError(f"content request failed for {cid}: {data.get('status')}")
-
-        d = data["data"]
-        if d.get("passwordStatus") not in (None, "passwordOk"):
-            raise GoFilePasswordRequired(
-                f"content {cid} is password protected (use --password)"
-            )
-
-        if d.get("type") == "folder":
-            here = prefix + [d["name"]]
-            for child in (d.get("children") or {}).values():
-                if child.get("type") == "folder":
-                    self._walk(child["id"], here, results)
-                else:
-                    results.append(("/".join(here), child["name"], child["link"]))
-        else:
-            results.append(("/".join(prefix), d["name"], d["link"]))
-
-    # -- downloading ----------------------------------------------------------
-
-    def download_file(self, link: str, dest: Path) -> int:
-        """Download one file; returns the file size. Skips existing files."""
-        dest.parent.mkdir(parents=True, exist_ok=True)
-        if dest.exists() and dest.stat().st_size > 0:
-            log.info("Skip existing: %s", dest)
-            return dest.stat().st_size
-
-        tmp = dest.with_name(dest.name + ".part")
-        headers = {
-            **EXTENDED_HEADERS,
-            "Cookie": f"accountToken={self._token}",
-            "Referer": link,
-            "Origin": link,
-        }
-        try:
-            with self.session.get(
-                link, headers=headers, stream=True, timeout=(10, 60), allow_redirects=True
-            ) as resp:
-                if resp.status_code == 404:
-                    raise GoFileNotFound(f"file {dest.name} not found (dead link)")
-                if resp.status_code != 200:
-                    raise GoFileError(f"HTTP {resp.status_code} while downloading {dest.name}")
-                total = int(resp.headers.get("Content-Length") or 0)
-                written = 0
-                with tmp.open("wb") as fh:
-                    for chunk in resp.iter_content(chunk_size=CHUNK_SIZE):
-                        fh.write(chunk)
-                        written += len(chunk)
-            if written == 0:
-                raise GoFileError(f"empty body for {dest.name}")
-            os.replace(tmp, dest)
-            return written
-        finally:
-            if tmp.exists() and not dest.exists():
-                try:
-                    tmp.unlink()
-                except OSError:
-                    pass

version-python/dlphelper/gwshop.py

@@ -1,353 +0,0 @@
-"""Games Workshop webshop catalog via Algolia.
-
-The warhammer.com shop is behind an AWS WAF challenge for non-browser
-clients, but its product search runs on Algolia and the app-id plus the
-public search-only key ship with every browser page. This module uses them
-to enumerate the whole product index and normalize it into the sorter's
-catalog schema (system/group/faction/subfaction/class/name/aliases).
-
-Credentials are read from (first hit wins):
-  1. env ALGOLIA_APP_ID + ALGOLIA_API_KEY
-  2. algolia.json next to the repo root (gitignored):
-       {"app_id": "...", "api_key": "..."}
-
-The key is the *search-only* public key (never an admin key). Treat any
-key found in algolia.json as still secret: never log or commit it.
-
-CLI:  python3 -m dlphelper.gwshop OUT.json [--index NAME] [--raw RAW.json]
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import logging
-import os
-import time
-from pathlib import Path
-from typing import Any
-
-import requests
-
-log = logging.getLogger(__name__)
-
-DEFAULT_INDEX = "prod-lazarus-product-en-ww"
-# The read host (-dsn) serves searches; credentials also work as query
-# parameters, which keeps every request a plain GET/POST with no headers
-# beyond content-type.
-_DSN_HOST = "https://{app_id}-dsn.algolia.net"
-
-_PAGE_SIZE = 1000  # Algolia max hitsPerPage per /query page
-_RETRY_DELAYS = (2, 5, 15)
-
-
-def load_credentials() -> tuple[str, str]:
-    """Resolve (app_id, api_key) from env or algolia.json."""
-    app_id = os.environ.get("ALGOLIA_APP_ID", "")
-    api_key = os.environ.get("ALGOLIA_API_KEY", "")
-    if app_id and api_key:
-        return app_id, api_key
-    cfg = Path(__file__).resolve().parent.parent / "algolia.json"
-    if cfg.is_file():
-        data = json.loads(cfg.read_text(encoding="utf-8"))
-        app_id = data.get("app_id", "") or app_id
-        api_key = data.get("api_key", "") or api_key
-    if not app_id or not api_key:
-        raise SystemExit(
-            "Algolia credentials missing: set ALGOLIA_APP_ID and "
-            "ALGOLIA_API_KEY, or create algolia.json with "
-            '{"app_id": "...", "api_key": "..."} (gitignored).'
-        )
-    return app_id, api_key
-
-
-class GwShopClient:
-    """Minimal Algolia search client for one application."""
-
-    def __init__(self, app_id: str, api_key: str) -> None:
-        self.base = _DSN_HOST.format(app_id=app_id)
-        self.params = {
-            "x-algolia-application-id": app_id,
-            "x-algolia-api-key": api_key,
-        }
-        self.session = requests.Session()
-
-    def _request(self, method: str, path: str, body: dict | None = None) -> Any:
-        url = f"{self.base}{path}"
-        last_err: Exception | None = None
-        for attempt, delay in enumerate((0,) + _RETRY_DELAYS, start=1):
-            if delay:
-                log.info("Algolia retry %d in %ds", attempt, delay)
-                time.sleep(delay)
-            try:
-                r = self.session.request(
-                    method,
-                    url,
-                    params=self.params,
-                    json=body,
-                    timeout=30,
-                )
-                if r.status_code == 200:
-                    return r.json()
-                last_err = RuntimeError(
-                    f"Algolia {method} {path} -> {r.status_code}: {r.text[:200]}"
-                )
-                if r.status_code in (401, 403):  # credentials: no point retrying
-                    break
-            except requests.RequestException as exc:
-                last_err = exc
-        raise RuntimeError(f"Algolia request failed: {last_err}") from last_err
-
-    def list_indices(self) -> list[dict]:
-        return self._request("GET", "/1/indexes").get("items", [])
-
-    # ---- full index retrieval -------------------------------------------
-    # The search-only key cannot use /browse (403) and normal pagination is
-    # capped at 1000 cumulative results. So we partition the index on the
-    # GameSystemsRoot hierarchical facet: each branch is fetched with an
-    # explicit filter, recursing deeper only when a branch exceeds one page.
-
-    _FACET = "GameSystemsRoot"
-
-    @staticmethod
-    def _escape(value: str) -> str:
-        return value.replace("\\", "\\\\").replace('"', '\\"')
-
-    def _filter_expr(self, path: list[tuple[int, str]]) -> str:
-        return " AND ".join(
-            f'{self._FACET}.lvl{lvl}:"{self._escape(value)}"'
-            for lvl, value in path
-        )
-
-    def _search_all(self, index: str, filters: str | None) -> tuple[list[dict], int]:
-        """All hits matching `filters` (must be <= pagination cap)."""
-        out: list[dict] = []
-        params = f"query=&hitsPerPage={_PAGE_SIZE}&page=0"
-        if filters:
-            params += f"&filters={requests.utils.quote(filters, safe='')}"
-        page = self._request(
-            "POST", f"/1/indexes/{index}/query", body={"params": params}
-        )
-        out.extend(page.get("hits", []))
-        return out, int(page.get("nbHits") or 0)
-
-    def _facet_counts(
-        self, index: str, level: int, filters: str | None
-    ) -> dict[str, int]:
-        params = f"query=&hitsPerPage=0&facets={self._FACET}.lvl{level}"
-        if filters:
-            params += f"&filters={requests.utils.quote(filters, safe='')}"
-        page = self._request(
-            "POST", f"/1/indexes/{index}/query", body={"params": params}
-        )
-        facets = page.get("facets") or {}
-        return facets.get(f"{self._FACET}.lvl{level}") or {}
-
-    def browse(self, index: str, hits_per_page: int = _PAGE_SIZE) -> list[dict]:
-        """Every record of the index, via facet-partitioned search."""
-        hits, total = self._search_all(index, None)
-        if total <= hits_per_page:
-            return hits
-        log.info("gwshop %s: %d hits, partitioning on %s", index, total, self._FACET)
-        out = self._partition(index, [], total)
-        covered = len({h.get("objectID") for h in out})
-        if covered < total:
-            log.warning(
-                "gwshop %s: %d of %d index records left out of the dump "
-                "(no fetchable %s branch path)",
-                index, total - covered, total, self._FACET,
-            )
-        return out
-
-    def _partition(
-        self, index: str, path: list[tuple[int, str]], total: int
-    ) -> list[dict]:
-        level = len(path)
-        filters = self._filter_expr(path) if path else None
-        counts = self._facet_counts(index, level, filters)
-        out: list[dict] = []
-        covered: set[Any] = set()
-        for value, count in sorted(counts.items()):
-            sub_path = path + [(level, value)]
-            sub_filters = self._filter_expr(sub_path)
-            if count <= _PAGE_SIZE:
-                hits, nb = self._search_all(index, sub_filters)
-                out.extend(hits)
-                covered.update(h.get("objectID") for h in hits)
-                log.info(
-                    "gwshop %s: %s -> %d hits", index, sub_filters[:80], nb
-                )
-            else:
-                sub = self._partition(index, sub_path, count)
-                out.extend(sub)
-                covered.update(h.get("objectID") for h in sub)
-        # nbHits summed across branches double-counts products listed in
-        # several branches, so coverage is counted per unique objectID.
-        residual = total - len(covered)
-        if residual > 0:
-            log.warning(
-                "gwshop %s: %d hits in branch carry no %s.lvl%d "
-                "(shallow facet path, left out of the dump)",
-                index, residual, self._FACET, level,
-            )
-        return out
-
-
-# ---------------------------------------------------------------------------
-# normalization
-# ---------------------------------------------------------------------------
-
-def _clean(node: str) -> str:
-    return node.strip()
-
-
-def hierarchy_paths(hit: dict) -> list[list[str]]:
-    """Every full path of the GameSystemsRoot hierarchical facet.
-
-    A hit may belong to several branches (e.g. an army box also being a
-    "Gaming Rules" entry); each branch is returned as its own path
-    [system, group, faction, subfaction_or_class, ...].
-    """
-    root = hit.get("GameSystemsRoot") or {}
-    paths: list[list[str]] = []
-    for lvl_key in sorted(root, key=lambda k: int(k[3:])):
-        for value in root.get(lvl_key) or []:
-            paths.append([_clean(p) for p in value.split(" > ")])
-    return paths
-
-
-def normalize(hit: dict) -> dict:
-    """One Algolia hit -> one or more catalog entries (usually one)."""
-    name = _clean(hit.get("name") or hit.get("objectID") or "")
-    entry: dict[str, Any] = {
-        "name": name,
-        "aliases": [],
-        "sku": hit.get("sku"),
-        "slug": hit.get("slug"),
-        "productType": hit.get("productType"),
-        "source": "gwshop",
-        "gw_paths": hierarchy_paths(hit),
-        "collections": hit.get("_collections") or [],
-    }
-    return entry
-
-
-def build_catalog(hits: list[dict]) -> dict:
-    """Index dump -> catalog document in the sorter's schema shape.
-
-    The hierarchy layer (system/group/faction/subfaction) is expanded from
-    gw_paths; name/aliases stay per product so the fuzzy matcher works on
-    the product name, exactly like data/library_catalog.json entries.
-    """
-    entries: list[dict] = []
-    seen: set[tuple] = set()
-    for hit in hits:
-        base = normalize(hit)
-        paths = base["gw_paths"]
-        if not paths:
-            key = (hit.get("objectID"), ())
-            if key in seen:
-                continue
-            seen.add(key)
-            entries.append(
-                {
-                    "system": None,
-                    "group": None,
-                    "faction": None,
-                    "subfaction": None,
-                    "class": None,
-                    "name": base["name"],
-                    "aliases": [],
-                    "sku": base["sku"],
-                    "slug": base["slug"],
-                    "productType": base["productType"],
-                    "source": "gwshop",
-                    "collections": base["collections"],
-                }
-            )
-            continue
-        # primary branch = longest path (most specific), e.g.
-        # [40k, Space Marines, Raven Guard, Walker]
-        primary = max(paths, key=len)
-        system = primary[0] if len(primary) > 0 else None
-        group = primary[1] if len(primary) > 1 else None
-        faction = primary[2] if len(primary) > 2 else None
-        # The LAST level is the class/category layer (GW uses shop
-        # categories there: Battleline, Walker, Vehicle, "Rules &
-        # Accessories", product lines like "Cadian"). Deeper sources
-        # (miniset style) may carry a real subfaction before it.
-        if len(primary) >= 5:
-            subfaction = primary[3]
-            klass = primary[4]
-        elif len(primary) == 4:
-            subfaction = None
-            klass = primary[3]
-        else:
-            subfaction = None
-            klass = None
-        key = (hit.get("objectID"), tuple(primary))
-        if key in seen:
-            continue
-        seen.add(key)
-        entries.append(
-            {
-                "system": system,
-                "group": group,
-                "faction": faction,
-                "subfaction": subfaction,
-                "class": klass,
-                "name": base["name"],
-                "aliases": [],
-                "sku": base["sku"],
-                "slug": base["slug"],
-                "productType": base["productType"],
-                "source": "gwshop",
-                "collections": base["collections"],
-            }
-        )
-    return {
-        "comment": (
-            "Generated by dlphelper.gwshop from the Games Workshop webshop "
-            "(Algolia index, deduplicated by product). Source-agnostic "
-            "catalog schema: system/group/faction/subfaction/class/name/aliases."
-        ),
-        "source": "gwshop",
-        "generated_entries": len(entries),
-        "entries": entries,
-    }
-
-
-def main() -> None:
-    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-    parser = argparse.ArgumentParser(
-        description="Dump the GW webshop catalog via Algolia."
-    )
-    parser.add_argument("out", help="output catalog JSON path")
-    parser.add_argument("--index", default=DEFAULT_INDEX)
-    parser.add_argument("--raw", help="also write the raw Algolia hits here")
-    args = parser.parse_args()
-
-    app_id, api_key = load_credentials()
-    client = GwShopClient(app_id, api_key)
-    hits = client.browse(args.index)
-    log.info("%d raw hits from %s", len(hits), args.index)
-
-    if args.raw:
-        Path(args.raw).write_text(
-            json.dumps(hits, ensure_ascii=False, indent=1), encoding="utf-8"
-        )
-        log.info("raw hits written to %s", args.raw)
-
-    catalog = build_catalog(hits)
-    Path(args.out).write_text(
-        json.dumps(catalog, ensure_ascii=False, indent=1), encoding="utf-8"
-    )
-    log.info(
-        "%d catalog entries written to %s",
-        catalog["generated_entries"],
-        args.out,
-    )
-
-
-if __name__ == "__main__":
-    main()

version-python/dlphelper/sorter.py

@@ -1,899 +0,0 @@
-"""Sorter: place collection files into the library via plan/apply.
-
-See docs/SORTING.md for the structure and the pipeline this implements:
-inspect (name + archive listing) -> classify (keyword tables) -> fuzzy match
-against the library catalog -> plan -> apply (hardlink, fallback copy).
-
-The sorter never deletes or modifies source files. Applying is append-only:
-targets that already exist are skipped, actions are audited in
-sorter_applied.jsonl.
-"""
-
-from __future__ import annotations
-
-import difflib
-import json
-import logging
-import os
-import shutil
-import subprocess
-import zipfile
-from dataclasses import dataclass, field
-from functools import lru_cache
-from pathlib import Path
-
-from dlphelper.catalogue import _clean_name, _match_class, _match_faction
-from dlphelper.warhammer_data import GROUP_OF_FACTION
-
-log = logging.getLogger(__name__)
-
-ARCHIVE_SUFFIXES = {".zip", ".7z", ".rar", ".tar", ".gz"}
-UNSORTED = "_unsorted"
-UNCLASSIFIED_CLASS = "Unclassified"
-# A file whose system the keyword tables could not identify only gets a
-# catalog unit when the fuzzy hit is strong - otherwise it would wander
-# into other games' sections (Blood Bowl, ...) on a guess.
-CROSS_SYSTEM_MIN = 0.75
-
-# Game systems whose sections the sorter must never place into. The
-# collection is warhammer-only: everything else (Middle-earth proxies,
-# ...) stays in _unsorted even when a name matches the catalog. The
-# catalog snapshot itself keeps all systems; the filter lives here.
-DROPPED_SYSTEMS = frozenset({"Middle-Earth"})
-
-# Weak acceptance tier: when nothing reached the normal threshold, a hit
-# at least this strong is accepted but flagged low confidence ("[w]" in
-# the plan, confidence field in the audit, listed in
-# sorter_low_confidence.txt).
-WEAK_MIN = 0.50
-
-# A hit whose name shares no word with the entry needs this much more
-# score to be trusted (char-ratio flukes like "aeldari rangers" -> "Dark
-# Reapers" sit at 0.60-0.67; real unit names usually share a token).
-TOKENLESS_BONUS = 0.08
-
-# Run-artifact files that must never be sorted into a library
-RUN_ARTIFACTS = {
-    "export.json", "export.html", "state.json", "manifest.jsonl",
-    "failed_urls.txt", "catalogue.md", "catalogue.jsonl",
-    "sorter_applied.jsonl", "catalogue_moves.jsonl", "urls.txt",
-    "urls_0807.txt", "urls_gofile.txt", "failed_0807.txt",
-    "extract_manifest.jsonl",
-}
-
-
-@dataclass
-class SortItem:
-    """One file to sort, plus everything learned about it."""
-
-    path: Path
-    size: int
-    name: str            # original file name
-    clean: str           # cleaned lower-case name
-    inner_names: list[str] = field(default_factory=list)  # archive listing
-    system: str = "Unknown"
-    group: str = UNSORTED
-    faction: str = UNSORTED
-    subfaction: str | None = None
-    klass: str | None = None
-    unit: str | None = None          # canonical catalog name, if matched
-    match_score: float = 0.0
-    match_kind: str = "none"         # catalog | weak | implicit | none | invalid
-    confidence: str = "normal"       # normal | low (match_kind == "weak")
-    action: str = "link"             # link | copy
-    target: Path | None = None
-
-
-# ---------------------------------------------------------------------------
-# inspection
-# ---------------------------------------------------------------------------
-
-def list_zip(path: Path, limit: int = 40) -> list[str]:
-    """Inner names of a .zip without extracting."""
-    try:
-        with zipfile.ZipFile(path) as zf:
-            return zf.namelist()[:limit]
-    except (zipfile.BadZipFile, OSError, RuntimeError) as exc:
-        log.debug("zip listing failed for %s: %s", path.name, exc)
-        return []
-
-
-def list_7z(path: Path, limit: int = 40) -> list[str]:
-    """Inner names of a .7z via the installed 7z binary, without extracting."""
-    if not shutil.which("7z"):
-        return []
-    try:
-        proc = subprocess.run(
-            ["7z", "l", "-ba", "-slt", str(path)],
-            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
-            encoding="utf-8", errors="replace", timeout=120,
-        )
-    except (OSError, subprocess.TimeoutExpired) as exc:
-        log.debug("7z listing failed for %s: %s", path.name, exc)
-        return []
-    names: list[str] = []
-    for block in proc.stdout.split("Path = ")[1:]:
-        name = block.splitlines()[0].strip() if block.splitlines() else ""
-        # -slt lists directories too; skip entries ending in /
-        if name and not name.endswith("/"):
-            names.append(name)
-        if len(names) >= limit:
-            break
-    return names
-
-
-def inspect(path: Path) -> SortItem:
-    size = path.stat().st_size if path.exists() else 0
-    item = SortItem(path=path, size=size, name=path.name, clean=_clean_name(path.name))
-    if not path.is_file():
-        return item
-    if path.suffix.lower() in ARCHIVE_SUFFIXES:
-        # An archive that is not an archive is almost always a saved HTML
-        # error/login page that got downloaded instead of the file.
-        if path.suffix.lower() == ".zip":
-            item.inner_names = list_zip(path)
-        else:
-            item.inner_names = list_7z(path)
-        if item.inner_names == [] and path.stat().st_size > 0:
-            head = b""
-            with path.open("rb") as fh:
-                head = fh.read(512)
-            if b"<!doctype html" in head.lower() or b"<html" in head.lower():
-                log.warning("%s is an HTML page, not an archive (bad download)", path.name)
-                item.match_kind = "invalid"
-    return item
-
-
-# ---------------------------------------------------------------------------
-# classification & matching
-# ---------------------------------------------------------------------------
-
-def classify_item(item: SortItem) -> None:
-    """Keyword classification from the file name, then refined by inner names."""
-    hit = _match_faction(item.clean)
-    if hit:
-        item.system, item.faction = hit
-    item.klass = _match_class(item.clean) or (
-        _match_class(_clean_inner(" ".join(item.inner_names)))
-        if item.inner_names else None
-    )
-
-    if item.faction == UNSORTED and item.inner_names:
-        # hash-named archives: vote on the inner names' classification
-        votes: dict[tuple[str, str], int] = {}
-        for inner in item.inner_names:
-            ihit = _match_faction(_clean_inner(inner))
-            if ihit:
-                votes[ihit] = votes.get(ihit, 0) + 1
-        if votes:
-            best = max(votes.items(), key=lambda kv: kv[1])[0]
-            item.system, item.faction = best
-            if item.klass is None:
-                inner_classes = [
-                    _match_class(_clean_name(inner)) for inner in item.inner_names
-                ]
-                cvotes: dict[str, int] = {}
-                for c in inner_classes:
-                    if c:
-                        cvotes[c] = cvotes.get(c, 0) + 1
-                if cvotes:
-                    item.klass = max(cvotes.items(), key=lambda kv: kv[1])[0]
-
-    item.group = GROUP_OF_FACTION.get(item.faction, UNSORTED)
-
-
-# faction group layer (GW naming, shared with warhammer_data)
-
-
-def load_catalog(path: Path,
-                 dropped: frozenset[str] | None = DROPPED_SYSTEMS) -> list[dict]:
-    """Load catalog entries, dropping non-collected game systems by default
-    (pass dropped=frozenset() to keep everything)."""
-    data = json.loads(path.read_text(encoding="utf-8"))
-    entries = data["entries"] if isinstance(data, dict) else data
-    if dropped:
-        entries = [e for e in entries if e.get("system") not in dropped]
-    for e in entries:
-        e["_clean"] = _clean_name(e["name"])
-        e["_clean_aliases"] = [_clean_name(a) for a in e.get("aliases", [])]
-    return entries
-
-
-def _best_ratio(
-    clean: str, candidates: list[str], allow_single: bool = True
-) -> float:
-    """Best sequence ratio, pruned with cheap upper-bound checks.
-
-    lru_cached because the same (name, entry) pair is scored over and
-    over across thousands of files.
-    """
-    return _best_ratio_cached(clean, tuple(candidates), allow_single)
-
-
-@lru_cache(maxsize=None)
-def _best_ratio_cached(
-    clean: str, candidates: tuple[str, ...], allow_single: bool = True
-) -> float:
-    best = 0.0
-    toks = set(_stem_tokens(clean))
-    best = _containment_boost(best, toks, candidates, allow_single)
-    for c in candidates:
-        sm = difflib.SequenceMatcher(None, clean, c)
-        if sm.real_quick_ratio() < best or sm.quick_ratio() < best:
-            continue
-        r = sm.ratio()
-        if r > best:
-            best = r
-            if best == 1.0:
-                break
-    return best
-
-
-def _stem_tokens(clean: str) -> list[str]:
-    """Tokens with a crude plural 's' stripped, for containment checks."""
-    return [t[:-1] if len(t) > 3 and t.endswith("s") else t
-            for t in clean.split()]
-
-
-def _containment_boost(
-    best: float, toks: set[str], candidates: tuple[str, ...],
-    allow_single: bool = True,
-) -> float:
-    """Token-containment bonus: a name whose tokens all appear in the
-    other (or vice versa) matches strongly even when string ratio is low
-    ("telemon" vs "Telemon Heavy Dreadnought"). Guarded: needs >= 2
-    tokens, or a single rare one (>= 7 chars), and the smaller side must
-    not consist of generic army words ("chaos lord" is contained in half
-    the catalog) so "head" or "guard" never trigger it. Inner-name votes
-    pass allow_single=False: one short inner file name ("lieutenant")
-    must not claim a whole pack."""
-    if not toks:
-        return best
-    for c in candidates:
-        ctoks = set(_stem_tokens(c))
-        if not ctoks:
-            continue
-        small = toks if toks <= ctoks else (ctoks if ctoks <= toks else None)
-        if small is None or any(t in GENERIC_TOKENS for t in small):
-            continue
-        if len(small) >= 2 or (allow_single and len(next(iter(small))) >= 7):
-            if best < 0.9:
-                best = 0.9
-            break
-    return best
-
-
-# words too generic for token-containment matching: entries built from
-# them alone would claim any file whose name happens to mention them
-GENERIC_TOKENS = frozenset({
-    "chaos", "lord", "space", "marine", "guard", "knight", "dark",
-    "death", "blood", "war", "warrior", "daemon", "hero", "god",
-    "free", "black", "white", "grey", "green", "gold", "iron",
-    "steel", "storm", "soul", "shadow", "bone", "ghost", "king",
-    "clan", "tribe", "legion", "regiment", "squad", "warhound",
-    "orc", "ogre", "elf", "dwarf", "giant", "company", "chapter",
-    "weapon", "gun", "gunner", "turret", "carriage", "upgrade",
-})  # noqa: C405
-
-
-# directory names that carry no unit information inside sculptor packs;
-# never used as match candidates
-_JUNK_DIRS = frozenset({
-    "files", "file", "images", "image", "img", "pictures", "pics",
-    "preview", "previews", "render", "renders", "photos", "screenshot",
-    "screenshots", "torso", "torsos", "head", "heads", "arm", "arms",
-    "leg", "legs", "back", "front", "left", "right", "top", "bottom",
-    "side", "weapon", "weapons", "accessories", "accessory", "bits",
-    "bitz", "parts", "part", "base", "bases", "support", "supports",
-    "presupported", "unsupported", "stl", "stls", "obj", "models",
-    "model", "output", "exports", "release", "releases", "license",
-    "readme", "sources", "docs", "textures", "new", "old", "final",
-    "fixed", "print", "prints", "plates", "plate", "assembly", "test",
-    "characters", "character",
-})  # noqa: C405
-
-
-def _clean_inner(name: str) -> str:
-    """Clean an archive-internal (relative) path: dir names join the vote
-    ("2E Catachany fellaz/Catachan/Jungle Fighters/x.stl" -> "2e catachany
-    fellaz catachan jungle fighters x") - the file stem alone loses the
-    archive's own folder structure."""
-    return _clean_name(name.replace("/", " ").replace("\\", " "))
-
-
-def _is_junk_dir(name: str) -> bool:
-    """True for directory names that must never be match candidates."""
-    clean = _clean_name(name)
-    if not clean or clean in _JUNK_DIRS or clean.isdigit() or len(clean) <= 2:
-        return True
-    return " " not in clean and clean in GENERIC_TOKENS
-
-
-def _token_contained(a_clean: str, b_clean: str) -> bool:
-    """True when one name's (stemmed) tokens all appear in the other's.
-
-    Guarded like _containment_boost: the smaller side must not consist of
-    generic army words alone. Used to gate cross-system adoption, where
-    plain char-ratio lies ("aeldari rangers" vs "dwarf rangers").
-    """
-    ta = set(_stem_tokens(a_clean))
-    tb = set(_stem_tokens(b_clean))
-    if not ta or not tb:
-        return False
-    small = ta if ta <= tb else (tb if tb <= ta else None)
-    if small is None:
-        return False
-    if len(small) == 1:
-        # a lone generic word ("guard", "orc") must not carry a match
-        return next(iter(small)) not in GENERIC_TOKENS
-    return True
-
-
-def _contained_somewhere(item: SortItem, entry: dict) -> bool:
-    """Token containment between the entry (or an alias) and the outer
-    name or any inner (relative-path) name."""
-    cands = [entry["_clean"], *entry["_clean_aliases"]]
-    probes = [item.clean, *[_clean_inner(n) for n in item.inner_names[:10]]]
-    return any(_token_contained(p, c) for p in probes for c in cands)
-
-
-def _shares_rare_token(a_clean: str, b_clean: str) -> bool:
-    """True when two cleaned names share at least one non-generic token
-    (>= 3 chars). Gates the weak tier: a sub-threshold hit with zero
-    shared words is a char-ratio fluke ("LICENSE.txt" -> "Veletaris Storm
-    Section"), not evidence.
-    """
-    ta = set(_stem_tokens(a_clean))
-    tb = set(_stem_tokens(b_clean))
-    shared = (ta & tb) - GENERIC_TOKENS
-    return any(len(t) >= 3 for t in shared)
-
-
-def _apply_entry(
-    item: SortItem, entry: dict, score: float, kind: str, confidence: str
-) -> None:
-    """Adopt a catalog entry into an item (dir re-homing and weak tier).
-
-    Unlike the pack/file path in match_catalog, the entry's class layer is
-    adopted unconditionally: a precise unit match ("Warp Spiders" -> Fly)
-    must beat a generic keyword class ("warlock" mentioned somewhere ->
-    Hero).
-    """
-    item.unit = entry["name"]
-    if entry.get("group"):
-        item.group = entry["group"]
-    else:
-        item.group = GROUP_OF_FACTION.get(item.faction, UNSORTED)
-    if entry.get("subfaction"):
-        item.subfaction = entry["subfaction"]
-    if entry.get("class"):
-        item.klass = entry["class"]
-    item.match_score = score
-    item.match_kind = kind
-    item.confidence = confidence
-    entry_faction = entry.get("faction")
-    if entry_faction and entry_faction not in ("Unit Type", "Heroes",
-                                               "Warriors", "Character"):
-        item.system = entry["system"]
-        item.faction = entry_faction
-
-
-def _accept_name_entry(
-    clean: str, entry: dict, score: float, threshold: float
-):
-    """(score, entry, weak) when `entry` may adopt a name scoring `score`.
-
-    Names sharing a rare token are trusted at the normal threshold (and
-    down to WEAK_MIN); tokenless hits need +TOKENLESS_BONUS and are never
-    weak. None when rejected.
-    """
-    shared = _shares_rare_token(clean, entry["_clean"]) or any(
-        _shares_rare_token(clean, a) for a in entry["_clean_aliases"])
-    if shared:
-        if score >= threshold:
-            return (score, entry, False)
-        if score >= WEAK_MIN:
-            return (score, entry, True)
-        return None
-    if score >= threshold + TOKENLESS_BONUS:
-        return (score, entry, False)
-    return None
-
-
-def _pool_best(
-    clean: str, pool: list[dict], threshold: float
-):
-    """Best ACCEPTED entry over a pool - normal beats weak, then higher
-    score; deeper-side ties are irrelevant here (single pool)."""
-    best_n = best_w = None
-    for entry in pool:
-        score = _best_ratio(clean, [entry["_clean"], *entry["_clean_aliases"]])
-        a = _accept_name_entry(clean, entry, score, threshold)
-        if not a:
-            continue
-        if a[2]:
-            if best_w is None or a[0] > best_w[0]:
-                best_w = a
-        elif best_n is None or a[0] > best_n[0]:
-            best_n = a
-    return best_n or best_w
-
-
-def _match_name(
-    clean: str,
-    catalog: list[dict],
-    system: str,
-    faction: str,
-    threshold: float,
-    cache: dict,
-):
-    """Fuzzy-match a bare name (a directory or pack name) against the catalog.
-
-    Same-system constraint; a different faction needs a strong hit
-    (CROSS_SYSTEM_MIN). Returns (score, entry, weak) or None - weak marks
-    a same-faction hit accepted only via the WEAK_MIN tier.
-    """
-    key = (clean, system, faction)
-    hit = cache.get(key, 0)
-    if hit != 0:
-        return hit or None
-    shim = SortItem(path=Path(), size=0, name="", clean=clean)
-    result = None
-    if system == "Unknown":
-        full = _best_in_pool(shim, catalog, use_inner=False)
-        if full and full[0] >= CROSS_SYSTEM_MIN and (
-                _token_contained(clean, full[1]["_clean"])
-                or any(_token_contained(clean, a)
-                       for a in full[1]["_clean_aliases"])):
-            result = (full[0], full[1], False)
-    else:
-        pool = [e for e in catalog if e.get("system") == system]
-        if faction != UNSORTED:
-            same = _pool_best(
-                clean, [e for e in pool if e.get("faction") == faction],
-                threshold)
-            other = _pool_best(
-                clean, [e for e in pool if e.get("faction") != faction],
-                threshold)
-            if other and (other[2] or other[0] < CROSS_SYSTEM_MIN):
-                # a different faction must win strong, never weak
-                other = None
-            cands = [c for c in (same, other) if c]
-            if cands:
-                normals = [c for c in cands if not c[2]]
-                result = max(normals or cands, key=lambda c: c[0])
-        else:
-            result = _pool_best(clean, pool, threshold)
-    cache[key] = result or 0
-    return result
-
-
-def _best_dir_match(
-    item: SortItem,
-    dirs: list[tuple[Path, str]],
-    catalog: list[dict],
-    threshold: float,
-    cache: dict,
-):
-    """Best matching directory in a deepest-first [(path, name)] chain.
-
-    Returns (score, entry, weak, path) of the winner - higher score wins,
-    equal keeps the deeper candidate - or None.
-    """
-    best_n = best_w = None
-    for real_dir, dir_name in dirs:
-        if _is_junk_dir(dir_name):
-            continue
-        m = _match_name(_clean_name(dir_name), catalog, item.system,
-                        item.faction, threshold, cache)
-        if not m:
-            continue
-        score, entry, weak = m
-        cand = (score, entry, weak, real_dir)
-        if weak:
-            if best_w is None or score > best_w[0]:
-                best_w = cand
-        elif best_n is None or score > best_n[0]:
-            best_n = cand
-    return best_n or best_w
-
-
-def _rehome_from_dirs(
-    item: SortItem,
-    dirs: list[tuple[Path, str]],
-    catalog: list[dict],
-    threshold: float,
-    cache: dict,
-    prefer_dirs_on_tie: bool = False,
-):
-    """Let matching directory names re-home an item ("deepest match wins").
-
-    Returns the matched directory if the item was re-homed, else None. A
-    dir hit takes over when it beats the item's current resolution - a
-    confident pack/file match is only replaced by a strictly better (or
-    equal, coming from below the pack level) hit; implicit items yield to
-    any accepted dir match.
-    """
-    if item.match_kind == "invalid":
-        return None
-    own = item.match_kind in ("catalog", "weak")
-    own_score = item.match_score if own else 0.0
-    m = _best_dir_match(item, dirs, catalog, threshold, cache)
-    if not m:
-        return None
-    score, entry, weak, real_dir = m
-    if own and score < own_score:
-        return None
-    if own and score == own_score:
-        # exact tie: the deeper side wins, which is the dir only in
-        # provenance mode (pack name sits above the inner dirs); in manual
-        # trees the file name itself is the deepest side - keep it
-        if not (prefer_dirs_on_tie and not weak):
-            return None
-    kind = "weak" if weak else "catalog"
-    _apply_entry(item, entry, score, kind, "low" if weak else "normal")
-    return real_dir
-
-
-def _best_in_pool(
-    item: SortItem, pool: list[dict], use_inner: bool = True
-) -> tuple[float, dict] | None:
-    """Best fuzzy score of `item` against all entries in `pool`.
-
-    Inner archive names are only consulted on request; a confident outer
-    (file) name must win over votes from random inner STL filenames.
-    """
-    best: tuple[float, dict] | None = None
-    for entry in pool:
-        cands = [entry["_clean"], *entry["_clean_aliases"]]
-        score = _best_ratio(item.clean, cands)
-        if use_inner and item.inner_names:
-            inner_clean = [_clean_inner(n) for n in item.inner_names[:10]]
-            for c in inner_clean:
-                # no single-token containment boost for inner votes
-                score = max(score, _best_ratio(c, cands, allow_single=False) * 0.95)
-        if best is None or score > best[0]:
-            best = (score, entry)
-    return best
-
-
-def match_catalog(item: SortItem, catalog: list[dict], threshold: float) -> None:
-    """Fuzzy-match against catalog entries in the same faction (or all).
-
-    The matched entry is hierarchy-authoritative: group, subfaction and
-    class come from the catalog (GW shop categories), and when the keyword
-    classification had no faction the entry's system/faction is adopted too.
-    """
-    if item.faction == UNSORTED and not item.inner_names and not item.clean:
-        return
-    if item.match_kind == "invalid":
-        return  # HTML-page masquerading as an archive; never fuzzy-match it
-    system_known = item.system != "Unknown"
-    same_faction = [e for e in catalog if e.get("faction") == item.faction]
-    if system_known:
-        # a fuzzy match must never jump into another game's section
-        same_faction = [e for e in same_faction if e.get("system") == item.system]
-    if same_faction:
-        best = _best_in_pool(item, same_faction, use_inner=False)
-        if best is None or best[0] < threshold:
-            # Outer name not confident: let the archive's inner file names
-            # vote (the only signal for hash-named gofile archives).
-            best = _best_in_pool(item, same_faction, use_inner=True)
-        if best is None or best[0] < threshold:
-            # The keyword tables can lag behind GW's current categorization
-            # (e.g. units moved between factions); retry within the same
-            # system before giving up - never across systems. Keep the
-            # higher of the two scores (a retry must not downgrade a
-            # weak-eligible same-faction hit).
-            sys_pool = [e for e in catalog if e.get("system") == item.system]
-            sys_best = _best_in_pool(item, sys_pool, use_inner=False)
-            if sys_best is not None and (best is None or sys_best[0] > best[0]):
-                best = sys_best
-            if best is None or best[0] < threshold:
-                sys_best = _best_in_pool(item, sys_pool, use_inner=True)
-                if sys_best is not None and (best is None or sys_best[0] > best[0]):
-                    best = sys_best
-            # a retry hit from a *different* faction is only trusted when
-            # it is strong AND anchored by token containment; weak
-            # inner-name votes must not re-home the file
-            if best is not None and best[1].get("faction") != item.faction \
-                    and (best[0] < CROSS_SYSTEM_MIN
-                         or not _contained_somewhere(item, best[1])):
-                best = None
-    elif system_known:
-        sys_pool = [e for e in catalog if e.get("system") == item.system]
-        best = _best_in_pool(item, sys_pool, use_inner=False)
-        if best is None or best[0] < threshold:
-            best = _best_in_pool(item, sys_pool, use_inner=True)
-    else:
-        # Neither system nor faction known: whole catalog, but only a
-        # strong hit (CROSS_SYSTEM_MIN) with token containment may adopt
-        # another game's section; weaker or unanchored guesses stay in
-        # _unsorted.
-        best = _best_in_pool(item, catalog, use_inner=False)
-        if best is None or best[0] < threshold:
-            best = _best_in_pool(item, catalog, use_inner=True)
-        if best is not None and best[0] < CROSS_SYSTEM_MIN:
-            best = None
-        elif best is not None and not _contained_somewhere(item, best[1]):
-            best = None
-    if best and best[0] >= threshold:
-        entry = best[1]
-        item.unit = entry["name"]
-        if entry.get("group"):
-            item.group = entry["group"]
-        elif item.faction == UNSORTED:
-            item.group = GROUP_OF_FACTION.get(entry["faction"], UNSORTED)
-        if entry.get("subfaction"):
-            item.subfaction = entry["subfaction"]
-        # GW's class layer (shop category) beats the keyword guess, but do
-        # not *narrow* it: "Battleline" is GW's bucket for core infantry,
-        # a keyword-derived Hero/Elite is more specific for a proxy file.
-        catalog_klass = entry.get("class")
-        if catalog_klass:
-            # a matched catalog unit's class layer is authoritative for
-            # the whole subtree - a keyword-voted class ("warlock" seen
-            # somewhere -> Hero) must not survive next to a precise unit
-            item.klass = catalog_klass
-        item.match_score = best[0]
-        item.match_kind = "catalog"
-        # The catalog entry's faction is authoritative - but only adopt it
-        # when it is a real army faction. Generic GW branches like
-        # "Unit Type" or factionless entries would degrade a confident
-        # keyword faction to a junk folder.
-        entry_faction = entry.get("faction")
-        generic_factions = {"Unit Type", "Heroes", "Warriors", "Character"}
-        if item.faction == UNSORTED:
-            if entry_faction and entry_faction not in generic_factions:
-                item.system = entry["system"]
-                item.faction = entry_faction
-            if entry.get("group") and entry["group"] not in ("Unit Type",):
-                item.group = entry["group"]
-            else:
-                item.group = GROUP_OF_FACTION.get(item.faction, UNSORTED)
-        elif entry_faction and entry_faction not in generic_factions and entry_faction != item.faction:
-            # GW moved this unit to another faction (or the keyword tables
-            # are stale): trust the catalog.
-            item.system = entry["system"]
-            item.faction = entry_faction
-            if entry.get("group") and entry["group"] != "Unit Type":
-                item.group = entry["group"]
-            else:
-                item.group = GROUP_OF_FACTION.get(entry_faction, UNSORTED)
-        elif entry.get("group") and entry["group"] not in ("Unit Type",):
-            item.group = entry["group"]
-    elif best and best[0] >= WEAK_MIN and (
-            _shares_rare_token(item.clean, best[1]["_clean"])
-            or any(_shares_rare_token(_clean_inner(n), best[1]["_clean"])
-                   for n in item.inner_names[:10])):
-        # Weak tier: a same-system hit below the threshold with at least
-        # one shared word, accepted only when nothing stronger exists.
-        # Flagged low confidence.
-        _apply_entry(item, best[1], best[0], "weak", "low")
-    else:
-        if item.faction == UNSORTED:
-            item.unit = item.name  # implicit entry: own (original) name
-            item.match_score = best[0] if best else 0.0
-            item.match_kind = "implicit"
-        else:
-            # Faction was known but nothing fuzzy-matched: keep the keyword
-            # faction and use the file's own name as the unit, but adopt the
-            # catalog's class layer when it adds information.
-            item.unit = item.name
-            item.match_score = best[0] if best else 0.0
-            item.match_kind = "implicit"
-
-
-def build_target(item: SortItem, library: Path, inner_rel: Path | None = None) -> Path:
-    parts: list[str] = []
-    if item.system == "Unknown":
-        parts = [UNSORTED]
-        if inner_rel is not None and item.unit:
-            # extracted archives keep their unit folder even when unplaced
-            parts.append(_display_name(item.unit))
-    else:
-        parts = [item.system]
-        if item.faction != UNSORTED:
-            if item.group:
-                parts.append(item.group)
-            parts.append(item.faction)
-            if item.subfaction:
-                parts.append(item.subfaction)
-            parts.append(item.klass or UNCLASSIFIED_CLASS)
-            if item.unit:
-                parts.append(_display_name(item.unit))
-        else:
-            parts.append(UNSORTED)
-    target_dir = Path(library).joinpath(*parts)
-    if inner_rel is not None:
-        target_dir = target_dir.joinpath(*inner_rel.parts)
-    return target_dir / item.name
-
-
-def _display_name(unit: str) -> str:
-    """Implicit entries carry the original filename; strip the extension."""
-    p = Path(unit)
-    return p.stem if p.suffix else unit
-
-
-# ---------------------------------------------------------------------------
-# plan / apply
-# ---------------------------------------------------------------------------
-
-def _drop_dropped_system(item: SortItem) -> None:
-    """Dropped systems (see DROPPED_SYSTEMS) count as unclassified: the
-    collection does not cover them, so their names must not place files
-    into those sections. Called after keyword classification, before
-    matching - the item can still be re-homed by other signals."""
-    if item.system in DROPPED_SYSTEMS:
-        item.system = "Unknown"
-        item.group = item.faction = UNSORTED
-        item.subfaction = None
-
-
-def sort_plan(
-    files: list[Path],
-    library: Path,
-    catalog: list[dict],
-    threshold: float = 0.62,
-    provenance: dict[str, str] | None = None,
-    root: Path | None = None,
-) -> list[SortItem]:
-    """Build sort items for `files`.
-
-    `provenance` (with `root`) activates extracted-archive mode: it maps
-    extraction directories (relative to `root`) to the archive they came
-    from. All files inside such a directory are classified and matched
-    once under the archive's name and then placed together, keeping the
-    archive's inner structure - a loose "Backpack.stl" alone is noise,
-    under "The Twin Lance/" it belongs to its unit.
-    """
-    prov = provenance or {}
-    unit_cache: dict[str, dict] = {}
-    dir_cache: dict = {}
-    items = []
-    for f in sorted(files):
-        if not f.is_file():
-            continue
-        if (f.name.endswith(".part") or f.name.startswith(".")
-                or f.name in RUN_ARTIFACTS):
-            continue
-        unit_name = dest_key = None
-        inner_rel: Path | None = None
-        if root is not None and prov:
-            rel = f.relative_to(root)
-            p = rel.parent
-            while p != Path("."):
-                if str(p) in prov:
-                    dest_key, unit_name = str(p), prov[str(p)]
-                    # dirs-only relative path (the file name itself is
-                    # appended by build_target; keeping it here would
-                    # duplicate it in the target path)
-                    inner_rel = rel.relative_to(p).parent
-                    break
-                p = p.parent
-        if unit_name is None:
-            item = inspect(f)
-            classify_item(item)
-            _drop_dropped_system(item)
-            match_catalog(item, catalog, threshold)
-            if root is not None:
-                # manual trees: let the real ancestor dir names compete
-                rel = f.relative_to(root)
-                dirs = [(root.joinpath(*rel.parent.parts[:i + 1]),
-                         rel.parent.parts[i])
-                        for i in range(len(rel.parent.parts) - 1, -1, -1)]
-                winner = _rehome_from_dirs(item, dirs, catalog, threshold,
-                                           dir_cache)
-                if winner is not None:
-                    inner_rel = f.parent.relative_to(winner)
-        else:
-            # one classification per extraction directory, reused for
-            # every file inside it
-            fields = unit_cache.get(dest_key)
-            if fields is None:
-                listing = sorted(
-                    str(g.relative_to(root / dest_key))
-                    for g in (root / dest_key).rglob("*") if g.is_file()
-                )[:40]
-                probe = SortItem(
-                    path=f, size=0, name=unit_name,
-                    clean=_clean_name(unit_name), inner_names=listing,
-                )
-                classify_item(probe)
-                _drop_dropped_system(probe)
-                match_catalog(probe, catalog, threshold)
-                fields = {k: getattr(probe, k) for k in (
-                    "system", "group", "faction", "subfaction", "klass",
-                    "unit", "match_score", "match_kind", "confidence",
-                )}
-                unit_cache[dest_key] = fields
-            item = inspect(f)
-            for k, v in fields.items():
-                setattr(item, k, v)
-            if inner_rel is not None and inner_rel.parts:
-                # extracted archives: inner dir names may claim subtrees
-                dirs = []
-                for i in range(len(inner_rel.parts) - 1, -1, -1):
-                    d = (root / dest_key).joinpath(*inner_rel.parts[:i + 1])
-                    dirs.append((d, inner_rel.parts[i]))
-                winner = _rehome_from_dirs(item, dirs, catalog, threshold,
-                                           dir_cache, prefer_dirs_on_tie=True)
-                if winner is not None:
-                    inner_rel = f.parent.relative_to(winner)
-        item.target = build_target(item, library, inner_rel=inner_rel)
-        item.action = "link"
-        items.append(item)
-    return items
-
-
-def apply_plan(items: list[SortItem], audit_path: Path) -> tuple[int, int, int]:
-    """Create targets. Hardlink preferred; copy when filesystems differ.
-
-    Returns (linked, copied, skipped).
-    """
-    audit_path.parent.mkdir(parents=True, exist_ok=True)
-    linked = copied = skipped = weak = 0
-    weak_lines: list[str] = []
-    with audit_path.open("a", encoding="utf-8") as audit:
-        for item in items:
-            src, dest = item.path, item.target
-            if item.match_kind == "invalid":
-                log.error("Not sorting %s: it is an HTML page, not an archive", src.name)
-                skipped += 1
-                continue
-            if dest is None or not src.is_file():
-                skipped += 1
-                continue
-            if dest.exists():
-                log.info("Skip, target exists: %s", dest)
-                skipped += 1
-                continue
-            dest.parent.mkdir(parents=True, exist_ok=True)
-            if os.stat(src).st_dev == os.stat(dest.parent).st_dev:
-                os.link(src, dest)
-                item.action = "link"
-                linked += 1
-            else:
-                log.warning(
-                    "Different filesystem - COPYING instead of hardlinking %s", src.name
-                )
-                shutil.copy2(src, dest)
-                item.action = "copy"
-                copied += 1
-            if item.match_kind == "weak":
-                weak += 1
-                weak_lines.append(
-                    f"{item.name}  (score {item.match_score:.2f})"
-                    f"\n      -> {dest}\n      from {src}\n"
-                )
-            audit.write(json.dumps({
-                "from": str(src), "to": str(dest), "action": item.action,
-                "unit": item.unit, "faction": item.faction, "class": item.klass,
-                "match": item.match_kind, "confidence": item.confidence,
-            }) + "\n")
-    if weak:
-        weak_doc = audit_path.parent / "sorter_low_confidence.txt"
-        weak_doc.write_text(
-            "Low-confidence placements - the sorter accepted these below the "
-            "normal\nthreshold because nothing better matched. Check and move "
-            "manually if wrong.\n\n" + "\n".join(weak_lines),
-            encoding="utf-8",
-        )
-        log.warning(
-            "%d low-confidence placements - see %s", weak, weak_doc
-        )
-    return linked, copied, skipped
-
-
-def format_plan(items: list[SortItem]) -> str:
-    lines = []
-    for item in items:
-        if item.match_kind == "invalid":
-            lines.append(f"[x] {item.name} - NOT AN ARCHIVE (HTML page, bad download)")
-            continue
-        src = f"{item.name} ({item.size // (1024 * 1024)} MiB)" if item.size > 4 * 1024 * 1024 else item.name
-        flag = {"catalog": "+", "weak": "w", "implicit": "~",
-                "none": "?"}[item.match_kind]
-        lines.append(f"[{flag}] {src}\n      -> {item.target}")
-    return "\n".join(lines)

version-python/dlphelper/state.py

@@ -1,78 +0,0 @@
-"""Persistent per-URL state: makes runs resumable and failures trackable.
-
-state.json lives in the run's OUT_DIR and maps every URL seen to a record:
-
-    {
-      "status": "ok" | "failed" | "error",
-      "attempts": <int>,
-      "error": "<last error message>",
-      "files": [{"path": ..., "size": ...}, ...],
-      "updated": "<iso timestamp>"
-    }
-
-Re-running skips URLs whose status is "ok"; failed URLs are retried and also
-written to failed_urls.txt for easy manual retry lists.
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import time
-from pathlib import Path
-
-log = logging.getLogger(__name__)
-
-
-class State:
-    """JSON-backed state store for one run directory."""
-
-    def __init__(self, path: Path):
-        self.path = path
-        self.records: dict[str, dict] = {}
-        if path.exists():
-            try:
-                self.records = json.loads(path.read_text(encoding="utf-8"))
-                if not isinstance(self.records, dict):
-                    raise ValueError("state root must be an object")
-            except (ValueError, OSError) as exc:
-                backup = path.with_suffix(".json.bak")
-                log.warning("Corrupt state file %s (%s); moving to %s and starting fresh",
-                            path, exc, backup)
-                path.replace(backup)
-                self.records = {}
-
-    def is_done(self, url: str) -> bool:
-        rec = self.records.get(url)
-        return bool(rec) and rec.get("status") == "ok"
-
-    def pending(self, urls: list[str]) -> list[str]:
-        return [u for u in urls if not self.is_done(u)]
-
-    def mark(self, url: str, **fields) -> dict:
-        rec = self.records.setdefault(url, {"status": "pending", "attempts": 0, "files": []})
-        for key, value in fields.items():
-            if value is not None:
-                rec[key] = value
-        rec["attempts"] = rec.get("attempts", 0) + 1
-        rec["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
-        self.save()
-        return rec
-
-    def failures(self) -> list[str]:
-        return [u for u, r in self.records.items() if r.get("status") != "ok"]
-
-    def save(self) -> None:
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        tmp = self.path.with_suffix(".json.tmp")
-        tmp.write_text(
-            json.dumps(self.records, indent=2, sort_keys=True), encoding="utf-8"
-        )
-        tmp.replace(self.path)
-
-
-def append_manifest(manifest_path: Path, row: dict) -> None:
-    """Append one JSON line to manifest.jsonl (per downloaded file)."""
-    manifest_path.parent.mkdir(parents=True, exist_ok=True)
-    with manifest_path.open("a", encoding="utf-8") as fh:
-        fh.write(json.dumps(row, ensure_ascii=False) + "\n")

version-python/dlphelper/warhammer_data.py

@@ -1,529 +0,0 @@
-"""Warhammer taxonomy and unit keyword tables used by the catalogue.
-
-Naming follows the Games Workshop webshop hierarchy (the same one
-dlphelper.gwshop dumps from Algolia):
-
-    <System>/<Group>/<Faction>/<Class>          e.g.
-    Warhammer 40,000/Space Marines/Ultramarines/Walker
-
-Two layers:
-- SYSTEMS: game system -> faction -> keyword aliases (matched against file
-  names). Faction detection is the primary signal; unit names double as
-  faction hints since they are listed under their faction. Space Marine
-  chapters are their own GW factions under the "Space Marines" group.
-- CLASSES: keyword -> class. With GW as the catalog source the class layer
-  holds GW shop categories (unit types: Battleline, Infantry, Walker,
-  Vehicle, Flyer, Monster, Mounted, Hero, Titanic).
-
-The unit table grows with the collection; it does not attempt to enumerate
-every unit Games Workshop ever released - unknown files land in the
-"unclassified" bucket of the catalogue report so gaps become visible.
-"""
-
-from __future__ import annotations
-
-# ---------------------------------------------------------------------------
-# Game systems and factions (keyword aliases, lowercase)
-# ---------------------------------------------------------------------------
-
-SYSTEMS: dict[str, dict[str, list[str]]] = {
-    "Warhammer 40,000": {
-        # -- Space Marines (chapters are their own GW factions) --------------
-        "Space Marines": [
-            "space marine", "astartes", "primaris", "intercessor", "reiver",
-            "aggressor", "inceptor", "hellblaster", "eradicator", "bladeguard",
-            "sternguard", "devastator", "tactical squad", "terminator squad",
-            "assault marine", "assault intercessor", "outrider", "scout squad",
-            "vanguard veteran", "invictor", "impulsor", "gladiator",
-            "land speeder", "drop pod", "inner circle", "honour guard",
-        ],
-        "Ultramarines": ["ultramarines", "guilliman", "calgar"],
-        "Blood Angels": [
-            "blood angel", "sanguinary guard", "sanguinor", "death company",
-            "baal predator", "mephiston", "dante", "astorath", "flesh tearers",
-            "lamberto", "furioso",
-        ],
-        "Dark Angels": [
-            "dark angel", "ravenwing", "deathwing", "inner circle companions",
-            "azrael", "asmodai", "belial", "samuel", "lion el'jonson",
-            "lion el jonson", "nephilim jetfighter",
-        ],
-        "Space Wolves": [
-            "space wolf", "space wolves", "wulfen", "thunderwolf", "wolf guard",
-            "rune priest", "wolf priest", "wolf lord", "logan grimnar",
-            "ragnar blackmane", "canis wolfborn", "fenrisian wolf",
-            "stormfang", "murderfang", "lukas the trickster",
-        ],
-        "Black Templars": [
-            "black templar", "crusader squad", "high marshal helbrecht",
-            "grimaldus", "the emperor's champion", "emperors champion",
-            "castellan", "land raider crusader",
-        ],
-        "Imperial Fists": [
-            "imperial fist", "rogal dorn", "lupos", "tor garadon",
-            "horatius hold", "dorn's arrow",
-        ],
-        "Raven Guard": [
-            "raven guard", "kayvaan shrike", "corvus", "sable brand",
-            "shadow captains",
-        ],
-        "Iron Hands": [
-            "iron hand", "iron hands", "ferrus manus", "iron father",
-            "medusan immortals",
-        ],
-        "White Scars": ["white scar", "jaghatai khan", "stormseer", "korsos"],
-        "Salamanders": [
-            "salamander", "vulkan", "he'stan", "adrax agatone",
-            "drake-scale", "flamecaster",
-        ],
-        "Grey Knights": [
-            "grey knight", "grand master", "brother-captain", "nemesis dreadknight",
-            "draigo", "castellan crowe", "sternguard knight",
-        ],
-        "Deathwatch": [
-            "deathwatch", "watch captain", "watch master", "kill team cassius",
-            "overkill", "corvus blackstar",
-        ],
-        "Crimson Fists": ["crimson fist", "kantor", "rynn's world"],
-        "Flesh Tearers": ["flesh tearer", "gabriel seth", "amissum"],
-        # -- Imperium --------------------------------------------------------
-        "Astra Militarum": [
-            "astra militarum", "imperial guard", "cadian", "catachan", "krieg",
-            "death korps", "tempestus", "scion", "valhallan", "steel legion",
-            "tallarn", "vostroyan", "tanith", "gaunt's ghosts", "leman russ",
-            "chimera", "basilisk", "hydra", "wyvern", "manticore", "deathstrike",
-            "sentinel", "taurox", "commissar", "bullgryn", "ogryn", "ratling",
-            "heavy weapon squad", "baneblade", "stormlord", "shadowsword",
-            "hellhammer", "malcador", "macharius", "valkyrie", "vulture",
-            "primaris psyker", "astropath", "priest",
-        ],
-        "Adeptus Mechanicus": [
-            "adeptus mechanicus", "mechanicum", "admech", "skitarii",
-            "cult mechanicus", "sicarian", "ruststalker", "infiltrator",
-            "kataphron", "kastelan", "ironstrider", "sydonian",
-            "onager", "dunecrawler", "maniple", "archmagos", "magos", "dominus",
-            "manipulus", "enginseer", "transuranic", "tech-priest", "cybernetica",
-            "thallax", "castellax", "vorax", "ordinatus", "belisarius cawl",
-        ],
-        "Adepta Sororitas": [
-            "adepta sororitas", "sisters of battle", "sister of battle",
-            "battle sister", "repentia", "retributor", "canoness", "dialogus",
-            "hospitaller", "seraphim", "zephyrim", "paragon warsuit",
-            "morvenn vahl", "junith", "immersa", "exorcist", "immolator",
-        ],
-        "Adeptus Custodes": [
-            "adeptus custodes", "custodes", "custodian", "allarus", "vertus praetor",
-            "venatari", "aquilon", "sagittarum", "hetaeron", "trajann", "constantin valdor",
-            "sisters of silence", "witchseeker", "prosecutor", "vigilator",
-            "caladius", "coronus", "telemon", "achillus", "galatus", "pallas",
-        ],
-        "Imperial Knights": [
-            "imperial knight", "knight errant", "knight paladin", "knight crusader",
-            "knight warden", "knight gallant", "knight lancer", "knight acheron",
-            "knight castellan", "knight valiant", "knight preceptor", "questoris",
-            "armiger", "warglaive", "helverin", "morialtis", "canis rex",
-            "cerastus", "atropos", "seneschal",
-        ],
-        "Imperial Agents": [
-            "inquisition", "inquisitor", "astra telepathica", "witch hunter",
-            "coteaz", "eisenhorn", "ravenor", "kyria draxus",
-            "officio assassinorum", "culexus", "vindicare", "callidus", "eversor",
-            "saint celestine", "living saint", "assassin",
-        ],
-        # -- Chaos -----------------------------------------------------------
-        "Chaos Space Marines": [
-            "chaos space marine", "black legion", "iron warriors", "word bearers",
-            "night lords", "night lord", "alpha legion", "chaos terminator", "chosen", "havoc",
-            "raptor", "warp talon", "chaos cultist", "cultist", "daemon prince",
-            "chaos lord", "chaos sorcerer", "abaddon", "kharn", "ahriman",
-            "typhus", "fabius bile", "vashtorr", "defiler", "forgefiend",
-            "maulerfiend", "helldrake", "helbrute", "chaos spawn",
-            "obliterator", "chaos rhino", "chaos vindicator", "spartan assault",
-            "mastodon", "typhon", "csm",
-        ],
-        "Death Guard": [
-            "death guard", "nurgle", "plague marine", "deathshroud", "mortarion",
-            "plaguecaster", "poxwalker", "blightlord", "foetid bloat-drone",
-            "plagueburst crawler", "myphitic blight-hauler", "blight hauler",
-            "tallyman", "foul blightspawn", "biologus putrifier",
-            "plague surgeon", "necrosius",
-        ],
-        "Thousand Sons": [
-            "thousand sons", "tzeentch", "rubric marine", "rubricae",
-            "scarab occult", "magnus the red", "ahzek ahriman", "arhiman",
-            "exalted sorcerer", "osiron", "mutalith vortex beast", "tzaangor",
-            "prospero",
-        ],
-        "Chaos Daemons": [
-            "chaos daemon", "chaos daemons", "daemonette", "bloodletter",
-            "plaguebearer", "horror", "screamer", "flesh hound", "juggernaut",
-            "keeper of secrets", "skarbrand", "kairos fateweaver", "rotigus",
-            "be'lakor", "great unclean one", "lord of change", "soul grinder",
-            "skull cannon", "burning chariot", "exalted flamer",
-            "fiend of slaanesh", "plague drone", "nurgling", "herald",
-            "skulltaker", "khorne", "slaanesh", "skull lord",
-            "shalaxi helbane",
-        ],
-        "World Eaters": [
-            "world eater", "angron", "eightbound", "exalted eightbound",
-            "jakhals", "berserker", "skull lord",
-        ],
-        "Emperor's Children": [
-            "emperor's children", "emperors children", "fulgrim",
-            "noise marine", "lucius the eternal", "flawless blades",
-        ],
-        "Iron Warriors": ["iron warrior", "perturabo"],
-        "Red Corsairs": ["red corsair", "huron blackheart"],
-        "Chaos Knights": [
-            "chaos knight", "knight desecrator", "knight despoiler",
-            "knight rampager", "kharibdyss", "renegade knight",
-        ],
-        # -- Xenos -----------------------------------------------------------
-        "Orks": [
-            "ork", "orks", "orkz", "nob", "boyz", "grot", "weirdboy", "painboy",
-            "warboss", "ghazghkull", "meganob", "deff dread", "killa kan",
-            "deffkopta", "stormboy", "kommando", "burna boy", "loota",
-            "trukk", "battlewagon", "gorkanaut", "morkanaut", "stompa", "squig",
-            "zagstruk", "snikrot", "speed freek", "boomdakka", "flash gitz",
-            "wazbom blasta", "dakka jet", "burna-bomber", "blitza-bomber",
-            "da troopz", "dad and sonz", "ass sassins", "abby boi",
-        ],
-        "T'au Empire": [
-            "tau", "t'au", "ta'u", "kroot", "vespid", "battlesuit", "crisis suit",
-            "broadside", "r'varna", "ghostkeel", "stormsurge",
-            "fire warrior", "pathfinder team", "devilfish", "hammerhead",
-            "sky ray", "piranha", "sniper drone team", "ethereal",
-            "commander farsight", "commander shadowsun", "aun'va", "darkstrider",
-            "longstrike", "mont'ka", "assorted drones", "breacher",
-            "bomber", "tetra", "barracuda", "sun shark", "razorshark",
-            "kv128 stormsurge", "tx4 piranha", "kv46",
-        ],
-        "Tyranids": [
-            "tyranid", "hive tyrant", "swarmlord", "carnifex", "hormagaunt",
-            "termagant", "termagaunt", "genestealer brood", "zoanthrope", "venomthrope",
-            "exocrine", "haruspex", "tyranid prime", "tyranid warrior",
-            "ravener", "gargoyle", "spore mine", "mawloc", "trygon",
-            "harridan", "harpy", "crone", "tervigon", "barbed hierodule",
-            "hierodule", "hierophant", "neurothrope", "deathleaper", "lictor", "pyrovore",
-            "biovore", "broodlord", "bio-titan", "dominatrix",
-        ],
-        "Genestealer Cults": [
-            "genestealer cult", "genestealer", "aberrant", "acolyte", "neophyte hybrid",
-            "primus", "magus", "patriarch", "purestrain genestealer",
-            "sanctus", "kelermorph", "goliath truck", "rockgrinder",
-            "jackal alcaeus", "cult limo", "hijacked",
-        ],
-        "Necrons": [
-            "necron", "c'tan", "canoptek", "wraith", "spyder", "scarab",
-            "tesseract vault", "doomsday ark", "annihilation barge",
-            "ghost ark", "monolith", "doom scythe", "night scythe",
-            "imotekh", "szarekh", "nemesor zahndrekh", "vargard obyron",
-            "lokhrust", "skorpekh", "flayed one", "deathmark", "lychguard",
-            "necron warrior", "necron immortal", "ovoid", "dzaan",
-        ],
-        "Aeldari": [
-            "aeldari", "eldar", "craftworld", "ulthwe", "biel-tan", "iyanden",
-            "alaitoc", "saim-hann", "farseer", "warlock",
-            "spiritseer", "autarch", "asurmen", "guardian defender",
-            "ranger", "wraithguard", "wraithblades", "wraithlord",
-            "wraithknight", "dire avenger", "howling banshee",
-            "striking scorpion", "swooping hawk", "warp spider",
-            "fire dragon", "dark reaper", "shining spears", "windrider",
-            "vyper", "wave serpent", "fire prism", "night spinner",
-            "war walker", "hemlock wraithfighter", "crimson hunter",
-            "shade runner", "skyrunner",
-        ],
-        "Drukhari": [
-            "drukhari", "dark eldar", "kabal", "wych", "wrack", "haemonculus",
-            "coven", "reaver jetbike", "hellion", "scourge", "mandrake",
-            "kabalite warrior", "raider", "ravager", "venom", "razorwing",
-            "voidraven", "grotesque", "talos", "cronos",
-            "beast pack", "khymera", "razorwing flock", "asdrubael vect",
-            "lilith", "sathonyx", "luriel",
-        ],
-        "Harlequins": [
-            "harlequin", "death jester", "shadowseer", "solitaire",
-            "troupe", "starweaver", "voidweaver",
-        ],
-        "Leagues of Votann": [
-            "leagues of votann", "votann", "hearthkyn", "ironkin",
-            "brokhyr", "kahl", "cthonian", "transhyperic", "thegn",
-        ],
-    },
-    "Age of Sigmar": {
-        "Stormcast Eternals": [
-            "stormcast", "eternal", "liberator", "sequitor", "evocator",
-            "vindictor", "judicators", "retributors", "sacrosanct",
-            "dracothian", "lord-arcane", "knight-questor", "neave blacktalon",
-            "stormdrake", "krondys", "karazai",
-        ],
-        "Seraphon": ["seraphon", "saurus", "skink", "slann", "carnosaur",
-                     "bastiladon", "stegadon", "razordon", "salamander"],
-        "Orruk Warclans": ["orruk", "warclan", "kruleboyz", "ironjawz",
-                          "brute", "gorger", "weirdnob", "warchanter"],
-        "Gloomspite Gitz": ["gloomspite", "gitz", "moonclan", "spiderfang",
-                            "troggoth", "loontouched", "zarbag", "skragrot"],
-        "Sylvaneth": ["sylvaneth", "treelord", "dryad", "branchwraith",
-                      "alarielle", "kurnothi"],
-        "Ossiarch Bonereapers": ["ossiarch", "bonereaper", "mortek", "keldrek",
-                                 "katakros"],
-        "Nighthaunt": ["nighthaunt", "grimghast", "bladegheist", "chainghast",
-                       "kurdoss", "lady olynder"],
-        "Lumineth Realm-lords": ["lumineth", "aelf", "vanari", "teclis",
-                                 "light of eltharion"],
-        "Daughters of Khaine": ["daughters of khaine", "khainite", "witch aelf",
-                                "morgwaeth", "morathi"],
-        "Fyreslayers": ["fyreslayer", "vulkite", "hearthguard", "auric",
-                        "grimhold"],
-        "Kharadron Overlords": ["kharadron", "skyfarer", "endofters",
-                                "grundstok", "arcanaut"],
-        "Idoneth Deepkin": ["idoneth", "deepkin", "namarti", "isharann",
-                            "volturnos"],
-        "Blades of Khorne": ["blades of khorne", "khorne", "skullgrinder",
-                             "blood warrior", "wrathmonger"],
-        "Maggotkin of Nurgle": ["maggotkin", "nurgle", "blightking",
-                                "rotmire", "orghotts"],
-        "Slaves to Darkness": ["slaves to darkness", "chaos warrior",
-                               "chaos knight", "archaon", "gaunt summoner",
-                               "everchosen"],
-        "Skaven": ["skaven", "clanrat", "stormvermin", "verminlord",
-                   "grey seer", "warplock", "ratling gun"],
-        "Soulblight Gravelords": ["soulblight", "gravelord", "vampire",
-                                  "deathrattle", "koldhunter", "belladamma"],
-        "Cities of Sigmar": ["cities of sigmar", "freeguild", "windspeaker",
-                             "callis and toll", "hammerers"],
-    },
-    "The Horus Heresy": {
-        "Space Marine Legions": [
-            "legiones astartes", "pre-heresy", "horus heresy", "justaerin",
-            "seeker squad", "suzerain", "invictarus", "veteran tactical",
-            "contemptor", "leviathan pattern", "sicaran", "fellblade",
-            "kratos", "fire raptor", "eisenstein",
-        ],
-        "Mechanicum": ["mechanicum", "fulgurite", "magos", "thallax"],
-    },
-    "Other Games": {
-        "Kill Team": [
-            "kill team", "killteam", "killzone", "operative",
-            "hand of the archon", "void-dance troupe", "veteran guardsmen",
-            "spector squad", "wolf scouts", "scout squad kill",
-        ],
-        "Necromunda": [
-            "necromunda", "goliath gang", "escher", "orlock", "cawdor",
-            "van saar", "ash wastes", "underhive", "sumpkrocs",
-            "forge boss", "sumpkroc", "matriarch", "phyrr cat",
-        ],
-        "Adeptus Titanicus": [
-            "adepticus titanicus", "titan legio", "reaver titan",
-            "warhound titan", "warlord titan", "imperator titan",
-            "warmaster titan", "titanicus",
-        ],
-        "Warhammer Underworlds": ["warhammer underworlds", "underworlds",
-                                  "warband", "shadeborn", "embergard"],
-        "Aeronautica Imperialis": ["aeronautica", "t'au air caste",
-                                   "lightning fighter", "thunderbolt"],
-    },
-    "The Old World": {
-        "Kingdom of Bretonnia": ["bretonnia", "grail knight", "peasant",
-                                 "questing knight", "perrin"],
-        "Orc and Goblin Tribes": ["greenskin", "goblin", "night goblin",
-                                  "forest goblin", "black orc"],
-        "Tomb Kings of Khemri": ["tomb king", "khemri", "necrolith",
-                                 "skeleton archer"],
-        "Empire of Man": ["empire of man", "state troop", "greatsword",
-                          "empire knight"],
-        "Warriors of Chaos": ["chaos dwarf", "dog of war", "chaos marauder"],
-        "High Elf Realms": ["high elf", "eltharion", "phoenix guard"],
-        "Wood Elf Realms": ["wood elf", "glade guard", "waywatcher"],
-        "Beastmen Brayherds": ["beastmen", "brayherd", "ghorgon", "cygor"],
-        "Dwarfen Mountain Holds": ["dwarf", "ironbreaker", "hammerer"],
-    },
-    "Middle-Earth": {
-        "Rivendell": ["rivendell", "elrond", "elendil", "isildur", "gil-galad",
-                      "beorning", "grimbeorn", "rumil", "orophin"],
-        "Minas Tirith": ["minas tirith", "gondor", "boromir", "faramir"],
-        "Mordor": ["mordor", "orc", "orcs", "nazgul", "witch-king"],
-        "Iron Hills": ["iron hills", "dwarf", "mattock", "dwalin"],
-        "Moria": ["moria", "cave troll", "balrog"],
-    },
-}
-
-# ---------------------------------------------------------------------------
-# Classes (keyword -> class). GW shop categories / unit types.
-# ---------------------------------------------------------------------------
-
-CLASSES: list[tuple[str, list[str]]] = [
-    ("Titanic", [
-        "titan", "stompa", "gargantuan", "hierophant", "harridan", "hierodule",
-        "imperator", "warmaster", "tesseract vault", "plaguereaper",
-        "warhound", "warlord titan", "ta'unar", "spermato", "titandeath",
-    ]),
-    ("Fortification", ["fortification", "bunker", "bastion", "wall of martyrs"]),
-    ("Flyer", [
-        "flyer", "heldrake", "dakka jet", "burna-bomber", "blitza-bomber",
-        "bomber", "sky ray", "stormraven", "storm talon", "stormhawk",
-        "caestus", "valkyrie", "vulture", "thunderbolt", "lightning",
-        "marauder", "razorwing", "voidraven", "crimson hunter",
-        "hemlock", "doom scythe", "night scythe", "harpy", "crone",
-        "fire raptor", "barracuda", "sun shark", "razorshark",
-        "wazbom blasta", "nephilim jetfighter", "corvus blackstar",
-    ]),
-    ("Vehicle", [
-        "transport", "rhino", "razorback", "chimera", "taurox", "trukk",
-        "battlewagon", "devilfish", "wave serpent", "ghost ark",
-        "land raider", "repulsor", "drop pod", "starweaver", "voidweaver",
-        "goliath truck", "rockgrinder", "cult limo", "raider", "venom",
-        "spartan assault", "predator", "vindicator", "whirlwind",
-        "basilisk", "hydra", "wyvern", "manticore", "deathstrike",
-        "onager dunecrawler", "kastelan", "hammerhead", "broadside",
-        "stormsurge", "r'varna", "soul grinder", "defiler", "forgefiend",
-        "maulerfiend", "plagueburst crawler", "annihilation barge",
-        "doomsday ark", "fire prism", "night spinner", "exorcist",
-        "immolator", "caladius", "coronus grav", "pallas grav",
-        "impulsor", "gladiator", "baal predator", "land speeder",
-        "ironstrider", "sydonian", "sentinel", "taurox", "kataphron",
-    ]),
-    ("Walker", [
-        "dreadnought", "redemptor", "contemptor", "leviathan pattern",
-        "brutalis", "ballistus", "deredeo", "wulfen dreadnought",
-        "killa kan", "deff dread", "wraithlord", "wraithknight",
-        "telemon", "achillus", "galatus", "paragon warsuit",
-        "nemesis dreadknight", "invictor", "talos", "cronos",
-    ]),
-    ("Monster", [
-        "monster", "carnifex", "exocrine", "haruspex", "mawloc", "trygon",
-        "tervigon", "hierodule", "hierophant", "mutalith", "bloat-drone",
-        "kharibdyss", "gorkanaut", "morkanaut", "magnus", "mortalith",
-        "fiend of slaanesh", "plague drone", "cave troll", "balrog",
-        "carnosaur", "stegadon", "bastiladon", "grorgeous",
-    ]),
-    ("Mounted", [
-        "mounted", "thunderwolf", "outrider", "bike squad", "scout biker",
-        "reaver jetbike", "windrider", "skyrunner", "warbiker",
-        "allarus custodian on", "vertus praetor", "drakeling",
-        "boarboy", "gruntaz", "khan on", "dracothian guard",
-    ]),
-    ("Battleline", [
-        "battleline", "troops", "tactical squad", "intercessor",
-        "incursor", "infiltrator", "infantry squad", "conscript",
-        "krieg", "catachan", "cadian", "vostroyan", "penal legion",
-        "ratling", "veteran squad", "fire warrior", "pathfinder team",
-        "kroot", "termagant", "hormagaunt", "necron warrior",
-        "necron immortal", "guardian defender", "storm guardian",
-        "kabalite warrior", "boyz", "gretchin", "scion", "scout squad",
-        "cultist", "plague marine squad",
-    ]),
-    ("Infantry", [
-        "infantry", "veteran", "terminator", "sternguard", "bladeguard",
-        "aggressor", "reiver", "wulfen", "wolf guard", "deathshroud",
-        "blightlord", "poxwalker", "rubric marine", "rubricae",
-        "scarab occult", "tzaangor", "chosen", "possessed", "obliterator",
-        "daemonette", "bloodletter", "plaguebearer", "nurgling",
-        "juggernaut", "chaos cultist", "acolyte hybrid", "aberrant",
-        "neophyte", "purestrain", "genestealer", "lictor", "deathmark",
-        "flayed one", "lychguard", "wraithguard", "wraithblades",
-        "howling banshee", "striking scorpion", "warp spider",
-        "fire dragon", "incubi", "wrack", "grotesque", "assault marine",
-        "assault intercessor", "inceptor", "raptor", "warp talon",
-        "flesh hound", "screamer", "hellion", "scourge", "mandrake",
-        "swooping hawk", "shining spears", "canoptek wraith",
-        "canoptek scarab", "canoptek spyder", "ravener", "gargoyle",
-        "spore mine", "pyrovore", "khymera", "beast pack",
-        "razorwing flock", "zephyrim", "seraphim", "stormboy",
-        "deffkopta", "skorpekh destroyer", "skorpekh", "retributor",
-        "dark reaper", "sicarian", "ruststalker", "skitarii",
-        "hormagaunt brood", "termagant brood", "vanguard",
-    ]),
-    ("Hero", [
-        "hero", "hq", "captain", "chapter master", "primarch", "lieutenant",
-        "librarian", "chaplain", "apothecary", "farseer", "warlock",
-        "spiritseer", "autarch", "warboss", "big mek", "weirdboy",
-        "painboy", "commander", "ethereal", "hive tyrant", "swarmlord",
-        "tyranid prime", "neurothrope", "deathleaper", "patriarch",
-        "primus", "magus", "sanctus", "kelermorph", "overlord",
-        "imotekh", "szarekh", "zahndrekh", "obyron", "plaguecaster",
-        "necrosius", "chaos lord", "chaos sorcerer", "daemon prince",
-        "exalted champion", "herald", "skulltaker", "kairos", "rotigus",
-        "be'lakor", "great unclean one", "lord of change",
-        "keeper of secrets", "skarbrand", "kharn", "abaddon", "typhus",
-        "mortarion", "magnus", "ahriman", "lucius", "fabius bile",
-        "vashtorr", "canoness", "morvenn", "junith", "immersa",
-        "dialogus", "hospitaller", "trajann", "valdor", "inquisitor",
-        "coteaz", "eisenhorn", "ravenor", "draxus", "celestine",
-        "commissar", "priest", "enginseer", "dominus", "manipulus",
-        "archmagos", "belisarius cawl", "farsight", "shadowsun",
-        "aun'va", "darkstrider", "longstrike", "shade runner",
-    ]),
-    ("Rules & Accessories", [
-        "codex", "rulebook", "rules", "data cards", "datacards",
-        "warscroll", "index card", "cheat sheet", "reference",
-    ]),
-]
-
-# Flattened per-system faction keyword index for quick matching
-FACTION_ALIASES: dict[str, dict[str, str]] = {
-    system: {alias: faction for faction, aliases in factions.items() for alias in aliases}
-    for system, factions in SYSTEMS.items()
-}
-
-UNIT_CLASSES: dict[str, str] = {kw: klass for klass, kws in CLASSES for kw in kws}
-
-# GW group of each faction (matches the GameSystemsRoot.lvl1 values in the
-# Algolia index). Used when a keyword-classified file cannot be fuzzy-matched.
-GROUP_OF_FACTION: dict[str, str] = {
-    faction: group
-    for group, factions in {
-        "Space Marines": [
-            "Space Marines", "Ultramarines", "Blood Angels", "Dark Angels",
-            "Space Wolves", "Black Templars", "Imperial Fists", "Raven Guard",
-            "Iron Hands", "White Scars", "Salamanders", "Grey Knights",
-            "Deathwatch", "Crimson Fists", "Flesh Tearers",
-        ],
-        "Armies of the Imperium": [
-            "Astra Militarum", "Adeptus Mechanicus", "Adepta Sororitas",
-            "Adeptus Custodes", "Imperial Knights", "Imperial Agents",
-        ],
-        "Armies of Chaos": [
-            "Chaos Space Marines", "Death Guard", "Thousand Sons",
-            "Chaos Daemons", "Chaos Knights", "World Eaters",
-            "Emperor's Children", "Iron Warriors", "Red Corsairs",
-        ],
-        "Xenos Armies": [
-            "Orks", "T'au Empire", "Tyranids", "Genestealer Cults", "Necrons",
-            "Aeldari", "Drukhari", "Harlequins", "Leagues of Votann",
-        ],
-        "Grand Alliance Order": [
-            "Stormcast Eternals", "Seraphon", "Sylvaneth",
-            "Lumineth Realm-lords", "Daughters of Khaine", "Fyreslayers",
-            "Kharadron Overlords", "Idoneth Deepkin", "Cities of Sigmar",
-        ],
-        "Grand Alliance Destruction": [
-            "Orruk Warclans", "Gloomspite Gitz",
-        ],
-        "Grand Alliance Death": [
-            "Ossiarch Bonereapers", "Nighthaunt", "Soulblight Gravelords",
-        ],
-        "Grand Alliance Chaos": [
-            "Slaves to Darkness", "Blades of Khorne", "Maggotkin of Nurgle",
-            "Skaven",
-        ],
-        "Space Marine Legions": ["Space Marine Legions", "Mechanicum"],
-        "Kill Team": ["Kill Team"],
-        "Necromunda": ["Necromunda"],
-        "Adeptus Titanicus": ["Adeptus Titanicus"],
-        "Warhammer Underworlds": ["Warhammer Underworlds"],
-        "Aeronautica Imperialis": ["Aeronautica Imperialis"],
-        "Armies of the Old World": [
-            "Kingdom of Bretonnia", "Orc and Goblin Tribes",
-            "Tomb Kings of Khemri", "Empire of Man", "Warriors of Chaos",
-            "High Elf Realms", "Wood Elf Realms", "Beastmen Brayherds",
-            "Dwarfen Mountain Holds",
-        ],
-        "Rivendell": ["Rivendell"],
-        "Minas Tirith": ["Minas Tirith"],
-        "Mordor": ["Mordor"],
-        "Iron Hills": ["Iron Hills"],
-        "Moria": ["Moria"],
-    }.items()
-    for faction in factions
-}

version-python/dlphelper/zero807.py

@@ -1,92 +0,0 @@
-"""Downloader for 0807.st direct links via curl.
-
-0807.st serves the files directly but rejects requests that don't look like a
-browser (requires a User-Agent and the Referer of the site page), and it
-rejects multi-connection/range abuse patterns differently than curl - which is
-why curl is used instead of aria2c here (see chat history: curl worked,
-aria2c's default request got 403).
-
-Downloads go to a .part file first and are renamed on success, so an
-interrupted download is resumable (--continue-at -) and partial files are
-never mistaken for finished ones.
-"""
-
-from __future__ import annotations
-
-import logging
-import os
-import subprocess
-from pathlib import Path
-from urllib.parse import urlparse
-
-log = logging.getLogger(__name__)
-
-USER_AGENT = "Mozilla/5.0"
-REFERER = "https://0807.st/"
-
-
-class Zero807Error(Exception):
-    """Base error for 0807 downloads."""
-
-
-class Zero807NotFound(Zero807Error):
-    """The file no longer exists (HTTP 404/410). Not retryable."""
-
-
-def target_name(url: str) -> str:
-    """Derive the filename from the URL path (e.g. /6saCNie.zip -> 6saCNie.zip)."""
-    name = Path(urlparse(url).path).name
-    if not name:
-        raise Zero807Error(f"cannot derive a filename from URL: {url}")
-    return name
-
-
-def download(
-    url: str,
-    out_dir: Path,
-    timeout_seconds: int = 7200,
-    resume: bool = True,
-) -> tuple[Path, int]:
-    """Download one URL with curl; returns (final_path, size_bytes)."""
-    name = target_name(url)
-    out_dir.mkdir(parents=True, exist_ok=True)
-    dest = out_dir / name
-    tmp = dest.with_name(dest.name + ".part")
-
-    if dest.exists() and dest.stat().st_size > 0:
-        log.info("Skip existing: %s", dest)
-        return dest, dest.stat().st_size
-
-    cmd = [
-        "curl", "-fSL",
-        "--connect-timeout", "20",
-        "--max-time", str(timeout_seconds),
-        "--retry", "2", "--retry-delay", "5",
-        "-A", USER_AGENT,
-        "-e", REFERER,
-        "-o", str(tmp),
-    ]
-    if resume:
-        cmd.append("--continue-at")
-        cmd.append("-")
-    cmd.append(url)
-
-    log.info("Downloading %s -> %s", url, dest.name)
-    proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
-                          encoding="utf-8", errors="replace")
-    if proc.returncode != 0:
-        tail = [line for line in proc.stderr.strip().splitlines() if line][-3:]
-        stderr = " | ".join(tail)
-        # -f makes curl exit 22 on any HTTP error; dead links fail for good
-        if "404" in stderr or "410" in stderr:
-            raise Zero807NotFound(f"dead link (HTTP 404/410): {stderr}")
-        raise Zero807Error(f"curl exit code {proc.returncode}: {stderr}")
-
-    size = tmp.stat().st_size if tmp.exists() else 0
-    if size == 0:
-        tmp.unlink(missing_ok=True)
-        raise Zero807Error(f"empty file for {name}")
-    if dest.suffix.lower() == ".zip" and not tmp.read_bytes()[:2] == b"PK":
-        log.warning("%s does not look like a ZIP file (first bytes are not PK)", name)
-    os.replace(tmp, dest)
-    return dest, size

version-python/parts/grab.sh

@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-THREAD_ID="${THREAD_ID:-1455993943471820842}"
-OUT_DIR="${OUT_DIR:-$PWD/thread}"
-HTML="$OUT_DIR/export.html"
-URLS="$OUT_DIR/urls.txt"   # aria2c input file: one URL per line
-
-# Token from env, otherwise prompt (input hidden)
-if [[ -z "${DISCORD_TOKEN:-}" ]]; then
-  read -rsp "Discord token: " DISCORD_TOKEN
-  echo
-fi
-
-mkdir -p "$OUT_DIR"
-
-# 1. Export the thread to HTML
-docker run --rm --user "$(id -u):$(id -g)" \
-  -v "$OUT_DIR:/out" \
-  tyrrrz/discordchatexporter:stable export \
-  -t "$DISCORD_TOKEN" -c "$THREAD_ID" -f HtmlDark -o /out/export.html
-
-unset DISCORD_TOKEN
-
-# 2. Find base64 strings starting with "https://", then decode them
-grep -oP 'aHR0cHM6Ly[89][A-Za-z0-9+/_-]*' "$HTML" \
-  | sort -u \
-  | while read -r s; do
-      s=$(printf '%s' "$s" | tr '_-' '/+' | tr -d '=')
-      while (( ${#s} % 4 )); do s="$s="; done
-      printf '%s' "$s" | base64 -d 2>/dev/null | tr -d '\r\n' || true
-      echo
-    done \
-  | grep -E '^https://[[:graph:]]+$' \
-  | awk '!seen[$0]++' > "$URLS"
-
-echo "Found $(wc -l < "$URLS") URLs -> $URLS"
-echo "Download with: aria2c -i \"$URLS\" -d \"$OUT_DIR/downloads\""

version-python/requirements.txt

@@ -1 +0,0 @@
-requests>=2.31

version-python/sorter.py

@@ -1,140 +0,0 @@
-#!/usr/bin/env python3
-"""Sorter CLI: fuzzysort collection files into the library (plan/apply).
-
-    # plan (default, touches nothing)
-    python3 sorter.py <dir> --library /path/to/library
-
-    # apply: hardlink files into the library (originals stay!)
-    python3 sorter.py <dir> --library /path/to/library --apply
-
-    # widen/narrow fuzzy matching
-    python3 sorter.py <dir> --library ... --threshold 0.5
-
-    # sort a single archive
-    python3 sorter.py "Thunderwolf Cavalry.7z" --library ...
-
-Flags:
-    --catalog FILE   default data/library_catalog.json
-    --audit FILE     default <library>/../_logs/sorter_applied.jsonl
-                     (bookkeeping stays OUT of the library, which may be a
-                     seeded torrent; --audit still overrides)
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import logging
-import sys
-from pathlib import Path
-
-from dlphelper.sorter import (
-    DROPPED_SYSTEMS,
-    format_plan,
-    load_catalog,
-    sort_plan,
-    apply_plan,
-)
-
-log = logging.getLogger("sorter")
-
-DEFAULT_CATALOG = Path(__file__).resolve().parent / "data" / "gwshop_catalog.json"
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Sort files into the library structure (dry-run by default)."
-    )
-    parser.add_argument("source", type=Path,
-                        help="a directory to sort or a single file")
-    parser.add_argument("--library", type=Path, required=True, metavar="DIR",
-                        help="library root the files will be linked into")
-    parser.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG, metavar="FILE")
-    parser.add_argument("--threshold", type=float, default=0.62,
-                        help="fuzzy match threshold (default 0.62)")
-    parser.add_argument("--apply", action="store_true",
-                        help="create the targets (hardlink, fallback copy)")
-    parser.add_argument("--audit", type=Path, default=None, metavar="FILE")
-    parser.add_argument("--provenance", type=Path, default=None, metavar="FILE",
-                        help="extract_manifest.jsonl from tools/extract_all.py: "
-                             "sort whole extraction directories as units")
-    parser.add_argument("--json", action="store_true",
-                        help="print the plan as JSON instead of text")
-    parser.add_argument("--drop-system", action="append", default=[],
-                        metavar="NAME",
-                        help="never place into this game system "
-                             "(default-dropped: Middle-Earth)")
-    parser.add_argument("--keep-system", action="append", default=[],
-                        metavar="NAME",
-                        help="re-enable a default-dropped game system")
-    args = parser.parse_args()
-
-    dropped = frozenset(DROPPED_SYSTEMS - set(args.keep_system)
-                        | set(args.drop_system))
-    if args.keep_system or args.drop_system:
-        log.info("System filter adjusted: dropping %s", ", ".join(sorted(dropped)) or "nothing")
-
-    logging.basicConfig(level=logging.INFO,
-                        format="%(asctime)s %(levelname)-7s %(message)s",
-                        datefmt="%H:%M:%S")
-
-    source = args.source
-    files = [source] if source.is_file() else sorted(
-        p for p in source.rglob("*")
-        # hidden trees like .broken/ must never reach the plan
-        if not any(part.startswith(".") for part in p.relative_to(source).parts)
-    )
-    catalog = load_catalog(args.catalog, dropped=dropped)
-    if dropped:
-        log.info("Dropped systems: %s", ", ".join(sorted(dropped)))
-    # Bookkeeping must never land inside the library itself (it may be a
-    # seeded torrent whose file set is frozen): default to a _logs folder
-    # beside it. An explicit --audit overrides this.
-    audit = args.audit or (args.library.parent / "_logs" / "sorter_applied.jsonl")
-    log.info("Audit file: %s", audit)
-
-    provenance = None
-    if args.provenance:
-        provenance = {}
-        with args.provenance.open(encoding="utf-8") as fh:
-            for line in fh:
-                rec = json.loads(line)
-                provenance[rec["dest"]] = rec["unit"]
-        log.info("Provenance: %d extraction directories", len(provenance))
-
-    items = sort_plan(files, args.library, catalog, threshold=args.threshold,
-                      provenance=provenance, root=source)
-
-    if args.json:
-        print(json.dumps([
-            {
-                "source": str(i.path), "name": i.name, "size": i.size,
-                "system": i.system, "group": i.group, "faction": i.faction,
-                "subfaction": i.subfaction, "class": i.klass, "unit": i.unit,
-                "score": round(i.match_score, 3), "kind": i.match_kind,
-                "inner_count": len(i.inner_names),
-                "target": str(i.target),
-            }
-            for i in items
-        ], indent=2, ensure_ascii=False))
-    else:
-        print(format_plan(items))
-
-    n_catalog = sum(1 for i in items if i.match_kind == "catalog")
-    n_implicit = sum(1 for i in items if i.match_kind == "implicit")
-    n_none = sum(1 for i in items if i.match_kind == "none")
-    log.info("Plan: %d files -> %d catalog matches [+] / %d implicit [~] / "
-             "%d unclassifiable [?]",
-             len(items), n_catalog, n_implicit, n_none)
-
-    if args.apply:
-        linked, copied, skipped = apply_plan(items, audit)
-        log.info("Applied: %d hardlinked, %d copied, %d skipped (audit: %s)",
-                 linked, copied, skipped, audit)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        sys.exit(130)

version-python/tests/test_catalogue.py

@@ -1,62 +0,0 @@
-#!/usr/bin/env python3
-"""Offline classifier test for the catalogue (no network, no deps).
-
-Run with: python3 tests/test_catalogue.py
-"""
-
-from __future__ import annotations
-
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dlphelper.catalogue import classify  # noqa: E402
-
-CASES = [
-    # (filename, expected system, expected faction, expected class or None-any)
-    # GW naming: chapters are factions, classes are GW unit types.
-    ("Thunderwolf Cavalry.7z", "Warhammer 40,000", "Space Wolves", "Mounted"),
-    ("Wulfen.zip", "Warhammer 40,000", "Space Wolves", "Infantry"),
-    ("Keeper of Secrets & Shalaxi Helbane.7z", "Warhammer 40,000", "Chaos Daemons", "Hero"),
-    ("Assorted Drones.zip", "Warhammer 40,000", "T'au Empire", None),
-    ("Commander Shadowsun.zip", "Warhammer 40,000", "T'au Empire", "Hero"),
-    ("Devilfish.zip", "Warhammer 40,000", "T'au Empire", "Vehicle"),
-    ("Battle Sister Scanz.zip", "Warhammer 40,000", "Adepta Sororitas", None),
-    ("Knights.zip", None, None, None),  # ambiguous keyword alone: unclassified
-    ("Khorne Demon Enginez and skull lordz.zip", "Warhammer 40,000", "Chaos Daemons", None),
-    ("Cultists.zip", "Warhammer 40,000", "Chaos Space Marines", "Battleline"),
-    ("Da Troopz.zip", "Warhammer 40,000", "Orks", None),
-    ("Seraphon.zip", "Age of Sigmar", "Seraphon", None),
-    ("Leviathan Pattern Dreadnought.zip", "The Horus Heresy", "Space Marine Legions", "Walker"),
-    ("Poxwalkers.zip", "Warhammer 40,000", "Death Guard", "Infantry"),
-    ("Kabalite Warriors.zip", "Warhammer 40,000", "Drukhari", "Battleline"),
-    ("Canoptek Wraiths.zip", "Warhammer 40,000", "Necrons", "Infantry"),
-    ("random thing with no keywords.zip", None, None, None),
-]
-
-
-def main() -> None:
-    failures = 0
-    for name, exp_system, exp_faction, exp_class in CASES:
-        system, faction, klass = classify(name)
-        problems = []
-        if exp_system and system != exp_system:
-            problems.append(f"system={system!r} != {exp_system!r}")
-        if exp_faction and faction != exp_faction:
-            problems.append(f"faction={faction!r} != {exp_faction!r}")
-        if exp_class and klass != exp_class:
-            problems.append(f"class={klass!r} != {exp_class!r}")
-        if problems:
-            failures += 1
-            print(f"FAIL {name!r}: {', '.join(problems)}")
-        else:
-            print(f"ok   {name!r} -> {system} / {faction} / {klass}")
-    if failures:
-        print(f"\n{failures} failures")
-        sys.exit(1)
-    print("\ntest_catalogue: OK")
-
-
-if __name__ == "__main__":
-    main()

version-python/tests/test_extract.py

@@ -1,68 +0,0 @@
-#!/usr/bin/env python3
-"""Offline extraction test with a synthetic export.json (no network, no deps).
-
-Run with: python3 tests/test_extract.py
-"""
-
-from __future__ import annotations
-
-import base64
-import json
-import sys
-import tempfile
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dlphelper.discord import classify, extract_urls  # noqa: E402
-
-
-def b64(s: str) -> str:
-    return base64.b64encode(s.encode()).decode()
-
-
-def main() -> None:
-    messages = [
-        {
-            "id": "1",
-            "timestamp": "2026-01-01T00:00:00",
-            "author": {"name": "poster"},
-            "content": f"chaos stuff: {b64('https://gofile.io/d/abc123')}",
-        },
-        {
-            "id": "2",
-            "timestamp": "2026-01-02T00:00:00",
-            "author": {"name": "poster"},
-            "content": (
-                f"0807 zip: {b64('https://0807.st/6saCNie.zip')}\n"
-                f"dupe of 1: {b64('https://gofile.io/d/abc123')}\n"
-                f"plain: https://0807.st/plainLink.zip and "
-                f"https://discord.com/channels/1/2 (unsupported)\n"
-                f"junk: {b64('https://pixeldrain.com/u/xyz')} "
-                "and not-our-domain base64: aHR0cHM6Ly9leGFtcGxlLm9yZw=="
-            ),
-        },
-    ]
-
-    with tempfile.TemporaryDirectory() as tmp:
-        export = Path(tmp) / "export.json"
-        export.write_text(json.dumps({"messages": messages}), encoding="utf-8")
-        found = extract_urls(export)
-
-    urls = {f.url for f in found}
-    expected = {"https://gofile.io/d/abc123", "https://0807.st/6saCNie.zip",
-                "https://0807.st/plainLink.zip"}
-    assert urls == expected, f"unexpected urls: {sorted(urls)}"
-
-    by_url = {f.url: f for f in found}
-    g = by_url["https://gofile.io/d/abc123"]
-    assert g.host == "gofile" and g.message_id == "1" and g.author == "poster"
-    assert classify("https://0807.st/x.zip") == "0807"
-    assert classify("https://gofile.io/d/x") == "gofile"
-    assert classify("https://example.org/x") is None
-
-    print("test_extract: OK")
-
-
-if __name__ == "__main__":
-    main()

version-python/tests/test_sorter_units.py

@@ -1,258 +0,0 @@
-"""Offline sorter tests: dir re-homing, weak tier, junk dirs (plain python3)."""
-import json
-import shutil
-import sys
-import tempfile
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dlphelper.sorter import (  # noqa: E402
-    apply_plan,
-    load_catalog,
-    sort_plan,
-)
-
-CATALOG = {
-    "entries": [
-        {"name": "Warp Spiders", "system": "Warhammer 40,000",
-         "group": "Xenos Armies", "faction": "Aeldari",
-         "class": "Fly", "aliases": []},
-        {"name": "Harlequin Troupe", "system": "Warhammer 40,000",
-         "group": "Xenos Armies", "faction": "Harlequins",
-         "class": "Battleline", "aliases": []},
-        {"name": "Guardian Squad", "system": "Warhammer 40,000",
-         "group": "Xenos Armies", "faction": "Aeldari",
-         "class": "Battleline", "aliases": []},
-        {"name": "Rangers", "system": "Warhammer 40,000",
-         "group": "Xenos Armies", "faction": "Aeldari",
-         "class": "Battleline", "aliases": []},
-        {"name": "Dark Reapers", "system": "Warhammer 40,000",
-         "group": "Xenos Armies", "faction": "Aeldari",
-         "class": "Aspect Warriors", "aliases": []},
-        {"name": "Mordor Orcs", "system": "Middle-Earth",
-         "group": "The Lord of the Rings™ - Evil", "faction": "Mordor",
-         "class": "Warriors", "aliases": []},
-    ]
-}
-
-
-def make_lib():
-    return Path(tempfile.mkdtemp(prefix="sorter_test_lib_"))
-
-
-def write(p: Path):
-    p.parent.mkdir(parents=True, exist_ok=True)
-    p.write_bytes(b"x")
-
-
-_LOADED = None
-
-
-def get_catalog():
-    global _LOADED
-    if _LOADED is None:
-        _LOADED = load_catalog_path()
-    return _LOADED
-
-
-def prov_items(root: Path, library):
-    (root / "extract_manifest.jsonl").write_text(json.dumps({
-        "archive": str(root / "Multi Pack.zip"),
-        "dest": "Multi Pack", "unit": "Multi Pack.zip",
-    }), encoding="utf-8")
-    files = sorted((root / "Multi Pack").rglob("*"))
-    return sort_plan(files, library, get_catalog(), provenance={
-        "Multi Pack": "Multi Pack.zip"}, root=root)
-
-
-def test_prov_dir_rehome():
-    """A matching inner dir name claims its subtree; junk dirs don't."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    pack = root / "Multi Pack"
-    write(pack / "14 Warp spiders" / "spider_body.stl")
-    write(pack / "14 Warp spiders" / "files" / "spider_arm.stl")
-    write(pack / "HEADS" / "head_a.stl")          # junk dir: stays with pack
-    write(pack / "Retro GrimDark Thing" / "t.stl")  # no catalog match
-    library = make_lib()
-    items = prov_items(root, library)
-    tgt = {i.name: i.target for i in items}
-    lib = str(library)
-    # warp spiders re-homed, incl. nested junk dir structure, class from catalog
-    assert "Warp Spiders" in str(tgt["spider_body.stl"]), tgt["spider_body.stl"]
-    assert "Fly" in str(tgt["spider_body.stl"]), tgt["spider_body.stl"]
-    assert "Hero" not in str(tgt["spider_body.stl"])
-    assert "files" in str(tgt["spider_arm.stl"])
-    # junk-dir file rides along with the pack's adopted unit
-    assert "Warp Spiders" in str(tgt["head_a.stl"])
-    assert "HEADS" in str(tgt["head_a.stl"])
-    # unmatched subtree keeps the pack unit folder
-    assert "Warp Spiders" in str(tgt["t.stl"]) and "Retro GrimDark Thing" in str(tgt["t.stl"])
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_prov_weak_tier():
-    """Below-threshold dir hit is accepted as low confidence."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    # pack name carries a keyword so the probe resolves system+faction -
-    # only then does the same-faction weak tier apply
-    pack = root / "Aeldari Pack"
-    write(pack / "guardian weapon team" / "g.stl")  # 0.59 vs Guardian Squad
-    library = make_lib()
-    (root / "extract_manifest.jsonl").write_text(json.dumps({
-        "archive": str(root / "Aeldari Pack.zip"),
-        "dest": "Aeldari Pack", "unit": "Aeldari Pack.zip",
-    }), encoding="utf-8")
-    files = sorted((root / "Aeldari Pack").rglob("*"))
-    items = sort_plan(files, library, get_catalog(), provenance={
-        "Aeldari Pack": "Aeldari Pack.zip"}, root=root)
-    it = next(i for i in items if i.name == "g.stl")
-    assert it.match_kind == "weak", (it.match_kind, it.confidence)
-    assert it.confidence == "low"
-    assert "Guardian" in str(it.target)
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_cross_system_needs_token_containment():
-    """'Aeldari Rangers' must not adopt Middle-earth 'Dwarf Rangers'."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    pack = root / "MQVxKzh"  # hash name: system stays unknown
-    write(pack / "Aeldari Rangers" / "Luminary - Arms -  (1).stl")
-    write(pack / "14 Warp spiders" / "sp.stl")
-    library = make_lib()
-    files = sorted((root / "MQVxKzh").rglob("*"))
-    items = sort_plan(files, library, get_catalog(), provenance={
-        "MQVxKzh": "MQVxKzh.zip"}, root=root)
-    by_name = {i.name: i for i in items}
-    lum = by_name["Luminary - Arms -  (1).stl"]
-    assert "Middle-Earth" not in str(lum.target), lum.target
-    spider = by_name["sp.stl"]
-    assert "Warp Spiders" in str(spider.target), spider.target
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_weak_needs_shared_token():
-    """Sub-threshold hit with zero shared words is a fluke: reject.
-
-    'frontline squad' vs 'Guardian Squad' scores 0.552 but shares no
-    token - it must NOT re-home, while 'guardian weapon team' (0.588,
-    shares 'guardian') is accepted weak.
-    """
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    pack = root / "Aeldari Pack"
-    write(pack / "frontline squad" / "f.stl")
-    write(pack / "guardian weapon team" / "g.stl")
-    library = make_lib()
-    (root / "extract_manifest.jsonl").write_text(json.dumps({
-        "archive": str(root / "Aeldari Pack.zip"),
-        "dest": "Aeldari Pack", "unit": "Aeldari Pack.zip",
-    }), encoding="utf-8")
-    files = sorted((root / "Aeldari Pack").rglob("*"))
-    items = sort_plan(files, library, get_catalog(), provenance={
-        "Aeldari Pack": "Aeldari Pack.zip"}, root=root)
-    by_name = {i.name: i for i in items}
-    f = by_name["f.stl"]
-    # both files ride along with the pack-level weak adoption
-    assert f.match_kind == "weak" and f.confidence == "low", (f.match_kind, f.target)
-    assert "frontline squad" in str(f.target)
-    g = by_name["g.stl"]
-    assert g.match_kind == "weak" and g.confidence == "low"
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_tokenless_needs_bonus():
-    """Same-faction adoption without a shared word needs +0.08.
-
-    'Aeldari Rangers' (0.667 vs 'Dark Reapers', tokenless) must not adopt
-    Dark Reapers; it instead adopts 'Rangers' (0.636, shares 'ranger').
-    """
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    pack = root / "Aeldari Pack"
-    write(pack / "Aeldari Rangers" / "r.stl")
-    library = make_lib()
-    (root / "extract_manifest.jsonl").write_text(json.dumps({
-        "archive": str(root / "Aeldari Pack.zip"),
-        "dest": "Aeldari Pack", "unit": "Aeldari Pack.zip",
-    }), encoding="utf-8")
-    files = sorted((root / "Aeldari Pack").rglob("*"))
-    items = sort_plan(files, library, get_catalog(), provenance={
-        "Aeldari Pack": "Aeldari Pack.zip"}, root=root)
-    it = next(i for i in items if i.name == "r.stl")
-    assert it.match_kind == "catalog" and it.unit == "Rangers", \
-        (it.match_kind, it.unit, it.match_score)
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_dropped_system_mordor():
-    """Middle-earth is dropped: a 'Mordor' pack lands in _unsorted, and
-    its inner dirs cannot adopt the (filtered) Mordor Orcs entry."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    write(root / "Mordor pack" / "boy.stl")
-    library = make_lib()
-    items = sort_plan([root / "Mordor pack" / "boy.stl"], library,
-                      get_catalog(), provenance={"Mordor pack": "Mordor pack.zip"},
-                      root=root)
-    it = items[0]
-    assert "Middle-Earth" not in str(it.target), it.target
-    assert "_unsorted/Mordor pack" in str(it.target).replace("\\", "/")
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_load_catalog_filters_dropped():
-    cat = load_catalog_path()
-    assert not any(e["system"] == "Middle-Earth" for e in cat)
-    assert any(e["name"] == "Warp Spiders" for e in cat)
-
-
-def test_nonprov_rehome():
-    """Loose noise file inside a named dir re-homes to the dir's unit."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    write(root / "Warp Spiders" / "body.stl")
-    library = make_lib()
-    files = sorted(root.rglob("*"))
-    items = sort_plan(files, library, get_catalog(), root=root)
-    it = next(i for i in items if i.name == "body.stl")
-    assert it.match_kind == "catalog", (it.match_kind, it.target)
-    assert "Warp Spiders" in str(it.target) and "Fly" in str(it.target)
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def test_apply_weak_doc():
-    """apply_plan writes the low-confidence review document."""
-    root = Path(tempfile.mkdtemp(prefix="sorter_test_src_"))
-    pack = root / "Multi Pack"
-    write(pack / "guardian squad extras" / "g.stl")
-    library = make_lib()
-    items = prov_items(root, library)
-    linked, copied, skipped = apply_plan(items, library / "sorter_applied.jsonl")
-    doc = library / "sorter_low_confidence.txt"
-    kinds = [i.match_kind for i in items if i.match_kind == "weak"]
-    if kinds:
-        assert doc.exists(), "weak doc missing"
-        assert "low-confidence" in doc.read_text(encoding="utf-8")
-    else:
-        assert not doc.exists()
-    assert (library / "sorter_applied.jsonl").exists()
-    shutil.rmtree(root); shutil.rmtree(library)
-
-
-def load_catalog_path():
-    import tempfile as tf
-    fh = tf.NamedTemporaryFile(mode="w", suffix=".json", delete=False)
-    json.dump(CATALOG, fh)
-    fh.close()
-    return load_catalog(Path(fh.name))
-
-
-def main():
-    catalog = load_catalog_path()
-    for fn in (test_prov_dir_rehome, test_prov_weak_tier, test_nonprov_rehome,
-               test_cross_system_needs_token_containment, test_apply_weak_doc,
-               test_weak_needs_shared_token, test_tokenless_needs_bonus,
-               test_dropped_system_mordor, test_load_catalog_filters_dropped):
-        fn()
-        print("OK", fn.__name__)
-
-
-if __name__ == "__main__":
-    main()

version-python/tools/debug_gofile.py

@@ -1,68 +0,0 @@
-#!/usr/bin/env python3
-"""One-off diagnostic: inspect the raw gofile API response for one content id.
-
-Prints everything except the account token. Creates exactly one guest account.
-"""
-
-import hashlib
-import json
-import sys
-import time
-
-import requests
-
-CID = sys.argv[1]
-
-USER_AGENT = "Mozilla/5.0"
-LOCALE = "en-US"
-TOKEN_SECRET = "12af056dacea0b"
-WINDOW_SEC = 14400
-COMMON = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip", "Accept": "*/*",
-          "Connection": "keep-alive"}
-
-resp = requests.post("https://api.gofile.io/accounts", headers=COMMON, timeout=15)
-data = resp.json()
-assert data.get("status") == "ok", data
-token = data["data"]["token"]
-
-window = int(time.time()) // WINDOW_SEC
-seed = f"{USER_AGENT}::{LOCALE}::{token}::{window}::{TOKEN_SECRET}"
-wtoken = hashlib.sha256(seed.encode()).hexdigest()
-headers = {
-    **COMMON,
-    "Authorization": f"Bearer {token}",
-    "X-Website-Token": wtoken,
-    "X-BL": LOCALE,
-}
-
-r = requests.get(
-    f"https://api.gofile.io/contents/{CID}?cache=true&sortField=createTime&sortDirection=1",
-    headers=headers, timeout=30,
-)
-print("HTTP", r.status_code)
-body = r.json()
-print("api status:", body.get("status"))
-
-
-def scrub(node):
-    """Remove anything token-like from the tree before printing."""
-    if isinstance(node, dict):
-        return {
-            k: ("<redacted>" if k in ("token", "accountToken") else scrub(v))
-            for k, v in node.items()
-        }
-    if isinstance(node, list):
-        return [scrub(x) for x in node]
-    return node
-
-
-d = body.get("data") or {}
-print("data keys:", sorted(d.keys()))
-print("type:", d.get("type"))
-children = d.get("children") or {}
-print("children count:", len(children))
-for cid, child in list(children.items())[:5]:
-    print("  child:", cid, "type:", child.get("type"), "name:", child.get("name"),
-          "has link:", "link" in child)
-print()
-print(json.dumps(scrub(body), indent=2)[:3000])

version-python/tools/extract_all.py

@@ -1,379 +0,0 @@
-#!/usr/bin/env python3
-"""Recursively extract every archive under the source roots into one tree.
-
-    python3 tools/extract_all.py <src_root>... --out <extracted_root>
-
-Behavior:
-- Every archive (zip/7z/rar/tar/gz/bz2/xz) is extracted into a directory
-  named after the archive (suffix stripped), preserving the relative path
-  under the source root:  <src>/zips/The Twin Lance.7z
-      -> <out>/1/zips/The Twin Lance/...
-- Archives found *inside* an extraction are extracted too (BFS, until no
-  archive remains), then deleted - the extracted tree holds no archives.
-- Plain files are hardlinked into the tree (fallback: copy), so nothing
-  is lost and no data is duplicated.
-- The source roots are never modified (originals stay).
-- A provenance manifest `<out>/extract_manifest.jsonl` records one line
-  per extraction: {"archive": ..., "dest": ..., "unit": ...} for the
-  sorter's --provenance unit mode.
-- Failures (bad download/HTML page, wrong password, disk full) are moved
-  to `<out>/.broken/<relpath>` and logged; the run never stops.
-
-Re-runs are safe: destinations recorded in the manifest are skipped (a
-partial dir from a crash is NOT skipped - it gets re-extracted); a final
-sweep expands archives stranded by interrupted nested expansions.
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-import os
-import shutil
-import subprocess
-import sys
-import tempfile
-from collections import deque
-from pathlib import Path
-
-log = logging.getLogger("extract_all")
-
-ARCHIVE_SUFFIXES = {".zip", ".7z", ".rar", ".tar", ".gz", ".bz2", ".xz"}
-EXTRACT_TIMEOUT = 4 * 3600  # seconds per archive
-MIN_FREE_BYTES = 20 * 1024**3  # stop extracting below this free space
-RUN_ARTIFACTS = {
-    "export.json", "export.html", "state.json", "manifest.jsonl",
-    "failed_urls.txt", "catalogue.md", "catalogue.jsonl",
-    "sorter_applied.jsonl", "catalogue_moves.jsonl", "urls.txt",
-    "urls_0807.txt", "urls_gofile.txt", "failed_0807.txt",
-    "extract_manifest.jsonl",
-}
-
-
-def archive_suffixes(name: str) -> list[str]:
-    """All trailing archive suffixes, e.g. 'x.tar.gz' -> ['.gz', '.tar']."""
-    out = []
-    p = Path(name)
-    while p.suffix.lower() in ARCHIVE_SUFFIXES:
-        out.append(p.suffix.lower())
-        p = p.with_suffix("")
-    return out
-
-
-def is_archive(path: Path) -> bool:
-    return bool(archive_suffixes(path.name))
-
-
-def display_unit(name: str) -> str:
-    """Archive name with all archive suffixes stripped: the unit folder."""
-    p = Path(name)
-    while p.suffix.lower() in ARCHIVE_SUFFIXES:
-        p = p.with_suffix("")
-    return p.name
-
-
-def free_bytes(path: Path) -> int:
-    return shutil.disk_usage(path).free
-
-
-def _fix_component(s: str) -> str:
-    """Repair filenames 7z wrote with raw legacy bytes (cp437 zip entries).
-
-    Raw non-UTF8 bytes in a name cannot be created on the CIFS share
-    (observed: errno 2 for a name containing byte 0x87). Decode the raw
-    bytes as cp437 so e.g. a raw-0x87 name becomes a proper 'ç' name.
-    """
-    try:
-        s.encode("utf-8")
-        return s
-    except UnicodeEncodeError:
-        pass
-    raw = os.fsencode(s)
-    try:
-        return raw.decode("utf-8")
-    except UnicodeDecodeError:
-        return raw.decode("cp437", "replace")
-
-
-def _move_tree(tmp: Path, dest: Path) -> int:
-    """Move a freshly extracted local tree onto the (possibly CIFS) dest.
-    Returns the number of files that could not be moved."""
-    bad = 0
-    for f in sorted(tmp.rglob("*")):
-        rel = f.relative_to(tmp)
-        tgt = dest.joinpath(*(_fix_component(p) for p in rel.parts))
-        if f.is_dir():
-            tgt.mkdir(parents=True, exist_ok=True)
-            continue
-        tgt.parent.mkdir(parents=True, exist_ok=True)
-        try:
-            if tgt.exists() and tgt.is_file() \
-                    and tgt.stat().st_size == f.stat().st_size:
-                f.unlink()
-                continue
-            if tgt.exists():
-                tgt.unlink()
-            shutil.move(str(f), str(tgt))
-        except OSError as exc:
-            bad += 1
-            log.error("Cannot place %s: %s", tgt, exc)
-    return bad
-
-
-def _run_7z(cmd: list[str]) -> tuple[int, str]:
-    try:
-        # 7z prints legacy-encoded inner filenames (non-UTF8 bytes); decode
-        # tolerantly or the whole run crashes (seen with byte 0x87).
-        proc = subprocess.run(
-            cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
-            encoding="utf-8", errors="replace", timeout=EXTRACT_TIMEOUT,
-        )
-    except subprocess.TimeoutExpired:
-        return -1, f"extraction timed out after {EXTRACT_TIMEOUT}s"
-    except OSError as exc:
-        return -1, f"7z failed to run: {exc}"
-    tail = "\n".join((proc.stdout or "").splitlines()[-5:])
-    return proc.returncode, tail
-
-
-def extract(archive: Path, dest: Path) -> tuple[bool, str, Path]:
-    """7z x into dest. Empty -p avoids interactive prompts (no password).
-
-    Returns (ok, error, dest_used): on a name collision (the archive stem
-    exists as a *file*, e.g. 'model.stl.zip' next to 'model.stl') the
-    destination gets a '_dup' suffix instead of crashing.
-
-    If 7z cannot write directly onto the (CIFS) destination - legacy
-    cp437 entry names - the archive is extracted to a local temp dir and
-    moved over with repaired filenames.
-    """
-    actual = dest
-    for _ in range(5):
-        try:
-            actual.mkdir(parents=True, exist_ok=True)
-            break
-        except FileExistsError:
-            actual = dest.with_name(dest.name + "_dup")
-        except NotADirectoryError as exc:
-            return False, f"cannot create destination {actual}: {exc}", dest
-    else:
-        return False, f"cannot create destination {dest} (repeated collisions)", dest
-    rc, tail = _run_7z(["7z", "x", "-y", "-p", f"-o{actual}", str(archive)])
-    if rc in (0, 1):  # 0 ok, 1 warnings (e.g. minor CRC)
-        if rc == 1:
-            log.warning("7z reported warnings extracting %s", archive.name)
-        return True, "", actual
-    if rc == -1:
-        return False, tail, actual
-    # Fallback: local extraction, then move with repaired filenames.
-    tmp = Path(tempfile.mkdtemp(prefix="7zext-", dir="/var/tmp"))
-    try:
-        rc2, tail2 = _run_7z(["7z", "x", "-y", "-p", f"-o{tmp}", str(archive)])
-        if rc2 not in (0, 1):
-            return False, f"7z exit {rc2}: {tail2}", actual
-        bad = _move_tree(tmp, actual)
-        if bad:
-            return False, f"{bad} files could not be moved onto the share", actual
-        return True, "", actual
-    finally:
-        shutil.rmtree(tmp, ignore_errors=True)
-
-
-def move_to_broken(path: Path, out_root: Path, src_root: Path, note: str) -> Path:
-    rel = path.relative_to(src_root)
-    dest = out_root / ".broken" / rel
-    dest.parent.mkdir(parents=True, exist_ok=True)
-    if dest.exists():
-        dest = dest.with_name(dest.name + ".1")
-    return shutil.move(str(path), str(dest))
-
-
-def is_macosx_junk(p: Path) -> bool:
-    """AppleDouble resource-fork 'zips' (.__MACOSX/._x.zip) - no content."""
-    return "__MACOSX" in p.parts or p.name.startswith("._")
-
-
-def expand_nested(dest: Path, out_root: Path, manifest, src_root: Path,
-                  done: set[str]) -> int:
-    """Extract archives inside `dest` until none remain. Returns count."""
-    done_count = 0
-    queue: deque[Path] = deque([dest])
-    while queue:
-        cur = queue.popleft()
-        for p in sorted(cur.rglob("*")):
-            if not p.is_file() or not is_archive(p):
-                continue
-            if is_macosx_junk(p):
-                p.unlink()
-                log.info("Junk (macOS AppleDouble) removed: %s", p.name)
-                continue
-            stem_dir = p.with_name(display_unit(p.name))
-            if str(stem_dir.relative_to(out_root)) in done:
-                # already extracted in a previous run; drop the archive copy
-                p.unlink()
-                log.info("Nested %s already extracted - archive removed", p.name)
-                continue
-            if free_bytes(out_root) < MIN_FREE_BYTES:
-                log.error("Out of disk space - skipping %s (kept in place)", p)
-                continue
-            ok, err, actual = extract(p, stem_dir)
-            if not ok:
-                log.error("FAILED nested extraction %s: %s", p, err)
-                move_to_broken(p, out_root, out_root, err)
-                continue
-            manifest.write(f"{json_line(p.relative_to(out_root), actual)}\n")
-            done.add(str(actual.relative_to(out_root)))
-            p.unlink()  # archive copy consumed
-            done_count += 1
-            queue.append(actual)
-    return done_count
-
-
-def json_line(archive_rel: Path, dest: Path) -> str:
-    import json
-    return json.dumps({
-        "archive": str(archive_rel),
-        "dest": str(dest.relative_to(_OUT_ROOT)),
-        "unit": display_unit(archive_rel.name),
-    }, ensure_ascii=False)
-
-
-_OUT_ROOT: Path = Path(".")  # set in main; used by json_line
-
-
-def process_root(src_root: Path, out_root: Path, manifest, done: set[str]) -> tuple[int, int, int]:
-    """Returns (extracted, mirrored, failed)."""
-    idx = src_root.name
-    extracted = mirrored = failed = 0
-    for f in sorted(src_root.rglob("*")):
-        if not f.is_file():
-            continue
-        if (f.name in RUN_ARTIFACTS or f.name.startswith(".")
-                or f.name.endswith(".part")):
-            continue
-        rel = f.relative_to(src_root)
-        if is_archive(f):
-            dest_rel = Path(idx) / Path(str(rel.with_name(display_unit(f.name))))
-            dest = out_root / dest_rel
-            # Skip on a manifest record, never on "dir non-empty": a crash
-            # mid-extraction leaves a partial dir that must be redone.
-            if str(dest_rel) in done:
-                log.info("Skip (already extracted): %s", rel)
-                continue
-            if free_bytes(out_root) < MIN_FREE_BYTES:
-                log.error("Out of disk space - skipping archive %s", f)
-                failed += 1
-                continue
-            ok, err, actual = extract(f, dest)
-            if not ok:
-                log.error("FAILED %s: %s", rel, err)
-                move_to_broken(f, out_root, src_root, err)
-                failed += 1
-                continue
-            manifest.write(f"{json_line(Path(idx) / rel, actual)}\n")
-            done.add(str(actual.relative_to(out_root)))
-            extracted += 1
-            log.info("Extracted %s -> %s", rel, actual.relative_to(out_root))
-            expand_nested(actual, out_root, manifest, src_root, done)
-        else:
-            link_dest = out_root / idx / rel
-            if link_dest.exists():
-                continue
-            link_dest.parent.mkdir(parents=True, exist_ok=True)
-            try:
-                os.link(f, link_dest)
-            except OSError:
-                shutil.copy2(f, link_dest)
-            mirrored += 1
-    return extracted, mirrored, failed
-
-
-def load_done(manifest_path: Path) -> set[str]:
-    """Dest dirs already recorded in the manifest (relative to out root)."""
-    import json
-    done: set[str] = set()
-    if manifest_path.exists():
-        with manifest_path.open(encoding="utf-8") as fh:
-            for line in fh:
-                line = line.strip()
-                if not line:
-                    continue
-                try:
-                    done.add(json.loads(line)["dest"])
-                except (json.JSONDecodeError, KeyError):
-                    continue
-    return done
-
-
-def sweep(out_root: Path, manifest, done: set[str]) -> int:
-    """Expand any archive left anywhere in the tree (interrupted nested
-    expansions leave stranded copies even when the parent has a record)."""
-    import json
-    swept = 0
-    for p in sorted(out_root.rglob("*")):
-        if not p.is_file() or not is_archive(p):
-            continue
-        if ".broken" in p.relative_to(out_root).parts:
-            continue
-        if is_macosx_junk(p):
-            p.unlink()
-            log.info("Junk (macOS AppleDouble) removed: %s", p.name)
-            continue
-        stem_dir = p.with_name(display_unit(p.name))
-        if str(stem_dir.relative_to(out_root)) in done:
-            p.unlink()
-            log.info("Sweep: %s already extracted - archive removed", p.name)
-            continue
-        if free_bytes(out_root) < MIN_FREE_BYTES:
-            log.error("Out of disk space - sweep skipping %s", p)
-            continue
-        ok, err, actual = extract(p, stem_dir)
-        if not ok:
-            log.error("FAILED sweep extraction %s: %s", p, err)
-            move_to_broken(p, out_root, out_root, err)
-            continue
-        manifest.write(f"{json_line(p.relative_to(out_root), actual)}\n")
-        done.add(str(actual.relative_to(out_root)))
-        p.unlink()
-        swept += 1
-        expand_nested(actual, out_root, manifest, out_root, done)
-    return swept
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    parser.add_argument("sources", nargs="+", type=Path)
-    parser.add_argument("--out", type=Path, required=True, metavar="DIR")
-    args = parser.parse_args()
-
-    logging.basicConfig(level=logging.INFO,
-                        format="%(asctime)s %(levelname)-7s %(message)s",
-                        datefmt="%H:%M:%S")
-
-    global _OUT_ROOT
-    _OUT_ROOT = args.out
-    args.out.mkdir(parents=True, exist_ok=True)
-
-    totals = [0, 0, 0]
-    done = load_done(args.out / "extract_manifest.jsonl")
-    log.info("%d destinations already recorded in manifest", len(done))
-    with (args.out / "extract_manifest.jsonl").open("a", encoding="utf-8") as manifest:
-        for src in args.sources:
-            log.info("== processing %s", src)
-            counts = process_root(src, args.out, manifest, done)
-            log.info("== %s: %d extracted, %d mirrored, %d failed",
-                     src.name, *counts)
-            for i, c in enumerate(counts):
-                totals[i] += c
-        swept = sweep(args.out, manifest, done)
-    log.info("TOTAL: %d archives extracted, %d files mirrored, %d failed, %d swept",
-             *totals, swept)
-    if totals[2]:
-        log.warning("Failures were moved under %s - inspect and re-run", args.out / ".broken")
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except KeyboardInterrupt:
-        sys.exit(130)

version-python/tools/gw_browse_test.py

@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-"""One-shot browse of the GW Algolia index -> gwshop_raw.json (gitignored)."""
-import json
-import logging
-import pathlib
-import sys
-
-sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))
-
-import dlphelper.gwshop as gw
-
-logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-
-app, key = gw.load_credentials()
-client = gw.GwShopClient(app, key)
-hits = client.browse("prod-lazarus-product-en-ww", hits_per_page=100)
-print("total hits", len(hits))
-pathlib.Path("gwshop_raw.json").write_text(
-    json.dumps(hits, ensure_ascii=False), encoding="utf-8"
-)
-print("written gwshop_raw.json")

version-python/tools/gw_build_catalog.py

@@ -1,42 +0,0 @@
-#!/usr/bin/env python3
-"""gwshop_raw.json -> data/gwshop_catalog.json + quick shape report."""
-import json
-import logging
-import pathlib
-import sys
-from collections import Counter
-
-sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))
-
-import dlphelper.gwshop as gw
-
-logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-
-hits = json.loads(pathlib.Path("gwshop_raw.json").read_text(encoding="utf-8"))
-print("raw hits:", len(hits))
-unique = len({h.get("objectID") for h in hits})
-print("unique objectIDs:", unique)
-
-catalog = gw.build_catalog(hits)
-pathlib.Path("data/gwshop_catalog.json").write_text(
-    json.dumps(catalog, ensure_ascii=False, indent=1), encoding="utf-8"
-)
-print("entries:", catalog["generated_entries"])
-
-entries = catalog["entries"]
-print("no system:", sum(1 for e in entries if e["system"] is None))
-print("by system:", Counter(e["system"] for e in entries).most_common())
-print("40k groups:", Counter(
-    e["group"] for e in entries if e["system"] == "Warhammer 40,000"
-).most_common())
-print("lvl depth:", Counter(
-    len((e.get("gw_paths") or [[]])[0]) for e in entries
-).most_common())
-print("classes (lvl>=4):", Counter(
-    e["class"] for e in entries if e["class"]
-).most_common(12))
-sample = next(
-    e for e in entries
-    if e["name"].startswith("Codex: Adeptus Custodes")
-)
-print(json.dumps(sample, indent=1, ensure_ascii=False)[:900])

version-python/tools/gw_inspect.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python3
-"""Inspect gwshop catalog shape: Space Marines chapters, AoS, dedupe stats."""
-import json
-import pathlib
-from collections import Counter
-
-cat = json.loads(pathlib.Path("data/gwshop_catalog.json").read_text(encoding="utf-8"))
-entries = cat["entries"]
-
-def show(pred, label, n=6):
-    print(f"--- {label}")
-    shown = 0
-    for e in entries:
-        if pred(e):
-            p = " > ".join(x for x in (e["system"], e["group"], e["faction"], e["subfaction"], e["class"]) if x)
-            print(f"  {e['name'][:60]!r}  ->  {p}")
-            shown += 1
-            if shown >= n:
-                break
-
-show(lambda e: e["group"] == "Space Marines", "Space Marines group samples")
-show(lambda e: e["group"] == "Armies of the Imperium" and e["subfaction"], "Imperium with subfaction")
-show(lambda e: e["system"] == "Age of Sigmar", "Age of Sigmar samples", 5)
-show(lambda e: e["system"] == "The Old World", "Old World samples", 5)
-show(lambda e: e["system"] == "Other Games", "Other Games samples", 5)
-show(lambda e: e["group"] == "Unit Type", "Unit Type branch", 6)
-
-names = Counter(e["name"].casefold() for e in entries)
-dups = {n: c for n, c in names.items() if c > 1}
-print("duplicate names:", len(dups), "example:", list(dups.items())[:3])
-
-# faction-level value census for 40k
-print("--- 40k factions census")
-print(Counter(e["faction"] for e in entries if e["system"] == "Warhammer 40,000").most_common(40))

version-python/tools/new_repo.py

@@ -1,108 +0,0 @@
-#!/usr/bin/env python3
-"""Create or list repositories on the Gitea instance at git.dieterunterstrich.de.
-
-Credentials are read from ~/.git-credentials (the line for that host); the
-secret is never printed. Useful when you need a new repo without touching the
-web UI.
-
-Usage:
-    python3 tools/new_repo.py list
-    python3 tools/new_repo.py create <name> [--private] [--desc "..."]
-"""
-
-from __future__ import annotations
-
-import argparse
-import base64
-import json
-import sys
-from pathlib import Path
-from urllib.request import Request, urlopen
-
-HOST = "git.dieterunterstrich.de"
-API = f"https://{HOST}/api/v1"
-CREDENTIALS_FILE = Path.home() / ".git-credentials"
-
-
-def load_credentials() -> tuple[str, str]:
-    """Return (username, password) for the Gitea host from ~/.git-credentials."""
-    for line in CREDENTIALS_FILE.read_text().splitlines():
-        line = line.strip()
-        if not line.startswith("https://") or HOST not in line:
-            continue
-        # https://user:pass@host
-        rest = line.removeprefix("https://")
-        userinfo = rest.split("@", 1)[0]
-        user, _, password = userinfo.partition(":")
-        return user, password
-    raise SystemExit(f"No credentials for {HOST} in {CREDENTIALS_FILE}")
-
-
-def api(method: str, path: str, body: dict | None = None) -> tuple[int, dict]:
-    """Call the Gitea API; returns (status_code, parsed_json)."""
-    user, password = load_credentials()
-    auth = base64.b64encode(f"{user}:{password}".encode()).decode()
-    data = json.dumps(body).encode() if body is not None else None
-    req = Request(
-        f"{API}{path}",
-        data=data,
-        method=method,
-        headers={
-            "Authorization": f"Basic {auth}",
-            "Content-Type": "application/json",
-        },
-    )
-    try:
-        resp = urlopen(req, timeout=30)
-        return resp.status, json.loads(resp.read() or b"{}")
-    except Exception as exc:  # noqa: BLE001
-        # HTTPError carries the response body, useful for API errors
-        code = getattr(exc, "code", None)
-        if code is not None:
-            try:
-                return code, json.loads(exc.read() or b"{}")
-            except Exception:  # noqa: BLE001
-                return code, {"message": str(exc)}
-        raise
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(description="Gitea repo helper")
-    sub = parser.add_subparsers(dest="command", required=True)
-
-    sub.add_parser("list")
-
-    p_create = sub.add_parser("create")
-    p_create.add_argument("name")
-    p_create.add_argument("--desc", default=None)
-    p_create.add_argument("--public", action="store_true")
-
-    args = parser.parse_args()
-
-    if args.command == "list":
-        status, repos = api("GET", "/user/repos?limit=100")
-        print(f"HTTP {status}")
-        for repo in repos:
-            print(f"  {repo['full_name']}  (private={repo['private']})")
-        return
-
-    status, result = api(
-        "POST",
-        "/user/repos",
-        {
-            "name": args.name,
-            "description": args.desc or "",
-            "private": not args.public,
-            "auto_init": False,
-            "default_branch": "main",
-        },
-    )
-    if status in (201, 200):
-        print(f"Created: {result['full_name']} -> {result['clone_url']}")
-    else:
-        print(f"Failed (HTTP {status}): {result.get('message', result)}")
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()

version-python/tools/plan_diff.py

@@ -1,26 +0,0 @@
-#!/usr/bin/env python3
-"""Diff old vs new sorter plans by source path -> target."""
-import json
-import sys
-
-changed = 0
-same = 0
-examples = []
-for n in (1, 2, 3):
-    old = {p["source"]: p for p in json.load(open(f"/tmp/plan{n}.json"))}
-    new = {p["source"]: p for p in json.load(open(f"/tmp/plan{n}b.json"))}
-    for src, np in new.items():
-        op = old.get(src)
-        if not op:
-            continue
-        if op["target"] != np["target"] or op["kind"] != np["kind"]:
-            changed += 1
-            if len(examples) < 40:
-                examples.append((n, op, np))
-        else:
-            same += 1
-print(f"same: {same}  changed: {changed}")
-for n, op, np in examples:
-    print(f"\n[{n}] {op['name']}  (score {op['score']} -> {np['score']})")
-    print(f"  old: {op['target']}")
-    print(f"  new: {np['target']}")

version-python/tools/plan_summary.py

@@ -1,22 +0,0 @@
-#!/usr/bin/env python3
-"""Summarize a sorter plan JSON: match kinds, top targets, oddities."""
-import json
-import sys
-from collections import Counter
-from pathlib import Path
-
-plan = json.loads(Path(sys.argv[1]).read_text())
-print(f"files: {len(plan)}")
-print("kinds:", Counter(p["kind"] for p in plan))
-print("actions:", Counter(p.get("action", "?") for p in plan))
-# system breakdown
-print("systems:", Counter(p["system"] for p in plan).most_common(10))
-# top-level targets
-roots = Counter(str(Path(p["target"]).relative_to(
-    Path(p["target"]).parents[3])) if len(Path(p["target"]).parents) > 3
-    else p["target"] for p in plan)
-for root, n in roots.most_common(15):
-    print(f"  {n:5d}  {root}")
-# anything with empty target?
-empties = [p for p in plan if not p.get("target")]
-print("empty targets:", len(empties))

version-rust/.gitignore

@@ -1 +0,0 @@
-/target

version-rust/tools/dump_warhammer_data.py

@@ -1,29 +0,0 @@
-#!/usr/bin/env python3
-"""Regenerate version-rust/data/warhammer_data.json from the python module.
-
-Run from the repo root (the version containing the python layout):
-
-    python3 version-rust/tools/dump_warhammer_data.py
-
-The Rust rewrite parses this file (src/warhammer_data.rs) instead of holding
-a hand-transcribed copy of the taxonomy tables, so the two versions work
-from byte-identical data.
-"""
-
-import json
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent / "version-python"))
-
-from dlphelper import warhammer_data as w  # noqa: E402
-
-out = Path(__file__).resolve().parent.parent / "data" / "warhammer_data.json"
-
-payload = {
-    "systems": w.SYSTEMS,
-    "classes": [[name, keywords] for name, keywords in w.CLASSES],
-    "group_of_faction": w.GROUP_OF_FACTION,
-}
-out.write_text(json.dumps(payload, ensure_ascii=False, indent=1), encoding="utf-8")
-print(f"wrote {out} ({out.stat().st_size} bytes)")