chore: start transition layout — python into version-python/, scaffold version-rust/

- git mv of all existing files into version-python/ (history preserved) - version-rust/: minimal cargo scaffold (edition 2024), no deps yet - root AGENTS.md documents the transition rules and artifact/schema contract - main is untouched; the python layout there stays as-is
2026-10-09 13:18:31 +02:00
parent 8f4cc55bb4
commit 271fb3bf68
38 changed files with 385 additions and 305 deletions
@@ -0,0 +1,111 @@
+# Warhammer DLP Helper
+
+A small "radarr for Warhammer proxy files" tool: point it at a Discord
+channel/thread and an output directory, and it exports the channel, extracts
+the encoded download links, and downloads everything - automatically choosing
+the right downloader per host:
+
+| Host        | Downloader | Notes |
+|-------------|------------|-------|
+| gofile.io   | built-in GoFile API client | one guest account per run (rate-limit friendly) |
+| 0807.st     | curl | needs browser-like headers (`-A`, `-e https://0807.st/`) |
+
+## Install
+
+    pip install -r requirements.txt   # just `requests`
+    docker pull tyrrrz/discordchatexporter:stable
+
+## Usage
+
+    export DISCORD_TOKEN="..."
+    python3 dlp.py <channel_or_thread_id> /path/to/output
+
+The token can also be passed with `--token` (or you'll be prompted, input
+hidden). Useful options:
+
+    --urls-file failed_urls.txt   retry a previous failure list, no re-export
+    --password <pw>               for password-protected gofile links
+    --only gofile|0807            process only one host
+    --extract                     unzip archives into per-archive folders
+    --hash                        record sha256 per file in the manifest
+    --dry-run                     just show what would be downloaded
+    --fresh                       force a fresh channel export
+
+## Several channels / watching for updates
+
+Put your channels into `subscriptions.json`:
+
+```json
+{
+  "subscriptions": [
+    {"name": "some-eldar",
+     "channel_id": "1455993943471820842",
+     "out_dir": "/big/storage/warhammer/some-eldar"}
+  ]
+}
+```
+
+then run (e.g. from a daily cron/systemd timer):
+
+    python3 dlp.py --subs subscriptions.json
+
+Subs mode re-exports each channel to pick up new links; `state.json` and the
+global store make sure only genuinely new URLs are downloaded.
+
+## Cataloguing the collection
+
+`catalogue.py` classifies every downloaded file into game system, faction and
+force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
+Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
+taxonomy in `dlphelper/warhammer_data.py`:
+
+    # preview a dlp.py run dir (manifest + downloads)
+    python3 catalogue.py /path/to/run_dir
+
+    # include the manual-era directories
+    python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
+
+    # catalogue arbitrary dirs only
+    python3 catalogue.py --scan /big/storage/manual
+
+    # actually reorganize into a library tree (audited, only classified files)
+    python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
+
+Outputs `catalogue.md` (readable) and `catalogue.jsonl` (machine, with source
+attribution per file). Unclassified files are listed, never guessed; fix them
+with an overrides JSON (`--overrides overrides.json`).
+
+## Already downloaded the old way?
+
+`--assume-done` imports a plain URL list into the global store so those links
+are never fetched again (no verification — use deliberately):
+
+    python3 dlp.py --assume-done old_urls.txt
+
+## Output layout
+
+    OUT_DIR/
+      export.json           Discord channel export (cached between runs)
+      state.json            per-URL status; re-runs skip everything marked "ok"
+      failed_urls.txt       URLs that failed after all retries
+      manifest.jsonl        one line per downloaded file (url, source message, path, size)
+      downloads/
+        gofile/<content>/<folders>/<files>
+        0807/<id>.zip
+| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
+| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
+
+Runs are resumable and idempotent: interrupted downloads use `.part` files and
+finished ones are skipped on the next run. Individual failures never stop the
+run; after 3 consecutive failures it pauses for a cooldown (rate-limit
+protection) and continues.
+
+## Why not use Lysagxra/GoFileDownloader directly?
+
+That tool creates a new GoFile guest account **per URL** and calls
+`sys.exit(1)` on the first failure, which is exactly why runs died after ~6
+downloads. This project reimplements its (working) API scheme - headers,
+`X-Website-Token`, download links with the `accountToken` cookie - but creates
+one guest account per run, records failures per file, and keeps going.
+
+`parts/grab.sh` is the original groundstone script, kept for reference.