Warhammer_DLP_Helper/AGENTS.md

AGENTS.md — Warhammer_DLP_Helper

Guidance for any AI agent (or human) working in this repository. Read this fully before making changes.

1. What this project is

The user collects Warhammer proxy models (3rd-party / fan sculpts, never official Games Workshop assets) that are shared in Discord channels. Links in those channels point to two hosters and are often base64-encoded inside message content (obfuscated by the posting bot).

The end goal is a torrent-first platform (think radarr/lidarr, but for proxy files). This repo is the Rust implementation of the current stage: one binary, dlphelper, that

1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. extracts the archives into a plain-file tree,
6. sorts the files into the library by system/group/faction/class/unit,
7. records a manifest that will later feed the torrent stage.

A webapp is the next stage. Deliberately not in scope (yet): .torrent creation, tracker upload, deseeeding, rename-into-library conventions. Design decisions should not close the door on these — especially manifest.jsonl and the --hash option exist for dedupe/cross-seeding later.

This repository was ported from a Python implementation, which the Rust version has now replaced (the ported modules keep Rust port of … provenance notes in their headers).

2. Repository & environment

• Remote: git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper — private repo, branch main. Push with the credentials already stored via git's credential store (~/.git-credentials); never paste them into commands or files.

• Machine: Linux, Rust (edition 2024), docker is available. The repo lives on NFS, where rustc SIGBUSes — always build/test with an explicit target directory:

  CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
  

• Reference clone: ~/GoFileDownloader (Lysagxra) — do not modify it; it is upstream reading material only.

3. Architecture

Cargo.toml                  crate `dlphelper` (edition 2024)
src/main.rs                 CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
src/orchestrate.rs          dlp pipeline: export -> extract URLs -> per-host download
src/discord.rs              DCE docker export (JSON) + base64/plain URL extraction
src/gofile.rs               GoFile API client (one guest account per run)
src/zero807.rs              curl downloader for 0807.st (direct links + /f/<id> folder pages)
src/state.rs                state.json store
src/globalstore.rs          cross-run URL store (~/.local/share/dlphelper/urls.json)
src/manifest.rs             manifest.jsonl appender
src/extract.rs              recursive archive extraction into a plain-file tree
src/catalogue.rs            scanner + keyword classifier + report writer
src/sorter.rs               library sorter: inspect/classify/fuzzy-match, plan + apply
src/difflib.rs              faithful CPython SequenceMatcher port (bit-identical ratios)
src/warhammer_data.rs       taxonomy tables (parsed from data/warhammer_data.json)
src/gwshop.rs               GW webshop catalog dump via Algolia
src/util.rs                 small shared helpers
data/gwshop_catalog.json    committed catalog snapshot (sorter default)
data/warhammer_data.json    committed taxonomy tables
docs/SORTING.md             the library/sorting contract (change it before the code)

Data flow:

graph LR
    A[channel id + token] --> B[DCE docker export -> export.json]
    B --> C[decode base64 in message content]
    C --> D{classify host}
    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
    D -->|0807.st| F[curl with UA + Referer]
    D -->|other| G[recorded as failed: unsupported host]
    E --> H[downloads/gofile/...]
    F --> I[downloads/0807/...]
    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
    J --> K[globalstore: urls.json, marks URL done forever]
    J --> L[extract -> plain-file tree]
    L --> M[sorter -> library, hardlinks + audit]

Key facts about the hosters (learned the hard way, do not re-learn):

• gofile.io: guest accounts are created via POST api.gofile.io/accounts. Creation is aggressively rate-limited — create at most one account per run. Content listing needs Authorization: Bearer <token>, an X-Website-Token header (sha256 over UA::en-US::<token>::<4h-window>::seed, see src/gofile.rs), and downloads need Cookie: accountToken=<token> plus Referer/Origin headers.
• 0807.st: serves files directly but 403s anything that doesn't look like a browser. curl -A "Mozilla/5.0" -e "https://0807.st/" works; aria2c's default multi-connection request does not. curl is the tool here. Folder pages (https://0807.st/f/<id>) are expanded: the page is parsed and each listed file is fetched from https://0807.st/<id>?dl=1 into downloads/0807/<page-id>/. The folder .zip export is refused by the site ("folder too large"), so per-file download is the only route in.
• Encoded links in the channels start as base64 strings matching aHR0cHM6Ly[89]....

4. Usage (what "done" looks like)

export DISCORD_TOKEN=...
dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash

--since 5d (or 12h, 2026-10-04) limits a run to recent messages — handy for incremental updates. Re-runs are safe: state.json and the global store skip everything already marked ok.

Retry only failures:

dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt

Several channels at once (subscriptions.json: list of {name, channel_id, out_dir}):

dlphelper dlp --subs subscriptions.json

Subs mode re-exports every channel by default (that is how new links get picked up) and relies on state.json + the global store to only download what is actually new. Use a cron/systemd timer for watch-like behavior.

Extract and sort:

dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
dlphelper sorter <out>/extracted --library <library> --apply

Manual import of old URL lists (known fully downloaded, never re-fetch):

dlphelper dlp --assume-done old_urls.txt

5. Run artifacts (all in OUT_DIR, gitignored)

File	Purpose
export.json	DCE export; cached — --fresh re-exports
state.json	url → {status, attempts, error, files[]}; corrupt files are backed up and reset
failed_urls.txt	rewritten each run: every non-ok URL, usable as --urls-file
manifest.jsonl	append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256)
downloads/	actual content; .part files mean "interrupted, will be resumed"
extracted/extract_manifest.jsonl	one line per extraction {archive, dest, unit} (feeds sorter --provenance)
~/.local/share/dlphelper/urls.json	global cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. --global-state moves it, --no-global disables it for a run.

Sorter bookkeeping (audit + low-confidence doc) is written to <library>/../_logs/ by default — never inside the library (see §7).

6. Rules

Secrets and tokens

1. Never print, log, commit or echo a Discord token or GoFile account token. Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: --token flag → $DISCORD_TOKEN → hidden prompt. It is handed to the DCE container via the TOKEN env var, never as a command-line argument.
3. ~/.git-credentials may only be read by tooling that needs it; never copy credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.

Failure policy (the core requirement)

5. A failure in one URL or file must never abort the run. Record it in state.json and continue. The per-URL loops in src/orchestrate.rs therefore have catch-alls — keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures, pause, then continue. Circuit-breakers that exit the process need an explicit justification in the commit message.
7. Retries: exponential-ish (retry_delay * attempt), default 3 attempts. Password-required gofile links are not retried (it can't help); dead links (HTTP 404/410) are never retried.

Idempotency

8. Re-runs must be safe and cheap: skip URLs marked ok (in state.json or the global store), skip existing non-empty files, write partial downloads to *.part and rename on success.
9. state.json is the source of truth for resume; keep its schema backward compatible (unknown keys preserved, missing keys defaulted). Same for the global store schema.
10. --assume-done is a deliberate, manual import step for old URL lists. Never invoke it automatically, and never let the tool mark URLs done without a verified successful download.

Rate limiting / network etiquette

11. One gofile guest account per run; ensure_token() backs off linearly (20s × attempt). Do not raise the guest-account frequency "to make tests pass". Testing account creation more than a couple times per hour is a bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without re-verifying the 403 behavior.

Code style

13. Rust 2024, idiomatic. Use log (not println!) in library modules; prints are fine only for CLI output (plan/dry-run).
14. Keep new dependencies justified — the crate already covers HTTP (reqwest), archives (sevenz-rust2, zip, unrar), fuzzy matching (the in-tree difflib port, not an external crate) and CLI (clap). The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
15. src/difflib.rs must stay bit-identical to CPython difflib — the sorter thresholds depend on exact ratios; do not "simplify" it.

Validation

16. Minimum validation for any change: CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test (the offline suite must stay green). Document what you could not test (live gofile/0807/Discord paths need real links and tokens).
17. Full live runs need a Discord token from the user — ask for it, never ask for it to be pasted into a file or committed.

Git

18. Branch main, descriptive commit messages, no force-push, no history rewriting on the shared remote.

7. Sorter & library structure

docs/SORTING.md is the working agreement for the library hierarchy and the sorting pipeline — change it first, then the code. Summary:

• Hierarchy: <library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>. Fully unclassifiable files land in <library>/_unsorted/ — never guessed.
• Matching rules are fixed: the outer file name wins when it scores ≥ threshold; inner archive names only vote as fallback. A fuzzy match never jumps into another game's section (keyword systems constrain the catalog pool; cross-system adoption needs CROSS_SYSTEM_MIN = 0.75 and token containment). Deepest-matching-dir wins (junk dir names are never candidates). Weak hits ≥ WEAK_MIN = 0.50 are accepted but flagged low-confidence. Tokenless normal adoption needs +0.08 (TOKENLESS_BONUS).
• Sorting runs in the sorter's default (non-provenance) mode: every file is classified on its own name (unclassified files land directly in _unsorted/<file>). --provenance (classify whole extraction dirs as units) exists but the live library does not use it; mixing the two modes creates duplicate placements.
• DROPPED_SYSTEMS (default {Middle-Earth}) filters catalog entries at load and counts keyword hits as unclassified; --drop-system/--keep-system adjust. The committed catalog snapshot keeps all systems.
• Catalog source: the GW webshop via Algolia (src/gwshop.rs). The app-id + public search-only key live in the gitignored algolia.json (or ALGOLIA_APP_ID/ALGOLIA_API_KEY); never commit any other Algolia key. data/gwshop_catalog.json is the committed snapshot and the sorter default.
• The dump partitions the index on the GameSystemsRoot hierarchical facet (the search-only key cannot /browse; plain pagination caps at 1000 hits). When a product is faceted under several branches of equal depth (e.g. a generic kit listed per chapter), the first longest path wins — Python's max(paths, key=len) semantics, reproduced explicitly because Rust's max_by_key returns the last maximum. Products carrying no GameSystemsRoot path (paints, Black Library books, White Dwarf, bases, gifts — no miniature kits) are left out; the coverage shortfall is logged.
• Bookkeeping (sorter_applied.jsonl, sorter_low_confidence.txt) is written to <library>/../_logs/ by default — never inside the library, which may be a seeded torrent whose file set is frozen.

8. Testing notes

• Tests live inline (#[cfg(test)]) in each module and run with cargo test. Keep them dependency-free and offline — they must never hit gofile/0807/Discord.
• tempfile tests stage under /var/tmp when available, because /tmp is a small tmpfs that trips the extractor's free-space guard (MIN_FREE_BYTES).
• To exercise downloads without Discord: build a --urls-file from known URLs and run with a scratch OUT_DIR.