Warhammer_DLP_Helper/AGENTS.md

AGENTS.md — Warhammer_DLP_Helper

Guidance for any AI agent (or human) working in this repository. Read this fully before making changes.

1. What this project is

The user collects Warhammer proxy models (3rd-party / fan sculpts, never official Games Workshop assets) that are shared in Discord channels. Links in those channels point to two hosters and are often base64-encoded inside message content (obfuscated by the posting bot).

The end goal is a torrent-first platform (think radarr/lidarr, but for proxy files). This repo is the current stage of that journey: a tool that

1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. records a manifest that will later feed the torrent stage.

Deliberately not in scope (yet): .torrent creation, tracker upload, deseeding, rename-into-library conventions. Design decisions should not close the door on these — especially manifest.jsonl and the --hash option exist for dedupe/cross-seeding later.

2. Repository & environment

• Remote: git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper — private repo, branch main. Push with the credentials already stored via git's credential store (~/.git-credentials); never paste them into commands or files.
• The server is Gitea 1.27. To create another repo, use python3 tools/new_repo.py list|create <name> (reads ~/.git-credentials itself, never prints secrets). Never delete or overwrite existing repos.
• Machine: Linux, Python 3.12+ (3.14 available), requests is the only runtime dependency (see requirements.txt), docker is available.
• Reference clone: ~/GoFileDownloader (Lysagxra) — do not modify it; it is upstream reading material only.

3. Architecture

dlp.py                     CLI orchestrator (entry point)
catalogue.py               CLI: classify the collection into system/faction/class
sorter.py                  CLI: fuzzysort files into the library (plan/apply,
                           hardlinks, default catalog data/gwshop_catalog.json)
dlphelper/
  discord.py               DCE docker export (JSON) + base64/plain URL extraction
  gofile.py                GoFile API client (one guest account per run)
  zero807.py               curl-based downloader for 0807.st direct links
  state.py                 state.json store + manifest.jsonl appender
  globalstore.py           cross-run URL store (~/.local/share/dlphelper/urls.json)
  catalogue.py             scanner + keyword classifier + report writer
  warhammer_data.py        taxonomy: GW-named systems/factions, GW shop
                           categories as classes, GROUP_OF_FACTION map
  gwshop.py                GW webshop catalog dump via Algolia (see §7.3)
  sorter.py                library sorter: inspect/classify/fuzzy-match
                           against the catalog, plan + hardlink apply,
                           optional --provenance unit mode for extracted trees
parts/grab.sh              legacy groundstone script (kept, not used by dlp.py)
tools/new_repo.py          Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py      inspect raw gofile API response for one content id
tools/extract_all.py       recursive archive extraction into a plain-file tree
                           (extract_manifest.jsonl feeds the sorter's
                           --provenance unit mode; failures go to .broken/;
                           7z-direct-to-CIFS failures fall back to a
                           /var/tmp extraction with cp437 name repair;
                           AppleDouble/__MACOSX junk is dropped)
tests/test_extract.py      offline extraction test (plain python3, no deps)
tests/test_catalogue.py    offline classifier test

Data flow:

graph LR
    A[channel id + token] --> B[DCE docker export -> export.json]
    B --> C[decode base64 in message content]
    C --> D{classify host}
    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
    D -->|0807.st| F[curl with UA + Referer]
    D -->|other| G[recorded as failed: unsupported host]
    E --> H[downloads/gofile/...]
    F --> I[downloads/0807/...]
    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
    J --> K[globalstore: urls.json, marks URL done forever]

Key facts about the hosters (learned the hard way, do not re-learn):

• gofile.io: guest accounts are created via POST api.gofile.io/accounts. Creation is aggressively rate-limited — create at most one account per run. Content listing needs Authorization: Bearer <token>, an X-Website-Token header (sha256 over UA::en-US::<token>::<4h-window>::seed, see gofile.py), and downloads need Cookie: accountToken=<token> plus Referer/Origin headers.
• 0807.st: serves files directly but 403s anything that doesn't look like a browser. curl -A "Mozilla/5.0" -e "https://0807.st/" works; aria2c's default multi-connection request does not. curl is the tool here.
• Encoded links in the channels start as base64 strings matching aHR0cHM6Ly[89]... — keep the extraction regex compatible with parts/grab.sh.

4. Usage (what "done" looks like)

export DISCORD_TOKEN=...
python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>

Retry only failures: python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt (or just re-run — state.json skips everything already marked ok).

Several channels at once (subscriptions.json: list of {name, channel_id, out_dir}):

python3 dlp.py --subs subscriptions.json

Subs mode re-exports every channel by default (that is how new links get picked up) and relies on state.json + the global store to only download what is actually new. Use a cron/systemd timer for watch-like behavior.

Manual import of the pre-dlp.py era (URLs you know are fully downloaded, never re-fetch them again):

python3 dlp.py --assume-done old_urls.txt

5. Run artifacts (all in OUT_DIR, gitignored)

File	Purpose
export.json	DCE export; cached — --fresh re-exports
state.json	url → {status, attempts, error, files[]}; corrupt files are backed up and reset
failed_urls.txt	rewritten each run: every non-ok URL, usable as --urls-file
manifest.jsonl	append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256)
downloads/	actual content; .part files mean "interrupted, will be resumed"
~/.local/share/dlphelper/urls.json	global cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. --global-state moves it, --no-global disables it for a run.

6. Rules

Secrets and tokens

1. Never print, log, commit or echo a Discord token or GoFile account token. Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: --token flag → $DISCORD_TOKEN → hidden prompt. It is handed to the DCE container via the TOKEN env var, never as a command-line argument.
3. ~/.git-credentials may only be read by tools/new_repo.py. Never copy credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.

Failure policy (the core requirement)

5. A failure in one URL or file must never abort the run. Catch, record in state.json, continue. The loops in dlp.py therefore have catch-alls — keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures, pause, then continue. Circuit-breakers that exit the process need a explicit justification in the PR/commit message.
7. Retries: exponential-ish (retry_delay * attempt), default 3 attempts. Password-required gofile links are not retried (it can't help).

Idempotency

8. Re-runs must be safe and cheap: skip URLs marked ok (in state.json or the global store), skip existing non-empty files, write partial downloads to *.part and rename on success.
9. state.json is the source of truth for resume; keep its schema backward compatible when touching state.py (unknown keys are preserved, missing keys defaulted). Same for the global store schema in globalstore.py.
10. --assume-done is a deliberate, manual import step for old URL lists. Never invoke it automatically, and never let the tool mark URLs done without a verified successful download.

Rate limiting / network etiquette

11. One gofile guest account per run; ensure_token() backs off linearly (20s × attempt). Do not raise the guest-account frequency "to make tests pass". Testing account creation more than a couple times per hour is a bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without re-verifying the 403 behavior.

Code style

13. Python 3.12+, type hints on public functions, logging not print in library code (prints are fine only for --dry-run output).
14. Stay on stdlib + requests. New dependencies need a real justification and belong in requirements.txt.
15. dlphelper/ modules must stay import-safe (no side effects at import).

Validation

15. Minimum validation for any change: python3 -m py_compile dlp.py catalogue.py dlphelper/*.py, plus the offline tests (python3 tests/test_extract.py, python3 tests/test_catalogue.py) if the touched modules changed. Document what you could not test (e.g. live gofile/0807 downloads need real links and tokens).
16. Full live runs need a Discord token from the user — ask for it, never ask for it to be pasted into a file or committed.

Git

18. Branch main, descriptive commit messages, no force-push, no history rewriting on the shared remote.
19. When creating a new repo use tools/new_repo.py (Gitea API); creation is fine, deletion/renaming of existing repos requires the user.

7. Roadmap (future stages, keep compatible)

1. Dedupe: hash-based (sha256) duplicate detection across threads via manifest.jsonl (the --hash flag feeds this). URL-level dedupe already exists via the global store; content-level (same file, different link) comes with hashing.

2. Torrent stage: mktorrent per downloaded folder, .torrent files into a watch dir, cross-seeding between direct-download and torrent copies.

3. Library organization (started): catalogue.py classifies every file into system / faction / class and can move files into <library>/<System>/<Group>/<Faction>/<Class>/ with --apply (audited in catalogue_moves.jsonl). The taxonomy lives in dlphelper/warhammer_data.py; it is keyword-based and grows with the collection - unclassified files are reported, never guessed. Manual fixes go into an overrides JSON (--overrides). The library hierarchy follows the GW webshop taxonomy (see below); docs/SORTING.md is the working agreement for the sorting pipeline.

   Sorter matching rules (fixed 2026-10-05 after the first full run): the outer file name wins when it scores >= threshold; inner archive names only vote as fallback (hash-named gofile archives need that). A fuzzy match never jumps into another game's section: keyword-identified systems constrain the catalog pool, and a file with neither system nor faction keyword only adopts a catalog unit on a strong hit (CROSS_SYSTEM_MIN = 0.75) - weaker guesses stay in _unsorted. Token-containment boosts exist for short names vs long catalog names ("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token stoplist.

   Refined 2026-10-07 (dir re-homing + weak tier + non-warhammer drop):

   • Deepest-matching-dir wins: ancestor directory names (inside the archive in provenance mode, real dirs in manual trees) compete with the pack/file name; the best accepted match re-homes the whole subtree. Junk dir names (files/images/HEADS/pure numbers/lone generic words) are never candidates. Fixes multi-unit sculptor packs (one "Retro Elvez" archive ≈ 20 Eldar units no longer all land in one keyword class).
   • Weak tier: when nothing reaches the threshold, a hit >= WEAK_MIN (0.50) is accepted but flagged low confidence - [w] in the plan, confidence: low in the audit, listed in <library>/sorter_low_confidence.txt. Weak and tokenless adoptions require a shared non-generic token; tokenless normal adoption needs +0.08 (TOKENLESS_BONUS); cross-system adoption additionally needs token containment (_token_contained), because char-ratio lies ("aeldari rangers" vs "dwarf rangers" = 0.786).
   • Non-warhammer systems dropped: DROPPED_SYSTEMS (default {Middle-Earth}) - catalog entries filtered at load, keyword hits count as unclassified; files stay in _unsorted. --drop-system/ --keep-system adjust. The catalog snapshot keeps all systems.

   The catalog deliberately covers all GW systems (needed for catalog-agnostic use) - wrong-system placement is prevented by matching discipline and the drop filter, not by shrinking the catalog.

   Catalog source: the GW webshop via Algolia (dlphelper/gwshop.py, working, verified 2026-10-04 — how it was obtained, do not re-derive): scraping warhammer.com directly is blocked — the shop HTML and all search/listing endpoints (/en-WW/shop/*, /app/resources/catalog/search/*, /app/resources/catalog/categories) sit behind an AWS WAF challenge (405/202 "goku" JS captcha) for any non-browser client, regardless of User-Agent; the sitemap declared in robots.txt does not exist; deep paths under the allowed /app/resources/catalog/product/* reach a separate commerce backend (plain 404 Not found) but the resource shape is unknown. What works: the shop's product search runs on Algolia (app-id M5ZIQZNQ2H) and its app-id plus the public search-only API key ship with the browser page; the user captured both from the browser devtools ("Copy as cURL" on one search request) and they are stored in the gitignored algolia.json (or ALGOLIA_APP_ID/ALGOLIA_API_KEY env vars) — the key is search-only, never commit any other Algolia key. The full product index prod-lazarus-product-en-ww (~3500 products, plus one sibling index per locale) is dumped with plain requests: the public search-only key cannot use Algolia's /browse (403) and normal pagination is capped at 1000 cumulative hits, so gwshop.py partitions the index on the GameSystemsRoot hierarchical facet — each branch is fetched with an explicit filter, recursing deeper only when a branch exceeds one page. (Live count 2026-10-04: 4183 records, of which 3491 are reachable via the facet; the ~692 left out are paints, Black Library books, White Dwarf issues, brushes, bases and gifts that carry no GameSystemsRoot path — no miniature kits among them.) Records are normalized into the source-agnostic catalog schema (system/group/faction/subfaction/class = the facet path, name/sku/productType attached) into data/gwshop_catalog.json, which is the sorter's default catalog and is committed as a snapshot; refresh with python3 -m dlphelper.gwshop data/gwshop_catalog.json. Other catalog sources stay pluggable and source-agnostic (miniset.net — reachable but stale, manual JSON overrides, data/library_catalog.json).

4. Watch mode: --subs subscriptions.json + a cron/timer already gives a primitive version; a real watch mode (per-subscription schedules, tear-off notifications, discord pings on new links) builds on it.

8. Testing notes

• tests/ currently holds a synthetic export.json test for extraction (tests/test_extract.py, runnable with plain python3). Keep tests dependency-free and offline — they must never hit gofile/0807/Discord.
• To exercise downloads without Discord: build a --urls-file from known URLs and run with a scratch OUT_DIR.