Warhammer_DLP_Helper/AGENTS.md

AGENTS.md — Warhammer_DLP_Helper

Guidance for any AI agent (or human) working in this repository. Read this fully before making changes.

1. What this project is

The user collects Warhammer proxy models (3rd-party / fan sculpts, never official Games Workshop assets) that are shared in Discord channels. Links in those channels point to two hosters and are often base64-encoded inside message content (obfuscated by the posting bot).

The end goal is a torrent-first platform (think radarr/lidarr, but for proxy files). This repo is the current stage of that journey: a tool that

1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. records a manifest that will later feed the torrent stage.

Deliberately not in scope (yet): .torrent creation, tracker upload, deseeding, rename-into-library conventions. Design decisions should not close the door on these — especially manifest.jsonl and the --hash option exist for dedupe/cross-seeding later.

2. Repository & environment

• Remote: git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper — private repo, branch main. Push with the credentials already stored via git's credential store (~/.git-credentials); never paste them into commands or files.
• The server is Gitea 1.27. To create another repo, use python3 tools/new_repo.py list|create <name> (reads ~/.git-credentials itself, never prints secrets). Never delete or overwrite existing repos.
• Machine: Linux, Python 3.12+ (3.14 available), requests is the only runtime dependency (see requirements.txt), docker is available.
• Reference clone: ~/GoFileDownloader (Lysagxra) — do not modify it; it is upstream reading material only.

3. Architecture

dlp.py                     CLI orchestrator (entry point)
catalogue.py               CLI: classify the collection into system/faction/class
dlphelper/
  discord.py               DCE docker export (JSON) + base64/plain URL extraction
  gofile.py                GoFile API client (one guest account per run)
  zero807.py               curl-based downloader for 0807.st direct links
  state.py                 state.json store + manifest.jsonl appender
  globalstore.py           cross-run URL store (~/.local/share/dlphelper/urls.json)
  catalogue.py             scanner + keyword classifier + report writer
  warhammer_data.py        taxonomy: systems, factions, force-org classes
parts/grab.sh              legacy groundstone script (kept, not used by dlp.py)
tools/new_repo.py          Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py      inspect raw gofile API response for one content id
tests/test_extract.py      offline extraction test (plain python3, no deps)
tests/test_catalogue.py    offline classifier test

Data flow:

graph LR
    A[channel id + token] --> B[DCE docker export -> export.json]
    B --> C[decode base64 in message content]
    C --> D{classify host}
    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
    D -->|0807.st| F[curl with UA + Referer]
    D -->|other| G[recorded as failed: unsupported host]
    E --> H[downloads/gofile/...]
    F --> I[downloads/0807/...]
    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
    J --> K[globalstore: urls.json, marks URL done forever]

Key facts about the hosters (learned the hard way, do not re-learn):

• gofile.io: guest accounts are created via POST api.gofile.io/accounts. Creation is aggressively rate-limited — create at most one account per run. Content listing needs Authorization: Bearer <token>, an X-Website-Token header (sha256 over UA::en-US::<token>::<4h-window>::seed, see gofile.py), and downloads need Cookie: accountToken=<token> plus Referer/Origin headers.
• 0807.st: serves files directly but 403s anything that doesn't look like a browser. curl -A "Mozilla/5.0" -e "https://0807.st/" works; aria2c's default multi-connection request does not. curl is the tool here.
• Encoded links in the channels start as base64 strings matching aHR0cHM6Ly[89]... — keep the extraction regex compatible with parts/grab.sh.

4. Usage (what "done" looks like)

export DISCORD_TOKEN=...
python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>

Retry only failures: python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt (or just re-run — state.json skips everything already marked ok).

Several channels at once (subscriptions.json: list of {name, channel_id, out_dir}):

python3 dlp.py --subs subscriptions.json

Subs mode re-exports every channel by default (that is how new links get picked up) and relies on state.json + the global store to only download what is actually new. Use a cron/systemd timer for watch-like behavior.

Manual import of the pre-dlp.py era (URLs you know are fully downloaded, never re-fetch them again):

python3 dlp.py --assume-done old_urls.txt

5. Run artifacts (all in OUT_DIR, gitignored)

File	Purpose
export.json	DCE export; cached — --fresh re-exports
state.json	url → {status, attempts, error, files[]}; corrupt files are backed up and reset
failed_urls.txt	rewritten each run: every non-ok URL, usable as --urls-file
manifest.jsonl	append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256)
downloads/	actual content; .part files mean "interrupted, will be resumed"
~/.local/share/dlphelper/urls.json	global cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. --global-state moves it, --no-global disables it for a run.

6. Rules

Secrets and tokens

1. Never print, log, commit or echo a Discord token or GoFile account token. Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: --token flag → $DISCORD_TOKEN → hidden prompt. It is handed to the DCE container via the TOKEN env var, never as a command-line argument.
3. ~/.git-credentials may only be read by tools/new_repo.py. Never copy credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.

Failure policy (the core requirement)

5. A failure in one URL or file must never abort the run. Catch, record in state.json, continue. The loops in dlp.py therefore have catch-alls — keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures, pause, then continue. Circuit-breakers that exit the process need a explicit justification in the PR/commit message.
7. Retries: exponential-ish (retry_delay * attempt), default 3 attempts. Password-required gofile links are not retried (it can't help).

Idempotency

8. Re-runs must be safe and cheap: skip URLs marked ok (in state.json or the global store), skip existing non-empty files, write partial downloads to *.part and rename on success.
9. state.json is the source of truth for resume; keep its schema backward compatible when touching state.py (unknown keys are preserved, missing keys defaulted). Same for the global store schema in globalstore.py.
10. --assume-done is a deliberate, manual import step for old URL lists. Never invoke it automatically, and never let the tool mark URLs done without a verified successful download.

Rate limiting / network etiquette

11. One gofile guest account per run; ensure_token() backs off linearly (20s × attempt). Do not raise the guest-account frequency "to make tests pass". Testing account creation more than a couple times per hour is a bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without re-verifying the 403 behavior.

Code style

13. Python 3.12+, type hints on public functions, logging not print in library code (prints are fine only for --dry-run output).
14. Stay on stdlib + requests. New dependencies need a real justification and belong in requirements.txt.
15. dlphelper/ modules must stay import-safe (no side effects at import).

Validation

15. Minimum validation for any change: python3 -m py_compile dlp.py catalogue.py dlphelper/*.py, plus the offline tests (python3 tests/test_extract.py, python3 tests/test_catalogue.py) if the touched modules changed. Document what you could not test (e.g. live gofile/0807 downloads need real links and tokens).
16. Full live runs need a Discord token from the user — ask for it, never ask for it to be pasted into a file or committed.

Git

18. Branch main, descriptive commit messages, no force-push, no history rewriting on the shared remote.
19. When creating a new repo use tools/new_repo.py (Gitea API); creation is fine, deletion/renaming of existing repos requires the user.

7. Roadmap (future stages, keep compatible)

1. Dedupe: hash-based (sha256) duplicate detection across threads via manifest.jsonl (the --hash flag feeds this). URL-level dedupe already exists via the global store; content-level (same file, different link) comes with hashing.

2. Torrent stage: mktorrent per downloaded folder, .torrent files into a watch dir, cross-seeding between direct-download and torrent copies.

3. Library organization (started): catalogue.py classifies every file into system / faction / force-org class and can move files into <library>/<System>/<Faction>/<Class>/ with --apply (audited in catalogue_moves.jsonl). The taxonomy lives in dlphelper/warhammer_data.py; it is keyword-based and grows with the collection - unclassified files are reported, never guessed. Manual fixes goes into an overrides JSON (--overrides).

   Catalog source findings (warhammer.com, verified 2026-10-04, do not re-probe blindly): the shop HTML and all search/listing endpoints (/en-WW/shop/*, /app/resources/catalog/search/*, /app/resources/catalog/categories) sit behind an AWS WAF challenge (405/202 "goku" JS captcha) for any non-browser client, regardless of User-Agent. Static assets, robots.txt and 404 pages are served normally. The sitemap declared in robots.txt does not exist. Deep paths under the allowed /app/resources/catalog/product/* reach a separate commerce backend (plain 404 Not found) but the correct resource shape is unknown. The site's product search is Algolia; its app-id and search-only key are public (they ship with the browser page) and would allow a full catalog dump via plain requests — this is the most promising path: capture one search request from the browser devtools ("Copy as cURL") and build dlphelper/gwshop.py on it. Catalog sources stay pluggable and source-agnostic regardless (miniset.net, manual JSON overrides, GW shop).

4. Watch mode: --subs subscriptions.json + a cron/timer already gives a primitive version; a real watch mode (per-subscription schedules, tear-off notifications, discord pings on new links) builds on it.

8. Testing notes

• tests/ currently holds a synthetic export.json test for extraction (tests/test_extract.py, runnable with plain python3). Keep tests dependency-free and offline — they must never hit gofile/0807/Discord.
• To exercise downloads without Discord: build a --urls-file from known URLs and run with a scratch OUT_DIR.