Warhammer_DLP_Helper/AGENTS.md

# AGENTS.md — Warhammer_DLP_Helper

Guidance for any AI agent (or human) working in this repository. Read this
fully before making changes.

## 1. What this project is

The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
official Games Workshop assets) that are shared in Discord channels. Links in
those channels point to two hosters and are often **base64-encoded** inside
message content (obfuscated by the posting bot).

The end goal is a **torrent-first platform** (think radarr/lidarr, but for
proxy files). This repo is the current stage of that journey: a tool that

1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. records a manifest that will later feed the torrent stage.

Deliberately **not** in scope (yet): .torrent creation, tracker upload,
deseeding, rename-into-library conventions. Design decisions should not close
the door on these — especially `manifest.jsonl` and the `--hash` option exist
for dedupe/cross-seeding later.

## 2. Repository & environment

- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
  **private** repo, branch `main`. Push with the credentials already stored
  via git's credential store (`~/.git-credentials`); never paste them into
  commands or files.
- The server is Gitea 1.27. To create *another* repo, use
  `python3 tools/new_repo.py list|create <name>` (reads `~/.git-credentials`
  itself, never prints secrets). Never delete or overwrite existing repos.
- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only
  runtime dependency (see `requirements.txt`), docker is available.
- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
  upstream reading material only.

## 3. Architecture

```
dlp.py                     CLI orchestrator (entry point)
catalogue.py               CLI: classify the collection into system/faction/class
dlphelper/
  discord.py               DCE docker export (JSON) + base64/plain URL extraction
  gofile.py                GoFile API client (one guest account per run)
  zero807.py               curl-based downloader for 0807.st direct links
  state.py                 state.json store + manifest.jsonl appender
  globalstore.py           cross-run URL store (~/.local/share/dlphelper/urls.json)
  catalogue.py             scanner + keyword classifier + report writer
  warhammer_data.py        taxonomy: systems, factions, force-org classes
parts/grab.sh              legacy groundstone script (kept, not used by dlp.py)
tools/new_repo.py          Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py      inspect raw gofile API response for one content id
tests/test_extract.py      offline extraction test (plain python3, no deps)
tests/test_catalogue.py    offline classifier test
```

Data flow:

```mermaid
graph LR
    A[channel id + token] --> B[DCE docker export -> export.json]
    B --> C[decode base64 in message content]
    C --> D{classify host}
    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
    D -->|0807.st| F[curl with UA + Referer]
    D -->|other| G[recorded as failed: unsupported host]
    E --> H[downloads/gofile/...]
    F --> I[downloads/0807/...]
    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
    J --> K[globalstore: urls.json, marks URL done forever]
```

Key facts about the hosters (learned the hard way, do not re-learn):

- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
  Creation is aggressively rate-limited — create **at most one account per
  run**. Content listing needs `Authorization: Bearer <token>`, an
  `X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
  see `gofile.py`), and downloads need `Cookie: accountToken=<token>` plus
  Referer/Origin headers.
- **0807.st**: serves files directly but 403s anything that doesn't look like
  a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
  default multi-connection request does not. curl is the tool here.
- Encoded links in the channels start as base64 strings matching
  `aHR0cHM6Ly[89]...` — keep the extraction regex compatible with
  `parts/grab.sh`.

## 4. Usage (what "done" looks like)

    export DISCORD_TOKEN=...
    python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>

Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt`
(or just re-run — state.json skips everything already marked ok).

Several channels at once (`subscriptions.json`: list of {name, channel_id,
out_dir}):

    python3 dlp.py --subs subscriptions.json

Subs mode re-exports every channel by default (that is how new links get
picked up) and relies on state.json + the global store to only download
what is actually new. Use a cron/systemd timer for watch-like behavior.

Manual import of the pre-dlp.py era (URLs you know are fully downloaded,
never re-fetch them again):

    python3 dlp.py --assume-done old_urls.txt

## 5. Run artifacts (all in OUT_DIR, gitignored)

| File | Purpose |
|------|---------|
| `export.json` | DCE export; cached — `--fresh` re-exports |
| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. |

## 6. Rules

### Secrets and tokens
1. Never print, log, commit or echo a Discord token or GoFile account token.
   Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
   It is handed to the DCE container via the `TOKEN` env var, never as a
   command-line argument.
3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy
   credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.

### Failure policy (the core requirement)
5. A failure in one URL or file must **never** abort the run. Catch, record in
   `state.json`, continue. The loops in `dlp.py` therefore have catch-alls —
   keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures,
   pause, then continue. Circuit-breakers that exit the process need a
   explicit justification in the PR/commit message.
7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
   Password-required gofile links are not retried (it can't help).

### Idempotency
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or**
   the global store), skip existing non-empty files, write partial downloads
   to `*.part` and rename on success.
9. `state.json` is the source of truth for resume; keep its schema backward
   compatible when touching `state.py` (unknown keys are preserved, missing
   keys defaulted). Same for the global store schema in `globalstore.py`.
10. `--assume-done` is a deliberate, manual import step for old URL lists.
    Never invoke it automatically, and never let the tool mark URLs done
    without a verified successful download.

### Rate limiting / network etiquette
11. One gofile guest account per run; `ensure_token()` backs off linearly
    (20s × attempt). Do not raise the guest-account frequency "to make tests
    pass". Testing account creation more than a couple times per hour is a
    bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
    re-verifying the 403 behavior.

### Code style
13. Python 3.12+, type hints on public functions, `logging` not `print` in
    library code (prints are fine only for `--dry-run` output).
14. Stay on stdlib + `requests`. New dependencies need a real justification
    and belong in `requirements.txt`.
15. `dlphelper/` modules must stay import-safe (no side effects at import).

### Validation
15. Minimum validation for any change: `python3 -m py_compile dlp.py
    catalogue.py dlphelper/*.py`, plus the offline tests (`python3
    tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
    modules changed. Document what you could *not* test (e.g. live
    gofile/0807 downloads need real links and tokens).
17. Full live runs need a Discord token from the user — ask for it, never
    ask for it to be pasted into a file or committed.

### Git
18. Branch `main`, descriptive commit messages, no force-push, no history
    rewriting on the shared remote.
19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
    fine, deletion/renaming of existing repos requires the user.

## 7. Roadmap (future stages, keep compatible)

1. **Dedupe**: hash-based (sha256) duplicate detection across threads via
   `manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already
   exists via the global store; content-level (same file, different link)
   comes with hashing.
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
   a watch dir, cross-seeding between direct-download and torrent copies.
3. **Library organization (started)**: `catalogue.py` classifies every file
   into system / faction / force-org class and can move files into
   `<library>/<System>/<Faction>/<Class>/` with `--apply` (audited in
   `catalogue_moves.jsonl`). The taxonomy lives in
   `dlphelper/warhammer_data.py`; it is keyword-based and grows with the
   collection - unclassified files are reported, never guessed. Manual fixes
   goes into an overrides JSON (`--overrides`).

   **Catalog source findings (warhammer.com, verified 2026-10-04, do not
   re-probe blindly):** the shop HTML and all search/listing endpoints
   (`/en-WW/shop/*`, `/app/resources/catalog/search/*`,
   `/app/resources/catalog/categories`) sit behind an **AWS WAF** challenge
   (405/202 "goku" JS captcha) for any non-browser client, regardless of
   User-Agent. Static assets, robots.txt and 404 pages are served normally.
   The sitemap declared in robots.txt does not exist. Deep paths under the
   allowed `/app/resources/catalog/product/*` reach a separate commerce
   backend (plain `404 Not found`) but the correct resource shape is
   unknown. The site's product search is **Algolia**; its app-id and
   search-only key are public (they ship with the browser page) and would
   allow a full catalog dump via plain `requests` — this is the most
   promising path: capture one search request from the browser devtools
   ("Copy as cURL") and build `dlphelper/gwshop.py` on it. Catalog sources
   stay pluggable and source-agnostic regardless (miniset.net, manual JSON
   overrides, GW shop).
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
   primitive version; a real watch mode (per-subscription schedules, tear-off
   notifications, discord pings on new links) builds on it.

## 8. Testing notes

- `tests/` currently holds a synthetic `export.json` test for extraction
  (`tests/test_extract.py`, runnable with plain `python3`). Keep tests
  dependency-free and offline — they must never hit gofile/0807/Discord.
- To exercise downloads without Discord: build a `--urls-file` from known
  URLs and run with a scratch OUT_DIR.