Warhammer_DLP_Helper/AGENTS.md

# AGENTS.md — Warhammer_DLP_Helper

Guidance for any AI agent (or human) working in this repository. Read this
fully before making changes.

## 1. What this project is

The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
official Games Workshop assets) that are shared in Discord channels. Links in
those channels point to two hosters and are often **base64-encoded** inside
message content (obfuscated by the posting bot).

The end goal is a **torrent-first platform** (think radarr/lidarr, but for
proxy files). This repo is the Rust implementation of the current stage: one
binary, `dlphelper`, that

1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. extracts the archives into a plain-file tree,
6. sorts the files into the library by system/group/faction/class/unit,
7. records a manifest that will later feed the torrent stage.

A webapp is the next stage. Deliberately **not** in scope (yet): `.torrent`
creation, tracker upload, deseeeding, rename-into-library conventions. Design
decisions should not close the door on these — especially `manifest.jsonl`
and the `--hash` option exist for dedupe/cross-seeding later.

This repository was ported from a Python implementation, which the Rust
version has now replaced (the ported modules keep `Rust port of …` provenance
notes in their headers).

## 2. Repository & environment

- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
  **private** repo, branch `main`. Push with the credentials already stored
  via git's credential store (`~/.git-credentials`); never paste them into
  commands or files.
- Machine: Linux, Rust (edition 2024), docker is available. The repo lives on
  **NFS**, where rustc SIGBUSes — always build/test with an explicit target
  directory:

      CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test

- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
  upstream reading material only.

## 3. Architecture

```
Cargo.toml                  crate `dlphelper` (edition 2024)
src/main.rs                 CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
src/orchestrate.rs          dlp pipeline: export -> extract URLs -> per-host download
src/discord.rs              DCE docker export (JSON) + base64/plain URL extraction
src/gofile.rs               GoFile API client (one guest account per run)
src/zero807.rs              curl downloader for 0807.st (direct links + /f/<id> folder pages)
src/state.rs                state.json store
src/globalstore.rs          cross-run URL store (~/.local/share/dlphelper/urls.json)
src/manifest.rs             manifest.jsonl appender
src/extract.rs              recursive archive extraction into a plain-file tree
src/catalogue.rs            scanner + keyword classifier + report writer
src/sorter.rs               library sorter: inspect/classify/fuzzy-match, plan + apply
src/difflib.rs              faithful CPython SequenceMatcher port (bit-identical ratios)
src/warhammer_data.rs       taxonomy tables (parsed from data/warhammer_data.json)
src/gwshop.rs               GW webshop catalog dump via Algolia
src/util.rs                 small shared helpers
data/gwshop_catalog.json    committed catalog snapshot (sorter default)
data/warhammer_data.json    committed taxonomy tables
docs/SORTING.md             the library/sorting contract (change it before the code)
```

Data flow:

```mermaid
graph LR
    A[channel id + token] --> B[DCE docker export -> export.json]
    B --> C[decode base64 in message content]
    C --> D{classify host}
    D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
    D -->|0807.st| F[curl with UA + Referer]
    D -->|other| G[recorded as failed: unsupported host]
    E --> H[downloads/gofile/...]
    F --> I[downloads/0807/...]
    E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
    J --> K[globalstore: urls.json, marks URL done forever]
    J --> L[extract -> plain-file tree]
    L --> M[sorter -> library, hardlinks + audit]
```

Key facts about the hosters (learned the hard way, do not re-learn):

- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
  Creation is aggressively rate-limited — create **at most one account per
  run**. Content listing needs `Authorization: Bearer <token>`, an
  `X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
  see `src/gofile.rs`), and downloads need `Cookie: accountToken=<token>` plus
  Referer/Origin headers.
- **0807.st**: serves files directly but 403s anything that doesn't look like
  a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
  default multi-connection request does not. curl is the tool here.
  Folder pages (`https://0807.st/f/<id>`) are expanded: the page is parsed and
  each listed file is fetched from `https://0807.st/<id>?dl=1` into
  `downloads/0807/<page-id>/`. The folder `.zip` export is refused by the site
  ("folder too large"), so per-file download is the only route in.
- Encoded links in the channels start as base64 strings matching
  `aHR0cHM6Ly[89]...`.

## 4. Usage (what "done" looks like)

    export DISCORD_TOKEN=...
    dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash

`--since 5d` (or `12h`, `2026-10-04`) limits a run to recent messages — handy
for incremental updates. Re-runs are safe: `state.json` and the global store
skip everything already marked ok.

Retry only failures:

    dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt

Several channels at once (`subscriptions.json`: list of `{name, channel_id,
out_dir}`):

    dlphelper dlp --subs subscriptions.json

Subs mode re-exports every channel by default (that is how new links get
picked up) and relies on `state.json` + the global store to only download what
is actually new. Use a cron/systemd timer for watch-like behavior.

Extract and sort:

    dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
    dlphelper sorter <out>/extracted --library <library> --apply

Manual import of old URL lists (known fully downloaded, never re-fetch):

    dlphelper dlp --assume-done old_urls.txt

## 5. Run artifacts (all in OUT_DIR, gitignored)

| File | Purpose |
|------|---------|
| `export.json` | DCE export; cached — `--fresh` re-exports |
| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
| `extracted/extract_manifest.jsonl` | one line per extraction `{archive, dest, unit}` (feeds `sorter --provenance`) |
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. `--global-state` moves it, `--no-global` disables it for a run. |

Sorter bookkeeping (audit + low-confidence doc) is written to
`<library>/../_logs/` by default — never inside the library (see §7).

## 6. Rules

### Secrets and tokens
1. Never print, log, commit or echo a Discord token or GoFile account token.
   Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
   It is handed to the DCE container via the `TOKEN` env var, never as a
   command-line argument.
3. `~/.git-credentials` may only be read by tooling that needs it; never copy
   credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.

### Failure policy (the core requirement)
5. A failure in one URL or file must **never** abort the run. Record it in
   `state.json` and continue. The per-URL loops in `src/orchestrate.rs`
   therefore have catch-alls — keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures,
   pause, then continue. Circuit-breakers that exit the process need an
   explicit justification in the commit message.
7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
   Password-required gofile links are not retried (it can't help); dead links
   (HTTP 404/410) are never retried.

### Idempotency
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json`
   **or** the global store), skip existing non-empty files, write partial
   downloads to `*.part` and rename on success.
9. `state.json` is the source of truth for resume; keep its schema backward
   compatible (unknown keys preserved, missing keys defaulted). Same for the
   global store schema.
10. `--assume-done` is a deliberate, manual import step for old URL lists.
    Never invoke it automatically, and never let the tool mark URLs done
    without a verified successful download.

### Rate limiting / network etiquette
11. One gofile guest account per run; `ensure_token()` backs off linearly
    (20s × attempt). Do not raise the guest-account frequency "to make tests
    pass". Testing account creation more than a couple times per hour is a
    bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
    re-verifying the 403 behavior.

### Code style
13. Rust 2024, idiomatic. Use `log` (not `println!`) in library modules;
    prints are fine only for CLI output (plan/dry-run).
14. Keep new dependencies justified — the crate already covers HTTP
    (`reqwest`), archives (`sevenz-rust2`, `zip`, `unrar`), fuzzy matching
    (the in-tree `difflib` port, **not** an external crate) and CLI (`clap`).
    The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
15. `src/difflib.rs` must stay bit-identical to CPython `difflib` — the sorter
    thresholds depend on exact ratios; do not "simplify" it.

### Validation
16. Minimum validation for any change: `CARGO_TARGET_DIR=/var/tmp/dlphelper-target
    cargo test` (the offline suite must stay green). Document what you could
    *not* test (live gofile/0807/Discord paths need real links and tokens).
17. Full live runs need a Discord token from the user — ask for it, never ask
    for it to be pasted into a file or committed.

### Git
18. Branch `main`, descriptive commit messages, no force-push, no history
    rewriting on the shared remote.

## 7. Sorter & library structure

`docs/SORTING.md` is the working agreement for the library hierarchy and the
sorting pipeline — **change it first, then the code**. Summary:

- Hierarchy: `<library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>`.
  Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- Matching rules are fixed: the outer file name wins when it scores ≥
  threshold; inner archive names only vote as fallback. A fuzzy match never
  jumps into another game's section (keyword systems constrain the catalog
  pool; cross-system adoption needs `CROSS_SYSTEM_MIN = 0.75` and token
  containment). Deepest-matching-dir wins (junk dir names are never
  candidates). Weak hits ≥ `WEAK_MIN = 0.50` are accepted but flagged
  low-confidence. Tokenless normal adoption needs `+0.08`
  (`TOKENLESS_BONUS`).
- `DROPPED_SYSTEMS` (default `{Middle-Earth}`) filters catalog entries at load
  and counts keyword hits as unclassified; `--drop-system`/`--keep-system`
  adjust. The committed catalog snapshot keeps all systems.
- Catalog source: the GW webshop via Algolia (`src/gwshop.rs`). The app-id +
  public **search-only** key live in the gitignored `algolia.json` (or
  `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY`); never commit any other Algolia key.
  `data/gwshop_catalog.json` is the committed snapshot and the sorter default.
- Bookkeeping (`sorter_applied.jsonl`, `sorter_low_confidence.txt`) is written
  to `<library>/../_logs/` by default — never inside the library, which may be
  a seeded torrent whose file set is frozen.

## 8. Testing notes

- Tests live inline (`#[cfg(test)]`) in each module and run with `cargo test`.
  Keep them dependency-free and offline — they must never hit
  gofile/0807/Discord.
- `tempfile` tests stage under `/var/tmp` when available, because `/tmp` is a
  small tmpfs that trips the extractor's free-space guard (`MIN_FREE_BYTES`).
- To exercise downloads without Discord: build a `--urls-file` from known URLs
  and run with a scratch OUT_DIR.