Ben_Kosytorz
93b86d34af
- src/gwshop.rs: pick the FIRST longest facet path on ties, matching the original Python max(paths, key=len). Rust's max_by_key returned the last, which silently reclassified ~2298 entries for products faceted under several branches (e.g. a generic kit listed per chapter). Adds a regression test. - AGENTS.md: document the first-longest tie-break and the facet-partition coverage shortfall (non-system merchandise left out; no model kits). - data/gwshop_catalog.json: regenerate with the fixed binary (3491 -> 3459 entries; 34 removed, 2 added, 188 structural + 785 collections changes).
266 lines
13 KiB
Markdown
266 lines
13 KiB
Markdown
# AGENTS.md — Warhammer_DLP_Helper
|
||
|
||
Guidance for any AI agent (or human) working in this repository. Read this
|
||
fully before making changes.
|
||
|
||
## 1. What this project is
|
||
|
||
The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
|
||
official Games Workshop assets) that are shared in Discord channels. Links in
|
||
those channels point to two hosters and are often **base64-encoded** inside
|
||
message content (obfuscated by the posting bot).
|
||
|
||
The end goal is a **torrent-first platform** (think radarr/lidarr, but for
|
||
proxy files). This repo is the Rust implementation of the current stage: one
|
||
binary, `dlphelper`, that
|
||
|
||
1. exports a Discord channel/thread (DiscordChatExporter via docker),
|
||
2. extracts base64-encoded and plain download links,
|
||
3. picks the right downloader per host automatically,
|
||
4. downloads everything with per-file tracking, retries and resume,
|
||
5. extracts the archives into a plain-file tree,
|
||
6. sorts the files into the library by system/group/faction/class/unit,
|
||
7. records a manifest that will later feed the torrent stage.
|
||
|
||
A webapp is the next stage. Deliberately **not** in scope (yet): `.torrent`
|
||
creation, tracker upload, deseeeding, rename-into-library conventions. Design
|
||
decisions should not close the door on these — especially `manifest.jsonl`
|
||
and the `--hash` option exist for dedupe/cross-seeding later.
|
||
|
||
This repository was ported from a Python implementation, which the Rust
|
||
version has now replaced (the ported modules keep `Rust port of …` provenance
|
||
notes in their headers).
|
||
|
||
## 2. Repository & environment
|
||
|
||
- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
|
||
**private** repo, branch `main`. Push with the credentials already stored
|
||
via git's credential store (`~/.git-credentials`); never paste them into
|
||
commands or files.
|
||
- Machine: Linux, Rust (edition 2024), docker is available. The repo lives on
|
||
**NFS**, where rustc SIGBUSes — always build/test with an explicit target
|
||
directory:
|
||
|
||
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
|
||
|
||
- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
|
||
upstream reading material only.
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
Cargo.toml crate `dlphelper` (edition 2024)
|
||
src/main.rs CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
|
||
src/orchestrate.rs dlp pipeline: export -> extract URLs -> per-host download
|
||
src/discord.rs DCE docker export (JSON) + base64/plain URL extraction
|
||
src/gofile.rs GoFile API client (one guest account per run)
|
||
src/zero807.rs curl downloader for 0807.st (direct links + /f/<id> folder pages)
|
||
src/state.rs state.json store
|
||
src/globalstore.rs cross-run URL store (~/.local/share/dlphelper/urls.json)
|
||
src/manifest.rs manifest.jsonl appender
|
||
src/extract.rs recursive archive extraction into a plain-file tree
|
||
src/catalogue.rs scanner + keyword classifier + report writer
|
||
src/sorter.rs library sorter: inspect/classify/fuzzy-match, plan + apply
|
||
src/difflib.rs faithful CPython SequenceMatcher port (bit-identical ratios)
|
||
src/warhammer_data.rs taxonomy tables (parsed from data/warhammer_data.json)
|
||
src/gwshop.rs GW webshop catalog dump via Algolia
|
||
src/util.rs small shared helpers
|
||
data/gwshop_catalog.json committed catalog snapshot (sorter default)
|
||
data/warhammer_data.json committed taxonomy tables
|
||
docs/SORTING.md the library/sorting contract (change it before the code)
|
||
```
|
||
|
||
Data flow:
|
||
|
||
```mermaid
|
||
graph LR
|
||
A[channel id + token] --> B[DCE docker export -> export.json]
|
||
B --> C[decode base64 in message content]
|
||
C --> D{classify host}
|
||
D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
|
||
D -->|0807.st| F[curl with UA + Referer]
|
||
D -->|other| G[recorded as failed: unsupported host]
|
||
E --> H[downloads/gofile/...]
|
||
F --> I[downloads/0807/...]
|
||
E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
|
||
J --> K[globalstore: urls.json, marks URL done forever]
|
||
J --> L[extract -> plain-file tree]
|
||
L --> M[sorter -> library, hardlinks + audit]
|
||
```
|
||
|
||
Key facts about the hosters (learned the hard way, do not re-learn):
|
||
|
||
- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
|
||
Creation is aggressively rate-limited — create **at most one account per
|
||
run**. Content listing needs `Authorization: Bearer <token>`, an
|
||
`X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
|
||
see `src/gofile.rs`), and downloads need `Cookie: accountToken=<token>` plus
|
||
Referer/Origin headers.
|
||
- **0807.st**: serves files directly but 403s anything that doesn't look like
|
||
a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
|
||
default multi-connection request does not. curl is the tool here.
|
||
Folder pages (`https://0807.st/f/<id>`) are expanded: the page is parsed and
|
||
each listed file is fetched from `https://0807.st/<id>?dl=1` into
|
||
`downloads/0807/<page-id>/`. The folder `.zip` export is refused by the site
|
||
("folder too large"), so per-file download is the only route in.
|
||
- Encoded links in the channels start as base64 strings matching
|
||
`aHR0cHM6Ly[89]...`.
|
||
|
||
## 4. Usage (what "done" looks like)
|
||
|
||
export DISCORD_TOKEN=...
|
||
dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash
|
||
|
||
`--since 5d` (or `12h`, `2026-10-04`) limits a run to recent messages — handy
|
||
for incremental updates. Re-runs are safe: `state.json` and the global store
|
||
skip everything already marked ok.
|
||
|
||
Retry only failures:
|
||
|
||
dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt
|
||
|
||
Several channels at once (`subscriptions.json`: list of `{name, channel_id,
|
||
out_dir}`):
|
||
|
||
dlphelper dlp --subs subscriptions.json
|
||
|
||
Subs mode re-exports every channel by default (that is how new links get
|
||
picked up) and relies on `state.json` + the global store to only download what
|
||
is actually new. Use a cron/systemd timer for watch-like behavior.
|
||
|
||
Extract and sort:
|
||
|
||
dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
|
||
dlphelper sorter <out>/extracted --library <library> --apply
|
||
|
||
Manual import of old URL lists (known fully downloaded, never re-fetch):
|
||
|
||
dlphelper dlp --assume-done old_urls.txt
|
||
|
||
## 5. Run artifacts (all in OUT_DIR, gitignored)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `export.json` | DCE export; cached — `--fresh` re-exports |
|
||
| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
|
||
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
|
||
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
|
||
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
|
||
| `extracted/extract_manifest.jsonl` | one line per extraction `{archive, dest, unit}` (feeds `sorter --provenance`) |
|
||
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. `--global-state` moves it, `--no-global` disables it for a run. |
|
||
|
||
Sorter bookkeeping (audit + low-confidence doc) is written to
|
||
`<library>/../_logs/` by default — never inside the library (see §7).
|
||
|
||
## 6. Rules
|
||
|
||
### Secrets and tokens
|
||
1. Never print, log, commit or echo a Discord token or GoFile account token.
|
||
Not even partial. Strip them from any output you paste into chats.
|
||
2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
|
||
It is handed to the DCE container via the `TOKEN` env var, never as a
|
||
command-line argument.
|
||
3. `~/.git-credentials` may only be read by tooling that needs it; never copy
|
||
credentials into scripts, tests, or fixtures.
|
||
4. Nothing downloaded and no export/state/manifest file is ever committed.
|
||
|
||
### Failure policy (the core requirement)
|
||
5. A failure in one URL or file must **never** abort the run. Record it in
|
||
`state.json` and continue. The per-URL loops in `src/orchestrate.rs`
|
||
therefore have catch-alls — keep them.
|
||
6. Cooldowns are the only permitted "stall": after N consecutive failures,
|
||
pause, then continue. Circuit-breakers that exit the process need an
|
||
explicit justification in the commit message.
|
||
7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
|
||
Password-required gofile links are not retried (it can't help); dead links
|
||
(HTTP 404/410) are never retried.
|
||
|
||
### Idempotency
|
||
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json`
|
||
**or** the global store), skip existing non-empty files, write partial
|
||
downloads to `*.part` and rename on success.
|
||
9. `state.json` is the source of truth for resume; keep its schema backward
|
||
compatible (unknown keys preserved, missing keys defaulted). Same for the
|
||
global store schema.
|
||
10. `--assume-done` is a deliberate, manual import step for old URL lists.
|
||
Never invoke it automatically, and never let the tool mark URLs done
|
||
without a verified successful download.
|
||
|
||
### Rate limiting / network etiquette
|
||
11. One gofile guest account per run; `ensure_token()` backs off linearly
|
||
(20s × attempt). Do not raise the guest-account frequency "to make tests
|
||
pass". Testing account creation more than a couple times per hour is a
|
||
bug, not a test strategy.
|
||
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
|
||
re-verifying the 403 behavior.
|
||
|
||
### Code style
|
||
13. Rust 2024, idiomatic. Use `log` (not `println!`) in library modules;
|
||
prints are fine only for CLI output (plan/dry-run).
|
||
14. Keep new dependencies justified — the crate already covers HTTP
|
||
(`reqwest`), archives (`sevenz-rust2`, `zip`, `unrar`), fuzzy matching
|
||
(the in-tree `difflib` port, **not** an external crate) and CLI (`clap`).
|
||
The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
|
||
15. `src/difflib.rs` must stay bit-identical to CPython `difflib` — the sorter
|
||
thresholds depend on exact ratios; do not "simplify" it.
|
||
|
||
### Validation
|
||
16. Minimum validation for any change: `CARGO_TARGET_DIR=/var/tmp/dlphelper-target
|
||
cargo test` (the offline suite must stay green). Document what you could
|
||
*not* test (live gofile/0807/Discord paths need real links and tokens).
|
||
17. Full live runs need a Discord token from the user — ask for it, never ask
|
||
for it to be pasted into a file or committed.
|
||
|
||
### Git
|
||
18. Branch `main`, descriptive commit messages, no force-push, no history
|
||
rewriting on the shared remote.
|
||
|
||
## 7. Sorter & library structure
|
||
|
||
`docs/SORTING.md` is the working agreement for the library hierarchy and the
|
||
sorting pipeline — **change it first, then the code**. Summary:
|
||
|
||
- Hierarchy: `<library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>`.
|
||
Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
|
||
- Matching rules are fixed: the outer file name wins when it scores ≥
|
||
threshold; inner archive names only vote as fallback. A fuzzy match never
|
||
jumps into another game's section (keyword systems constrain the catalog
|
||
pool; cross-system adoption needs `CROSS_SYSTEM_MIN = 0.75` and token
|
||
containment). Deepest-matching-dir wins (junk dir names are never
|
||
candidates). Weak hits ≥ `WEAK_MIN = 0.50` are accepted but flagged
|
||
low-confidence. Tokenless normal adoption needs `+0.08`
|
||
(`TOKENLESS_BONUS`).
|
||
- Sorting runs in the sorter's **default (non-provenance) mode**: every
|
||
file is classified on its own name (unclassified files land directly in
|
||
`_unsorted/<file>`). `--provenance` (classify whole extraction dirs as
|
||
units) exists but the live library does **not** use it; mixing the two
|
||
modes creates duplicate placements.
|
||
- `DROPPED_SYSTEMS` (default `{Middle-Earth}`) filters catalog entries at load
|
||
and counts keyword hits as unclassified; `--drop-system`/`--keep-system`
|
||
adjust. The committed catalog snapshot keeps all systems.
|
||
- Catalog source: the GW webshop via Algolia (`src/gwshop.rs`). The app-id +
|
||
public **search-only** key live in the gitignored `algolia.json` (or
|
||
`ALGOLIA_APP_ID`/`ALGOLIA_API_KEY`); never commit any other Algolia key.
|
||
`data/gwshop_catalog.json` is the committed snapshot and the sorter default.
|
||
- The dump partitions the index on the `GameSystemsRoot` hierarchical facet
|
||
(the search-only key cannot `/browse`; plain pagination caps at 1000 hits).
|
||
When a product is faceted under several branches of equal depth (e.g. a
|
||
generic kit listed per chapter), the **first longest path wins** — Python's
|
||
`max(paths, key=len)` semantics, reproduced explicitly because Rust's
|
||
`max_by_key` returns the *last* maximum. Products carrying no
|
||
`GameSystemsRoot` path (paints, Black Library books, White Dwarf, bases,
|
||
gifts — no miniature kits) are left out; the coverage shortfall is logged.
|
||
- Bookkeeping (`sorter_applied.jsonl`, `sorter_low_confidence.txt`) is written
|
||
to `<library>/../_logs/` by default — never inside the library, which may be
|
||
a seeded torrent whose file set is frozen.
|
||
|
||
## 8. Testing notes
|
||
|
||
- Tests live inline (`#[cfg(test)]`) in each module and run with `cargo test`.
|
||
Keep them dependency-free and offline — they must never hit
|
||
gofile/0807/Discord.
|
||
- `tempfile` tests stage under `/var/tmp` when available, because `/tmp` is a
|
||
small tmpfs that trips the extractor's free-space guard (`MIN_FREE_BYTES`).
|
||
- To exercise downloads without Discord: build a `--urls-file` from known URLs
|
||
and run with a scratch OUT_DIR.
|