276 lines
15 KiB
Markdown
276 lines
15 KiB
Markdown
# AGENTS.md — Warhammer_DLP_Helper
|
||
|
||
Guidance for any AI agent (or human) working in this repository. Read this
|
||
fully before making changes.
|
||
|
||
## 1. What this project is
|
||
|
||
The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
|
||
official Games Workshop assets) that are shared in Discord channels. Links in
|
||
those channels point to two hosters and are often **base64-encoded** inside
|
||
message content (obfuscated by the posting bot).
|
||
|
||
The end goal is a **torrent-first platform** (think radarr/lidarr, but for
|
||
proxy files). This repo is the current stage of that journey: a tool that
|
||
|
||
1. exports a Discord channel/thread (DiscordChatExporter via docker),
|
||
2. extracts base64-encoded and plain download links,
|
||
3. picks the right downloader per host automatically,
|
||
4. downloads everything with per-file tracking, retries and resume,
|
||
5. records a manifest that will later feed the torrent stage.
|
||
|
||
Deliberately **not** in scope (yet): .torrent creation, tracker upload,
|
||
deseeding, rename-into-library conventions. Design decisions should not close
|
||
the door on these — especially `manifest.jsonl` and the `--hash` option exist
|
||
for dedupe/cross-seeding later.
|
||
|
||
## 2. Repository & environment
|
||
|
||
- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
|
||
**private** repo, branch `main`. Push with the credentials already stored
|
||
via git's credential store (`~/.git-credentials`); never paste them into
|
||
commands or files.
|
||
- The server is Gitea 1.27. To create *another* repo, use
|
||
`python3 tools/new_repo.py list|create <name>` (reads `~/.git-credentials`
|
||
itself, never prints secrets). Never delete or overwrite existing repos.
|
||
- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only
|
||
runtime dependency (see `requirements.txt`), docker is available.
|
||
- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
|
||
upstream reading material only.
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
dlp.py CLI orchestrator (entry point)
|
||
catalogue.py CLI: classify the collection into system/faction/class
|
||
sorter.py CLI: fuzzysort files into the library (plan/apply,
|
||
hardlinks, default catalog data/gwshop_catalog.json)
|
||
dlphelper/
|
||
discord.py DCE docker export (JSON) + base64/plain URL extraction
|
||
gofile.py GoFile API client (one guest account per run)
|
||
zero807.py curl-based downloader for 0807.st direct links
|
||
state.py state.json store + manifest.jsonl appender
|
||
globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json)
|
||
catalogue.py scanner + keyword classifier + report writer
|
||
warhammer_data.py taxonomy: GW-named systems/factions, GW shop
|
||
categories as classes, GROUP_OF_FACTION map
|
||
gwshop.py GW webshop catalog dump via Algolia (see §7.3)
|
||
sorter.py library sorter: inspect/classify/fuzzy-match
|
||
against the catalog, plan + hardlink apply,
|
||
optional --provenance unit mode for extracted trees
|
||
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
|
||
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
|
||
tools/debug_gofile.py inspect raw gofile API response for one content id
|
||
tools/extract_all.py recursive archive extraction into a plain-file tree
|
||
(extract_manifest.jsonl feeds the sorter's
|
||
--provenance unit mode; failures go to .broken/;
|
||
7z-direct-to-CIFS failures fall back to a
|
||
/var/tmp extraction with cp437 name repair;
|
||
AppleDouble/__MACOSX junk is dropped)
|
||
tests/test_extract.py offline extraction test (plain python3, no deps)
|
||
tests/test_catalogue.py offline classifier test
|
||
```
|
||
|
||
Data flow:
|
||
|
||
```mermaid
|
||
graph LR
|
||
A[channel id + token] --> B[DCE docker export -> export.json]
|
||
B --> C[decode base64 in message content]
|
||
C --> D{classify host}
|
||
D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
|
||
D -->|0807.st| F[curl with UA + Referer]
|
||
D -->|other| G[recorded as failed: unsupported host]
|
||
E --> H[downloads/gofile/...]
|
||
F --> I[downloads/0807/...]
|
||
E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
|
||
J --> K[globalstore: urls.json, marks URL done forever]
|
||
```
|
||
|
||
Key facts about the hosters (learned the hard way, do not re-learn):
|
||
|
||
- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
|
||
Creation is aggressively rate-limited — create **at most one account per
|
||
run**. Content listing needs `Authorization: Bearer <token>`, an
|
||
`X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
|
||
see `gofile.py`), and downloads need `Cookie: accountToken=<token>` plus
|
||
Referer/Origin headers.
|
||
- **0807.st**: serves files directly but 403s anything that doesn't look like
|
||
a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
|
||
default multi-connection request does not. curl is the tool here.
|
||
- Encoded links in the channels start as base64 strings matching
|
||
`aHR0cHM6Ly[89]...` — keep the extraction regex compatible with
|
||
`parts/grab.sh`.
|
||
|
||
## 4. Usage (what "done" looks like)
|
||
|
||
export DISCORD_TOKEN=...
|
||
python3 dlp.py <channel_id> /big/storage/warhammer/<channel-name>
|
||
|
||
Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt`
|
||
(or just re-run — state.json skips everything already marked ok).
|
||
|
||
Several channels at once (`subscriptions.json`: list of {name, channel_id,
|
||
out_dir}):
|
||
|
||
python3 dlp.py --subs subscriptions.json
|
||
|
||
Subs mode re-exports every channel by default (that is how new links get
|
||
picked up) and relies on state.json + the global store to only download
|
||
what is actually new. Use a cron/systemd timer for watch-like behavior.
|
||
|
||
Manual import of the pre-dlp.py era (URLs you know are fully downloaded,
|
||
never re-fetch them again):
|
||
|
||
python3 dlp.py --assume-done old_urls.txt
|
||
|
||
## 5. Run artifacts (all in OUT_DIR, gitignored)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `export.json` | DCE export; cached — `--fresh` re-exports |
|
||
| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
|
||
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
|
||
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
|
||
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
|
||
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. |
|
||
|
||
## 6. Rules
|
||
|
||
### Secrets and tokens
|
||
1. Never print, log, commit or echo a Discord token or GoFile account token.
|
||
Not even partial. Strip them from any output you paste into chats.
|
||
2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
|
||
It is handed to the DCE container via the `TOKEN` env var, never as a
|
||
command-line argument.
|
||
3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy
|
||
credentials into scripts, tests, or fixtures.
|
||
4. Nothing downloaded and no export/state/manifest file is ever committed.
|
||
|
||
### Failure policy (the core requirement)
|
||
5. A failure in one URL or file must **never** abort the run. Catch, record in
|
||
`state.json`, continue. The loops in `dlp.py` therefore have catch-alls —
|
||
keep them.
|
||
6. Cooldowns are the only permitted "stall": after N consecutive failures,
|
||
pause, then continue. Circuit-breakers that exit the process need a
|
||
explicit justification in the PR/commit message.
|
||
7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
|
||
Password-required gofile links are not retried (it can't help).
|
||
|
||
### Idempotency
|
||
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or**
|
||
the global store), skip existing non-empty files, write partial downloads
|
||
to `*.part` and rename on success.
|
||
9. `state.json` is the source of truth for resume; keep its schema backward
|
||
compatible when touching `state.py` (unknown keys are preserved, missing
|
||
keys defaulted). Same for the global store schema in `globalstore.py`.
|
||
10. `--assume-done` is a deliberate, manual import step for old URL lists.
|
||
Never invoke it automatically, and never let the tool mark URLs done
|
||
without a verified successful download.
|
||
|
||
### Rate limiting / network etiquette
|
||
11. One gofile guest account per run; `ensure_token()` backs off linearly
|
||
(20s × attempt). Do not raise the guest-account frequency "to make tests
|
||
pass". Testing account creation more than a couple times per hour is a
|
||
bug, not a test strategy.
|
||
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
|
||
re-verifying the 403 behavior.
|
||
|
||
### Code style
|
||
13. Python 3.12+, type hints on public functions, `logging` not `print` in
|
||
library code (prints are fine only for `--dry-run` output).
|
||
14. Stay on stdlib + `requests`. New dependencies need a real justification
|
||
and belong in `requirements.txt`.
|
||
15. `dlphelper/` modules must stay import-safe (no side effects at import).
|
||
|
||
### Validation
|
||
15. Minimum validation for any change: `python3 -m py_compile dlp.py
|
||
catalogue.py dlphelper/*.py`, plus the offline tests (`python3
|
||
tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
|
||
modules changed. Document what you could *not* test (e.g. live
|
||
gofile/0807 downloads need real links and tokens).
|
||
17. Full live runs need a Discord token from the user — ask for it, never
|
||
ask for it to be pasted into a file or committed.
|
||
|
||
### Git
|
||
18. Branch `main`, descriptive commit messages, no force-push, no history
|
||
rewriting on the shared remote.
|
||
19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
|
||
fine, deletion/renaming of existing repos requires the user.
|
||
|
||
## 7. Roadmap (future stages, keep compatible)
|
||
|
||
1. **Dedupe**: hash-based (sha256) duplicate detection across threads via
|
||
`manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already
|
||
exists via the global store; content-level (same file, different link)
|
||
comes with hashing.
|
||
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
|
||
a watch dir, cross-seeding between direct-download and torrent copies.
|
||
3. **Library organization (started)**: `catalogue.py` classifies every file
|
||
into system / faction / class and can move files into
|
||
`<library>/<System>/<Group>/<Faction>/<Class>/` with `--apply` (audited in
|
||
`catalogue_moves.jsonl`). The taxonomy lives in
|
||
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
|
||
collection - unclassified files are reported, never guessed. Manual fixes
|
||
go into an overrides JSON (`--overrides`). The library hierarchy follows
|
||
the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
|
||
working agreement for the sorting pipeline.
|
||
|
||
**Sorter matching rules (fixed 2026-10-05 after the first full run):**
|
||
the outer file name wins when it scores >= threshold; inner archive
|
||
names only vote as fallback (hash-named gofile archives need that). A
|
||
fuzzy match never jumps into another game's section: keyword-identified
|
||
systems constrain the catalog pool, and a file with neither system nor
|
||
faction keyword only adopts a catalog unit on a strong hit
|
||
(`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`.
|
||
Token-containment boosts exist for short names vs long catalog names
|
||
("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token
|
||
stoplist. The catalog deliberately covers all GW systems (needed for
|
||
catalog-agnostic use) - wrong-system placement is prevented by matching
|
||
discipline, not by shrinking the catalog.
|
||
|
||
**Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
|
||
working, verified 2026-10-04 — how it was obtained, do not re-derive):**
|
||
scraping warhammer.com directly is blocked — the shop HTML and all
|
||
search/listing endpoints (`/en-WW/shop/*`,
|
||
`/app/resources/catalog/search/*`, `/app/resources/catalog/categories`)
|
||
sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any
|
||
non-browser client, regardless of User-Agent; the sitemap declared in
|
||
robots.txt does not exist; deep paths under the allowed
|
||
`/app/resources/catalog/product/*` reach a separate commerce backend
|
||
(plain `404 Not found`) but the resource shape is unknown. What works:
|
||
the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and
|
||
its app-id plus the **public search-only API key** ship with the browser
|
||
page; the user captured both from the browser devtools ("Copy as cURL"
|
||
on one search request) and they are stored in the gitignored
|
||
`algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the
|
||
key is search-only, never commit any other Algolia key. The full
|
||
product index `prod-lazarus-product-en-ww` (~3500 products, plus one
|
||
sibling index per locale) is dumped with plain `requests`: the
|
||
public search-only key cannot use Algolia's `/browse` (403) and normal
|
||
pagination is capped at 1000 cumulative hits, so `gwshop.py`
|
||
**partitions the index on the `GameSystemsRoot` hierarchical facet** —
|
||
each branch is fetched with an explicit filter, recursing deeper only
|
||
when a branch exceeds one page. (Live count 2026-10-04: 4183 records,
|
||
of which 3491 are reachable via the facet; the ~692 left out are paints,
|
||
Black Library books, White Dwarf issues, brushes, bases and gifts that
|
||
carry no `GameSystemsRoot` path — no miniature kits among them.) Records are normalized into the
|
||
source-agnostic catalog schema (system/group/faction/subfaction/class
|
||
= the facet path, name/sku/productType attached) into
|
||
`data/gwshop_catalog.json`, which is the sorter's default catalog and
|
||
is **committed as a snapshot**; refresh with
|
||
`python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog
|
||
sources stay pluggable and source-agnostic (miniset.net — reachable
|
||
but stale, manual JSON overrides, `data/library_catalog.json`).
|
||
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
|
||
primitive version; a real watch mode (per-subscription schedules, tear-off
|
||
notifications, discord pings on new links) builds on it.
|
||
|
||
## 8. Testing notes
|
||
|
||
- `tests/` currently holds a synthetic `export.json` test for extraction
|
||
(`tests/test_extract.py`, runnable with plain `python3`). Keep tests
|
||
dependency-free and offline — they must never hit gofile/0807/Discord.
|
||
- To exercise downloads without Discord: build a `--urls-file` from known
|
||
URLs and run with a scratch OUT_DIR.
|