Ben_Kosytorz/Warhammer_DLP_Helper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: flatten the crate to the repo root and drop the python version

Browse Source 
The Rust rewrite has replaced the original python implementation, so the transition layout is no longer needed:

- move version-rust/{Cargo.toml,Cargo.lock,README.md,.gitignore,src,data} to the repo root; the crate is now the repo.
- move docs/SORTING.md and data/gwshop_catalog.json to the root.
- remove version-python/ entirely, plus the dev-only version-rust/tools/dump_warhammer_data.py (its only job was regenerating data/warhammer_data.json from python).
- rewrite AGENTS.md and README.md for the single-project layout; strip all version-python/version-rust references from code comments.
- fix the gwshop credential lookup: algolia.json now sits at CARGO_MANIFEST_DIR (crate root == repo root), not its parent.
- restore the credentials/run-output ignores in .gitignore (algolia.json, .env, *.token, downloads/, state.json, manifest.jsonl, ...).
- docs/SORTING.md: retarget the extraction/sorter/dlp examples at the dlphelper binary instead of the deleted python scripts.

cargo test: 86/86 green.
This commit is contained in:
Ben_Kosytorz
2026-10-11 09:22:41 +02:00
parent 356b9089b7
commit 6c6a3273bb
54 changed files with 331 additions and 5378 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+247 -21
View File
@@ -1,26 +1,252 @@
# AGENTS.md — Warhammer_DLP_Helper (transition layout)
# AGENTS.md — Warhammer_DLP_Helper
This repo is mid-transition. **The `main` branch still carries the original
flat python layout** — read `version-python/AGENTS.md` there (and on this
branch at `version-python/AGENTS.md`) for the full project rules: secrets,
failure policy, idempotency, rate limiting, testing. Those rules still apply.
Guidance for any AI agent (or human) working in this repository. Read this
fully before making changes.
## Layout on this branch (`version-rust`)
## 1. What this project is
- `version-python/` — the existing, working python tooling, moved here with
`git mv` (history preserved). All rules and path references in its
`AGENTS.md` resolve relative to `version-python/`.
- `version-rust/` — the Rust rewrite (cargo project) that will eventually
replace the python version. See `version-rust/README.md` for the port plan
and crate choices.
The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never
official Games Workshop assets) that are shared in Discord channels. Links in
those channels point to two hosters and are often **base64-encoded** inside
message content (obfuscated by the posting bot).
## Transition rules
The end goal is a **torrent-first platform** (think radarr/lidarr, but for
proxy files). This repo is the Rust implementation of the current stage: one
binary, `dlphelper`, that
1. Python fixes go into `version-python/` on `main` (flat layout there) and
are merged into `version-python/` here; never fork the logic in two
places.
2. The Rust version must stay compatible with the run artifacts
(`state.json`, `manifest.jsonl`, `~/.local/share/dlphelper/urls.json`) —
schemas are the contract.
3. `data/gwshop_catalog.json` (committed snapshot) is shared; the sorter in
both versions must be able to read the same catalog schema.
1. exports a Discord channel/thread (DiscordChatExporter via docker),
2. extracts base64-encoded and plain download links,
3. picks the right downloader per host automatically,
4. downloads everything with per-file tracking, retries and resume,
5. extracts the archives into a plain-file tree,
6. sorts the files into the library by system/group/faction/class/unit,
7. records a manifest that will later feed the torrent stage.
A webapp is the next stage. Deliberately **not** in scope (yet): `.torrent`
creation, tracker upload, deseeeding, rename-into-library conventions. Design
decisions should not close the door on these — especially `manifest.jsonl`
and the `--hash` option exist for dedupe/cross-seeding later.
This repository was ported from a Python implementation, which the Rust
version has now replaced (the ported modules keep `Rust port of …` provenance
notes in their headers).
## 2. Repository & environment
- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` —
**private** repo, branch `main`. Push with the credentials already stored
via git's credential store (`~/.git-credentials`); never paste them into
commands or files.
- Machine: Linux, Rust (edition 2024), docker is available. The repo lives on
**NFS**, where rustc SIGBUSes — always build/test with an explicit target
directory:
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is
upstream reading material only.
## 3. Architecture
```
Cargo.toml crate `dlphelper` (edition 2024)
src/main.rs CLI (clap subcommands: dlp|catalogue|sorter|extract|gwshop)
src/orchestrate.rs dlp pipeline: export -> extract URLs -> per-host download
src/discord.rs DCE docker export (JSON) + base64/plain URL extraction
src/gofile.rs GoFile API client (one guest account per run)
src/zero807.rs curl downloader for 0807.st (direct links + /f/<id> folder pages)
src/state.rs state.json store
src/globalstore.rs cross-run URL store (~/.local/share/dlphelper/urls.json)
src/manifest.rs manifest.jsonl appender
src/extract.rs recursive archive extraction into a plain-file tree
src/catalogue.rs scanner + keyword classifier + report writer
src/sorter.rs library sorter: inspect/classify/fuzzy-match, plan + apply
src/difflib.rs faithful CPython SequenceMatcher port (bit-identical ratios)
src/warhammer_data.rs taxonomy tables (parsed from data/warhammer_data.json)
src/gwshop.rs GW webshop catalog dump via Algolia
src/util.rs small shared helpers
data/gwshop_catalog.json committed catalog snapshot (sorter default)
data/warhammer_data.json committed taxonomy tables
docs/SORTING.md the library/sorting contract (change it before the code)
```
Data flow:
```mermaid
graph LR
A[channel id + token] --> B[DCE docker export -> export.json]
B --> C[decode base64 in message content]
C --> D{classify host}
D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files]
D -->|0807.st| F[curl with UA + Referer]
D -->|other| G[recorded as failed: unsupported host]
E --> H[downloads/gofile/...]
F --> I[downloads/0807/...]
E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
J --> K[globalstore: urls.json, marks URL done forever]
J --> L[extract -> plain-file tree]
L --> M[sorter -> library, hardlinks + audit]
```
Key facts about the hosters (learned the hard way, do not re-learn):
- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`.
Creation is aggressively rate-limited — create **at most one account per
run**. Content listing needs `Authorization: Bearer <token>`, an
`X-Website-Token` header (sha256 over `UA::en-US::<token>::<4h-window>::seed`,
see `src/gofile.rs`), and downloads need `Cookie: accountToken=<token>` plus
Referer/Origin headers.
- **0807.st**: serves files directly but 403s anything that doesn't look like
a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's
default multi-connection request does not. curl is the tool here.
Folder pages (`https://0807.st/f/<id>`) are expanded: the page is parsed and
each listed file is fetched from `https://0807.st/<id>?dl=1` into
`downloads/0807/<page-id>/`. The folder `.zip` export is refused by the site
("folder too large"), so per-file download is the only route in.
- Encoded links in the channels start as base64 strings matching
`aHR0cHM6Ly[89]...`.
## 4. Usage (what "done" looks like)
export DISCORD_TOKEN=...
dlphelper dlp <channel_id> /big/storage/warhammer/<channel-name> --hash
`--since 5d` (or `12h`, `2026-10-04`) limits a run to recent messages — handy
for incremental updates. Re-runs are safe: `state.json` and the global store
skip everything already marked ok.
Retry only failures:
dlphelper dlp <id> <out> --urls-file <out>/failed_urls.txt
Several channels at once (`subscriptions.json`: list of `{name, channel_id,
out_dir}`):
dlphelper dlp --subs subscriptions.json
Subs mode re-exports every channel by default (that is how new links get
picked up) and relies on `state.json` + the global store to only download what
is actually new. Use a cron/systemd timer for watch-like behavior.
Extract and sort:
dlphelper extract <out>/downloads/0807 <out>/downloads/gofile --out <out>/extracted
dlphelper sorter <out>/extracted --library <library> --apply
Manual import of old URL lists (known fully downloaded, never re-fetch):
dlphelper dlp --assume-done old_urls.txt
## 5. Run artifacts (all in OUT_DIR, gitignored)
| File | Purpose |
|------|---------|
| `export.json` | DCE export; cached — `--fresh` re-exports |
| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset |
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
| `extracted/extract_manifest.jsonl` | one line per extraction `{archive, dest, unit}` (feeds `sorter --provenance`) |
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR. `--global-state` moves it, `--no-global` disables it for a run. |
Sorter bookkeeping (audit + low-confidence doc) is written to
`<library>/../_logs/` by default — never inside the library (see §7).
## 6. Rules
### Secrets and tokens
1. Never print, log, commit or echo a Discord token or GoFile account token.
Not even partial. Strip them from any output you paste into chats.
2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt.
It is handed to the DCE container via the `TOKEN` env var, never as a
command-line argument.
3. `~/.git-credentials` may only be read by tooling that needs it; never copy
credentials into scripts, tests, or fixtures.
4. Nothing downloaded and no export/state/manifest file is ever committed.
### Failure policy (the core requirement)
5. A failure in one URL or file must **never** abort the run. Record it in
`state.json` and continue. The per-URL loops in `src/orchestrate.rs`
therefore have catch-alls — keep them.
6. Cooldowns are the only permitted "stall": after N consecutive failures,
pause, then continue. Circuit-breakers that exit the process need an
explicit justification in the commit message.
7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts.
Password-required gofile links are not retried (it can't help); dead links
(HTTP 404/410) are never retried.
### Idempotency
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json`
**or** the global store), skip existing non-empty files, write partial
downloads to `*.part` and rename on success.
9. `state.json` is the source of truth for resume; keep its schema backward
compatible (unknown keys preserved, missing keys defaulted). Same for the
global store schema.
10. `--assume-done` is a deliberate, manual import step for old URL lists.
Never invoke it automatically, and never let the tool mark URLs done
without a verified successful download.
### Rate limiting / network etiquette
11. One gofile guest account per run; `ensure_token()` backs off linearly
(20s × attempt). Do not raise the guest-account frequency "to make tests
pass". Testing account creation more than a couple times per hour is a
bug, not a test strategy.
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
re-verifying the 403 behavior.
### Code style
13. Rust 2024, idiomatic. Use `log` (not `println!`) in library modules;
prints are fine only for CLI output (plan/dry-run).
14. Keep new dependencies justified — the crate already covers HTTP
(`reqwest`), archives (`sevenz-rust2`, `zip`, `unrar`), fuzzy matching
(the in-tree `difflib` port, **not** an external crate) and CLI (`clap`).
The tar/gz/bz2/xz paths shell out to the system binaries deliberately.
15. `src/difflib.rs` must stay bit-identical to CPython `difflib` — the sorter
thresholds depend on exact ratios; do not "simplify" it.
### Validation
16. Minimum validation for any change: `CARGO_TARGET_DIR=/var/tmp/dlphelper-target
cargo test` (the offline suite must stay green). Document what you could
*not* test (live gofile/0807/Discord paths need real links and tokens).
17. Full live runs need a Discord token from the user — ask for it, never ask
for it to be pasted into a file or committed.
### Git
18. Branch `main`, descriptive commit messages, no force-push, no history
rewriting on the shared remote.
## 7. Sorter & library structure
`docs/SORTING.md` is the working agreement for the library hierarchy and the
sorting pipeline — **change it first, then the code**. Summary:
- Hierarchy: `<library>/<System>/<Group>/<Faction>/<Subfaction?>/<Class>/<Unit>/<file>`.
Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- Matching rules are fixed: the outer file name wins when it scores ≥
threshold; inner archive names only vote as fallback. A fuzzy match never
jumps into another game's section (keyword systems constrain the catalog
pool; cross-system adoption needs `CROSS_SYSTEM_MIN = 0.75` and token
containment). Deepest-matching-dir wins (junk dir names are never
candidates). Weak hits ≥ `WEAK_MIN = 0.50` are accepted but flagged
low-confidence. Tokenless normal adoption needs `+0.08`
(`TOKENLESS_BONUS`).
- `DROPPED_SYSTEMS` (default `{Middle-Earth}`) filters catalog entries at load
and counts keyword hits as unclassified; `--drop-system`/`--keep-system`
adjust. The committed catalog snapshot keeps all systems.
- Catalog source: the GW webshop via Algolia (`src/gwshop.rs`). The app-id +
public **search-only** key live in the gitignored `algolia.json` (or
`ALGOLIA_APP_ID`/`ALGOLIA_API_KEY`); never commit any other Algolia key.
`data/gwshop_catalog.json` is the committed snapshot and the sorter default.
- Bookkeeping (`sorter_applied.jsonl`, `sorter_low_confidence.txt`) is written
to `<library>/../_logs/` by default — never inside the library, which may be
a seeded torrent whose file set is frozen.
## 8. Testing notes
- Tests live inline (`#[cfg(test)]`) in each module and run with `cargo test`.
Keep them dependency-free and offline — they must never hit
gofile/0807/Discord.
- `tempfile` tests stage under `/var/tmp` when available, because `/tmp` is a
small tmpfs that trips the extractor's free-space guard (`MIN_FREE_BYTES`).
- To exercise downloads without Discord: build a `--urls-file` from known URLs
and run with a scratch OUT_DIR.