diff --git a/AGENTS.md b/AGENTS.md index 05694f4..74a0334 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,310 +1,26 @@ -# AGENTS.md — Warhammer_DLP_Helper +# AGENTS.md — Warhammer_DLP_Helper (transition layout) -Guidance for any AI agent (or human) working in this repository. Read this -fully before making changes. +This repo is mid-transition. **The `main` branch still carries the original +flat python layout** — read `version-python/AGENTS.md` there (and on this +branch at `version-python/AGENTS.md`) for the full project rules: secrets, +failure policy, idempotency, rate limiting, testing. Those rules still apply. -## 1. What this project is +## Layout on this branch (`version-rust`) -The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never -official Games Workshop assets) that are shared in Discord channels. Links in -those channels point to two hosters and are often **base64-encoded** inside -message content (obfuscated by the posting bot). +- `version-python/` — the existing, working python tooling, moved here with + `git mv` (history preserved). All rules and path references in its + `AGENTS.md` resolve relative to `version-python/`. +- `version-rust/` — the Rust rewrite (cargo project) that will eventually + replace the python version. See `version-rust/README.md` for the port plan + and crate choices. -The end goal is a **torrent-first platform** (think radarr/lidarr, but for -proxy files). This repo is the current stage of that journey: a tool that +## Transition rules -1. exports a Discord channel/thread (DiscordChatExporter via docker), -2. extracts base64-encoded and plain download links, -3. picks the right downloader per host automatically, -4. downloads everything with per-file tracking, retries and resume, -5. records a manifest that will later feed the torrent stage. - -Deliberately **not** in scope (yet): .torrent creation, tracker upload, -deseeding, rename-into-library conventions. Design decisions should not close -the door on these — especially `manifest.jsonl` and the `--hash` option exist -for dedupe/cross-seeding later. - -## 2. Repository & environment - -- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` — - **private** repo, branch `main`. Push with the credentials already stored - via git's credential store (`~/.git-credentials`); never paste them into - commands or files. -- The server is Gitea 1.27. To create *another* repo, use - `python3 tools/new_repo.py list|create ` (reads `~/.git-credentials` - itself, never prints secrets). Never delete or overwrite existing repos. -- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only - runtime dependency (see `requirements.txt`), docker is available. -- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is - upstream reading material only. - -## 3. Architecture - -``` -dlp.py CLI orchestrator (entry point) -catalogue.py CLI: classify the collection into system/faction/class -sorter.py CLI: fuzzysort files into the library (plan/apply, - hardlinks, default catalog data/gwshop_catalog.json) -dlphelper/ - discord.py DCE docker export (JSON) + base64/plain URL extraction - gofile.py GoFile API client (one guest account per run) - zero807.py curl-based downloader for 0807.st direct links - state.py state.json store + manifest.jsonl appender - globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json) - catalogue.py scanner + keyword classifier + report writer - warhammer_data.py taxonomy: GW-named systems/factions, GW shop - categories as classes, GROUP_OF_FACTION map - gwshop.py GW webshop catalog dump via Algolia (see §7.3) - sorter.py library sorter: inspect/classify/fuzzy-match - against the catalog, plan + hardlink apply, - optional --provenance unit mode for extracted trees -parts/grab.sh legacy groundstone script (kept, not used by dlp.py) -tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself) -tools/debug_gofile.py inspect raw gofile API response for one content id -tools/extract_all.py recursive archive extraction into a plain-file tree - (extract_manifest.jsonl feeds the sorter's - --provenance unit mode; failures go to .broken/; - 7z-direct-to-CIFS failures fall back to a - /var/tmp extraction with cp437 name repair; - AppleDouble/__MACOSX junk is dropped) -tests/test_extract.py offline extraction test (plain python3, no deps) -tests/test_catalogue.py offline classifier test -``` - -Data flow: - -```mermaid -graph LR - A[channel id + token] --> B[DCE docker export -> export.json] - B --> C[decode base64 in message content] - C --> D{classify host} - D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files] - D -->|0807.st| F[curl with UA + Referer] - D -->|other| G[recorded as failed: unsupported host] - E --> H[downloads/gofile/...] - F --> I[downloads/0807/...] - E & F --> J[state.json + manifest.jsonl + failed_urls.txt] - J --> K[globalstore: urls.json, marks URL done forever] -``` - -Key facts about the hosters (learned the hard way, do not re-learn): - -- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`. - Creation is aggressively rate-limited — create **at most one account per - run**. Content listing needs `Authorization: Bearer `, an - `X-Website-Token` header (sha256 over `UA::en-US::::<4h-window>::seed`, - see `gofile.py`), and downloads need `Cookie: accountToken=` plus - Referer/Origin headers. -- **0807.st**: serves files directly but 403s anything that doesn't look like - a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's - default multi-connection request does not. curl is the tool here. -- Encoded links in the channels start as base64 strings matching - `aHR0cHM6Ly[89]...` — keep the extraction regex compatible with - `parts/grab.sh`. - -## 4. Usage (what "done" looks like) - - export DISCORD_TOKEN=... - python3 dlp.py /big/storage/warhammer/ - -Retry only failures: `python3 dlp.py --urls-file /failed_urls.txt` -(or just re-run — state.json skips everything already marked ok). - -Several channels at once (`subscriptions.json`: list of {name, channel_id, -out_dir}): - - python3 dlp.py --subs subscriptions.json - -Subs mode re-exports every channel by default (that is how new links get -picked up) and relies on state.json + the global store to only download -what is actually new. Use a cron/systemd timer for watch-like behavior. - -Manual import of the pre-dlp.py era (URLs you know are fully downloaded, -never re-fetch them again): - - python3 dlp.py --assume-done old_urls.txt - -## 5. Run artifacts (all in OUT_DIR, gitignored) - -| File | Purpose | -|------|---------| -| `export.json` | DCE export; cached — `--fresh` re-exports | -| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset | -| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` | -| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) | -| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" | -| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. | - -## 6. Rules - -### Secrets and tokens -1. Never print, log, commit or echo a Discord token or GoFile account token. - Not even partial. Strip them from any output you paste into chats. -2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt. - It is handed to the DCE container via the `TOKEN` env var, never as a - command-line argument. -3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy - credentials into scripts, tests, or fixtures. -4. Nothing downloaded and no export/state/manifest file is ever committed. - -### Failure policy (the core requirement) -5. A failure in one URL or file must **never** abort the run. Catch, record in - `state.json`, continue. The loops in `dlp.py` therefore have catch-alls — - keep them. -6. Cooldowns are the only permitted "stall": after N consecutive failures, - pause, then continue. Circuit-breakers that exit the process need a - explicit justification in the PR/commit message. -7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts. - Password-required gofile links are not retried (it can't help). - -### Idempotency -8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or** - the global store), skip existing non-empty files, write partial downloads - to `*.part` and rename on success. -9. `state.json` is the source of truth for resume; keep its schema backward - compatible when touching `state.py` (unknown keys are preserved, missing - keys defaulted). Same for the global store schema in `globalstore.py`. -10. `--assume-done` is a deliberate, manual import step for old URL lists. - Never invoke it automatically, and never let the tool mark URLs done - without a verified successful download. - -### Rate limiting / network etiquette -11. One gofile guest account per run; `ensure_token()` backs off linearly - (20s × attempt). Do not raise the guest-account frequency "to make tests - pass". Testing account creation more than a couple times per hour is a - bug, not a test strategy. -12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without - re-verifying the 403 behavior. - -### Code style -13. Python 3.12+, type hints on public functions, `logging` not `print` in - library code (prints are fine only for `--dry-run` output). -14. Stay on stdlib + `requests`. New dependencies need a real justification - and belong in `requirements.txt`. -15. `dlphelper/` modules must stay import-safe (no side effects at import). - -### Validation -15. Minimum validation for any change: `python3 -m py_compile dlp.py - catalogue.py dlphelper/*.py`, plus the offline tests (`python3 - tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched - modules changed. Document what you could *not* test (e.g. live - gofile/0807 downloads need real links and tokens). -17. Full live runs need a Discord token from the user — ask for it, never - ask for it to be pasted into a file or committed. - -### Git -18. Branch `main`, descriptive commit messages, no force-push, no history - rewriting on the shared remote. -19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is - fine, deletion/renaming of existing repos requires the user. - -## 7. Roadmap (future stages, keep compatible) - -1. **Dedupe**: hash-based (sha256) duplicate detection across threads via - `manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already - exists via the global store; content-level (same file, different link) - comes with hashing. -2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into - a watch dir, cross-seeding between direct-download and torrent copies. -3. **Library organization (started)**: `catalogue.py` classifies every file - into system / faction / class and can move files into - `/////` with `--apply` (audited in - `catalogue_moves.jsonl`). The taxonomy lives in - `dlphelper/warhammer_data.py`; it is keyword-based and grows with the - collection - unclassified files are reported, never guessed. Manual fixes - go into an overrides JSON (`--overrides`). The library hierarchy follows - the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the - working agreement for the sorting pipeline. - - **Sorter matching rules (fixed 2026-10-05 after the first full run):** - the outer file name wins when it scores >= threshold; inner archive - names only vote as fallback (hash-named gofile archives need that). A - fuzzy match never jumps into another game's section: keyword-identified - systems constrain the catalog pool, and a file with neither system nor - faction keyword only adopts a catalog unit on a strong hit - (`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`. - Token-containment boosts exist for short names vs long catalog names - ("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token - stoplist. - - **Refined 2026-10-07 (dir re-homing + weak tier + non-warhammer drop):** - - **Deepest-matching-dir wins**: ancestor directory names (inside the - archive in provenance mode, real dirs in manual trees) compete with - the pack/file name; the best accepted match re-homes the whole - subtree. Junk dir names (files/images/HEADS/pure numbers/lone generic - words) are never candidates. Fixes multi-unit sculptor packs (one - "Retro Elvez" archive ≈ 20 Eldar units no longer all land in one - keyword class). - - **Weak tier**: when nothing reaches the threshold, a hit >= `WEAK_MIN` - (0.50) is accepted but flagged low confidence - `[w]` in the plan, - `confidence: low` in the audit, listed in - `sorter_low_confidence.txt` in the log dir (see below). Weak and tokenless adoptions - require a shared non-generic token; tokenless normal adoption needs - +0.08 (`TOKENLESS_BONUS`); cross-system adoption additionally needs - token containment (`_token_contained`), because char-ratio lies - ("aeldari rangers" vs "dwarf rangers" = 0.786). - - **Non-warhammer systems dropped**: `DROPPED_SYSTEMS` (default - `{Middle-Earth}`) - catalog entries filtered at load, keyword hits - count as unclassified; files stay in `_unsorted`. `--drop-system`/ - `--keep-system` adjust. The catalog snapshot keeps all systems. - - **Probe votes on full inner paths** (2026-10-07, after run 5): pack - probes clean the whole relative path, so inner dir names vote - ("Catachan/Jungle Fighters/" resolves Astra Militarum). No - single-token containment boost for inner votes (a root file named - "lieutenant" once claimed ~7000 files into one Iron Hands unit); - full-catalog and different-faction adoptions need token - containment (`_contained_somewhere`). - - - **Bookkeeping location (fixed 2026-10-08)**: the audit and the - low-confidence doc are written to `/../_logs/` by default - (`--audit` overrides) - never inside the library itself, because the - library may be a seeded torrent whose file set is frozen. - - The catalog deliberately covers all GW systems (needed for - catalog-agnostic use) - wrong-system placement is prevented by matching - discipline and the drop filter, not by shrinking the catalog. - - **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`, - working, verified 2026-10-04 — how it was obtained, do not re-derive):** - scraping warhammer.com directly is blocked — the shop HTML and all - search/listing endpoints (`/en-WW/shop/*`, - `/app/resources/catalog/search/*`, `/app/resources/catalog/categories`) - sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any - non-browser client, regardless of User-Agent; the sitemap declared in - robots.txt does not exist; deep paths under the allowed - `/app/resources/catalog/product/*` reach a separate commerce backend - (plain `404 Not found`) but the resource shape is unknown. What works: - the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and - its app-id plus the **public search-only API key** ship with the browser - page; the user captured both from the browser devtools ("Copy as cURL" - on one search request) and they are stored in the gitignored - `algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the - key is search-only, never commit any other Algolia key. The full - product index `prod-lazarus-product-en-ww` (~3500 products, plus one - sibling index per locale) is dumped with plain `requests`: the - public search-only key cannot use Algolia's `/browse` (403) and normal - pagination is capped at 1000 cumulative hits, so `gwshop.py` - **partitions the index on the `GameSystemsRoot` hierarchical facet** — - each branch is fetched with an explicit filter, recursing deeper only - when a branch exceeds one page. (Live count 2026-10-04: 4183 records, - of which 3491 are reachable via the facet; the ~692 left out are paints, - Black Library books, White Dwarf issues, brushes, bases and gifts that - carry no `GameSystemsRoot` path — no miniature kits among them.) Records are normalized into the - source-agnostic catalog schema (system/group/faction/subfaction/class - = the facet path, name/sku/productType attached) into - `data/gwshop_catalog.json`, which is the sorter's default catalog and - is **committed as a snapshot**; refresh with - `python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog - sources stay pluggable and source-agnostic (miniset.net — reachable - but stale, manual JSON overrides, `data/library_catalog.json`). -4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a - primitive version; a real watch mode (per-subscription schedules, tear-off - notifications, discord pings on new links) builds on it. - -## 8. Testing notes - -- `tests/` currently holds a synthetic `export.json` test for extraction - (`tests/test_extract.py`, runnable with plain `python3`). Keep tests - dependency-free and offline — they must never hit gofile/0807/Discord. -- To exercise downloads without Discord: build a `--urls-file` from known - URLs and run with a scratch OUT_DIR. +1. Python fixes go into `version-python/` on `main` (flat layout there) and + are merged into `version-python/` here; never fork the logic in two + places. +2. The Rust version must stay compatible with the run artifacts + (`state.json`, `manifest.jsonl`, `~/.local/share/dlphelper/urls.json`) — + schemas are the contract. +3. `data/gwshop_catalog.json` (committed snapshot) is shared; the sorter in + both versions must be able to read the same catalog schema. diff --git a/.gitignore b/version-python/.gitignore similarity index 100% rename from .gitignore rename to version-python/.gitignore diff --git a/version-python/AGENTS.md b/version-python/AGENTS.md new file mode 100644 index 0000000..05694f4 --- /dev/null +++ b/version-python/AGENTS.md @@ -0,0 +1,310 @@ +# AGENTS.md — Warhammer_DLP_Helper + +Guidance for any AI agent (or human) working in this repository. Read this +fully before making changes. + +## 1. What this project is + +The user collects **Warhammer proxy models** (3rd-party / fan sculpts, never +official Games Workshop assets) that are shared in Discord channels. Links in +those channels point to two hosters and are often **base64-encoded** inside +message content (obfuscated by the posting bot). + +The end goal is a **torrent-first platform** (think radarr/lidarr, but for +proxy files). This repo is the current stage of that journey: a tool that + +1. exports a Discord channel/thread (DiscordChatExporter via docker), +2. extracts base64-encoded and plain download links, +3. picks the right downloader per host automatically, +4. downloads everything with per-file tracking, retries and resume, +5. records a manifest that will later feed the torrent stage. + +Deliberately **not** in scope (yet): .torrent creation, tracker upload, +deseeding, rename-into-library conventions. Design decisions should not close +the door on these — especially `manifest.jsonl` and the `--hash` option exist +for dedupe/cross-seeding later. + +## 2. Repository & environment + +- Remote: `git.dieterunterstrich.de/Ben_Kosytorz/Warhammer_DLP_Helper` — + **private** repo, branch `main`. Push with the credentials already stored + via git's credential store (`~/.git-credentials`); never paste them into + commands or files. +- The server is Gitea 1.27. To create *another* repo, use + `python3 tools/new_repo.py list|create ` (reads `~/.git-credentials` + itself, never prints secrets). Never delete or overwrite existing repos. +- Machine: Linux, Python 3.12+ (3.14 available), `requests` is the only + runtime dependency (see `requirements.txt`), docker is available. +- Reference clone: `~/GoFileDownloader` (Lysagxra) — do not modify it; it is + upstream reading material only. + +## 3. Architecture + +``` +dlp.py CLI orchestrator (entry point) +catalogue.py CLI: classify the collection into system/faction/class +sorter.py CLI: fuzzysort files into the library (plan/apply, + hardlinks, default catalog data/gwshop_catalog.json) +dlphelper/ + discord.py DCE docker export (JSON) + base64/plain URL extraction + gofile.py GoFile API client (one guest account per run) + zero807.py curl-based downloader for 0807.st direct links + state.py state.json store + manifest.jsonl appender + globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json) + catalogue.py scanner + keyword classifier + report writer + warhammer_data.py taxonomy: GW-named systems/factions, GW shop + categories as classes, GROUP_OF_FACTION map + gwshop.py GW webshop catalog dump via Algolia (see §7.3) + sorter.py library sorter: inspect/classify/fuzzy-match + against the catalog, plan + hardlink apply, + optional --provenance unit mode for extracted trees +parts/grab.sh legacy groundstone script (kept, not used by dlp.py) +tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself) +tools/debug_gofile.py inspect raw gofile API response for one content id +tools/extract_all.py recursive archive extraction into a plain-file tree + (extract_manifest.jsonl feeds the sorter's + --provenance unit mode; failures go to .broken/; + 7z-direct-to-CIFS failures fall back to a + /var/tmp extraction with cp437 name repair; + AppleDouble/__MACOSX junk is dropped) +tests/test_extract.py offline extraction test (plain python3, no deps) +tests/test_catalogue.py offline classifier test +``` + +Data flow: + +```mermaid +graph LR + A[channel id + token] --> B[DCE docker export -> export.json] + B --> C[decode base64 in message content] + C --> D{classify host} + D -->|gofile.io| E[GoFileClient: 1 guest account, list contents, download files] + D -->|0807.st| F[curl with UA + Referer] + D -->|other| G[recorded as failed: unsupported host] + E --> H[downloads/gofile/...] + F --> I[downloads/0807/...] + E & F --> J[state.json + manifest.jsonl + failed_urls.txt] + J --> K[globalstore: urls.json, marks URL done forever] +``` + +Key facts about the hosters (learned the hard way, do not re-learn): + +- **gofile.io**: guest accounts are created via `POST api.gofile.io/accounts`. + Creation is aggressively rate-limited — create **at most one account per + run**. Content listing needs `Authorization: Bearer `, an + `X-Website-Token` header (sha256 over `UA::en-US::::<4h-window>::seed`, + see `gofile.py`), and downloads need `Cookie: accountToken=` plus + Referer/Origin headers. +- **0807.st**: serves files directly but 403s anything that doesn't look like + a browser. `curl -A "Mozilla/5.0" -e "https://0807.st/"` works; aria2c's + default multi-connection request does not. curl is the tool here. +- Encoded links in the channels start as base64 strings matching + `aHR0cHM6Ly[89]...` — keep the extraction regex compatible with + `parts/grab.sh`. + +## 4. Usage (what "done" looks like) + + export DISCORD_TOKEN=... + python3 dlp.py /big/storage/warhammer/ + +Retry only failures: `python3 dlp.py --urls-file /failed_urls.txt` +(or just re-run — state.json skips everything already marked ok). + +Several channels at once (`subscriptions.json`: list of {name, channel_id, +out_dir}): + + python3 dlp.py --subs subscriptions.json + +Subs mode re-exports every channel by default (that is how new links get +picked up) and relies on state.json + the global store to only download +what is actually new. Use a cron/systemd timer for watch-like behavior. + +Manual import of the pre-dlp.py era (URLs you know are fully downloaded, +never re-fetch them again): + + python3 dlp.py --assume-done old_urls.txt + +## 5. Run artifacts (all in OUT_DIR, gitignored) + +| File | Purpose | +|------|---------| +| `export.json` | DCE export; cached — `--fresh` re-exports | +| `state.json` | url → {status, attempts, error, files[]}; corrupt files are backed up and reset | +| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` | +| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) | +| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" | +| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. | + +## 6. Rules + +### Secrets and tokens +1. Never print, log, commit or echo a Discord token or GoFile account token. + Not even partial. Strip them from any output you paste into chats. +2. The Discord token goes: `--token` flag → `$DISCORD_TOKEN` → hidden prompt. + It is handed to the DCE container via the `TOKEN` env var, never as a + command-line argument. +3. `~/.git-credentials` may only be read by `tools/new_repo.py`. Never copy + credentials into scripts, tests, or fixtures. +4. Nothing downloaded and no export/state/manifest file is ever committed. + +### Failure policy (the core requirement) +5. A failure in one URL or file must **never** abort the run. Catch, record in + `state.json`, continue. The loops in `dlp.py` therefore have catch-alls — + keep them. +6. Cooldowns are the only permitted "stall": after N consecutive failures, + pause, then continue. Circuit-breakers that exit the process need a + explicit justification in the PR/commit message. +7. Retries: exponential-ish (`retry_delay * attempt`), default 3 attempts. + Password-required gofile links are not retried (it can't help). + +### Idempotency +8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or** + the global store), skip existing non-empty files, write partial downloads + to `*.part` and rename on success. +9. `state.json` is the source of truth for resume; keep its schema backward + compatible when touching `state.py` (unknown keys are preserved, missing + keys defaulted). Same for the global store schema in `globalstore.py`. +10. `--assume-done` is a deliberate, manual import step for old URL lists. + Never invoke it automatically, and never let the tool mark URLs done + without a verified successful download. + +### Rate limiting / network etiquette +11. One gofile guest account per run; `ensure_token()` backs off linearly + (20s × attempt). Do not raise the guest-account frequency "to make tests + pass". Testing account creation more than a couple times per hour is a + bug, not a test strategy. +12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without + re-verifying the 403 behavior. + +### Code style +13. Python 3.12+, type hints on public functions, `logging` not `print` in + library code (prints are fine only for `--dry-run` output). +14. Stay on stdlib + `requests`. New dependencies need a real justification + and belong in `requirements.txt`. +15. `dlphelper/` modules must stay import-safe (no side effects at import). + +### Validation +15. Minimum validation for any change: `python3 -m py_compile dlp.py + catalogue.py dlphelper/*.py`, plus the offline tests (`python3 + tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched + modules changed. Document what you could *not* test (e.g. live + gofile/0807 downloads need real links and tokens). +17. Full live runs need a Discord token from the user — ask for it, never + ask for it to be pasted into a file or committed. + +### Git +18. Branch `main`, descriptive commit messages, no force-push, no history + rewriting on the shared remote. +19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is + fine, deletion/renaming of existing repos requires the user. + +## 7. Roadmap (future stages, keep compatible) + +1. **Dedupe**: hash-based (sha256) duplicate detection across threads via + `manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already + exists via the global store; content-level (same file, different link) + comes with hashing. +2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into + a watch dir, cross-seeding between direct-download and torrent copies. +3. **Library organization (started)**: `catalogue.py` classifies every file + into system / faction / class and can move files into + `/////` with `--apply` (audited in + `catalogue_moves.jsonl`). The taxonomy lives in + `dlphelper/warhammer_data.py`; it is keyword-based and grows with the + collection - unclassified files are reported, never guessed. Manual fixes + go into an overrides JSON (`--overrides`). The library hierarchy follows + the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the + working agreement for the sorting pipeline. + + **Sorter matching rules (fixed 2026-10-05 after the first full run):** + the outer file name wins when it scores >= threshold; inner archive + names only vote as fallback (hash-named gofile archives need that). A + fuzzy match never jumps into another game's section: keyword-identified + systems constrain the catalog pool, and a file with neither system nor + faction keyword only adopts a catalog unit on a strong hit + (`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`. + Token-containment boosts exist for short names vs long catalog names + ("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token + stoplist. + + **Refined 2026-10-07 (dir re-homing + weak tier + non-warhammer drop):** + - **Deepest-matching-dir wins**: ancestor directory names (inside the + archive in provenance mode, real dirs in manual trees) compete with + the pack/file name; the best accepted match re-homes the whole + subtree. Junk dir names (files/images/HEADS/pure numbers/lone generic + words) are never candidates. Fixes multi-unit sculptor packs (one + "Retro Elvez" archive ≈ 20 Eldar units no longer all land in one + keyword class). + - **Weak tier**: when nothing reaches the threshold, a hit >= `WEAK_MIN` + (0.50) is accepted but flagged low confidence - `[w]` in the plan, + `confidence: low` in the audit, listed in + `sorter_low_confidence.txt` in the log dir (see below). Weak and tokenless adoptions + require a shared non-generic token; tokenless normal adoption needs + +0.08 (`TOKENLESS_BONUS`); cross-system adoption additionally needs + token containment (`_token_contained`), because char-ratio lies + ("aeldari rangers" vs "dwarf rangers" = 0.786). + - **Non-warhammer systems dropped**: `DROPPED_SYSTEMS` (default + `{Middle-Earth}`) - catalog entries filtered at load, keyword hits + count as unclassified; files stay in `_unsorted`. `--drop-system`/ + `--keep-system` adjust. The catalog snapshot keeps all systems. + - **Probe votes on full inner paths** (2026-10-07, after run 5): pack + probes clean the whole relative path, so inner dir names vote + ("Catachan/Jungle Fighters/" resolves Astra Militarum). No + single-token containment boost for inner votes (a root file named + "lieutenant" once claimed ~7000 files into one Iron Hands unit); + full-catalog and different-faction adoptions need token + containment (`_contained_somewhere`). + + - **Bookkeeping location (fixed 2026-10-08)**: the audit and the + low-confidence doc are written to `/../_logs/` by default + (`--audit` overrides) - never inside the library itself, because the + library may be a seeded torrent whose file set is frozen. + + The catalog deliberately covers all GW systems (needed for + catalog-agnostic use) - wrong-system placement is prevented by matching + discipline and the drop filter, not by shrinking the catalog. + + **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`, + working, verified 2026-10-04 — how it was obtained, do not re-derive):** + scraping warhammer.com directly is blocked — the shop HTML and all + search/listing endpoints (`/en-WW/shop/*`, + `/app/resources/catalog/search/*`, `/app/resources/catalog/categories`) + sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any + non-browser client, regardless of User-Agent; the sitemap declared in + robots.txt does not exist; deep paths under the allowed + `/app/resources/catalog/product/*` reach a separate commerce backend + (plain `404 Not found`) but the resource shape is unknown. What works: + the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and + its app-id plus the **public search-only API key** ship with the browser + page; the user captured both from the browser devtools ("Copy as cURL" + on one search request) and they are stored in the gitignored + `algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the + key is search-only, never commit any other Algolia key. The full + product index `prod-lazarus-product-en-ww` (~3500 products, plus one + sibling index per locale) is dumped with plain `requests`: the + public search-only key cannot use Algolia's `/browse` (403) and normal + pagination is capped at 1000 cumulative hits, so `gwshop.py` + **partitions the index on the `GameSystemsRoot` hierarchical facet** — + each branch is fetched with an explicit filter, recursing deeper only + when a branch exceeds one page. (Live count 2026-10-04: 4183 records, + of which 3491 are reachable via the facet; the ~692 left out are paints, + Black Library books, White Dwarf issues, brushes, bases and gifts that + carry no `GameSystemsRoot` path — no miniature kits among them.) Records are normalized into the + source-agnostic catalog schema (system/group/faction/subfaction/class + = the facet path, name/sku/productType attached) into + `data/gwshop_catalog.json`, which is the sorter's default catalog and + is **committed as a snapshot**; refresh with + `python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog + sources stay pluggable and source-agnostic (miniset.net — reachable + but stale, manual JSON overrides, `data/library_catalog.json`). +4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a + primitive version; a real watch mode (per-subscription schedules, tear-off + notifications, discord pings on new links) builds on it. + +## 8. Testing notes + +- `tests/` currently holds a synthetic `export.json` test for extraction + (`tests/test_extract.py`, runnable with plain `python3`). Keep tests + dependency-free and offline — they must never hit gofile/0807/Discord. +- To exercise downloads without Discord: build a `--urls-file` from known + URLs and run with a scratch OUT_DIR. diff --git a/README.md b/version-python/README.md similarity index 100% rename from README.md rename to version-python/README.md diff --git a/catalogue.py b/version-python/catalogue.py similarity index 100% rename from catalogue.py rename to version-python/catalogue.py diff --git a/data/gwshop_catalog.json b/version-python/data/gwshop_catalog.json similarity index 100% rename from data/gwshop_catalog.json rename to version-python/data/gwshop_catalog.json diff --git a/data/library_catalog.json b/version-python/data/library_catalog.json similarity index 100% rename from data/library_catalog.json rename to version-python/data/library_catalog.json diff --git a/dlp.py b/version-python/dlp.py similarity index 100% rename from dlp.py rename to version-python/dlp.py diff --git a/dlphelper/__init__.py b/version-python/dlphelper/__init__.py similarity index 100% rename from dlphelper/__init__.py rename to version-python/dlphelper/__init__.py diff --git a/dlphelper/catalogue.py b/version-python/dlphelper/catalogue.py similarity index 100% rename from dlphelper/catalogue.py rename to version-python/dlphelper/catalogue.py diff --git a/dlphelper/discord.py b/version-python/dlphelper/discord.py similarity index 100% rename from dlphelper/discord.py rename to version-python/dlphelper/discord.py diff --git a/dlphelper/globalstore.py b/version-python/dlphelper/globalstore.py similarity index 100% rename from dlphelper/globalstore.py rename to version-python/dlphelper/globalstore.py diff --git a/dlphelper/gofile.py b/version-python/dlphelper/gofile.py similarity index 100% rename from dlphelper/gofile.py rename to version-python/dlphelper/gofile.py diff --git a/dlphelper/gwshop.py b/version-python/dlphelper/gwshop.py similarity index 100% rename from dlphelper/gwshop.py rename to version-python/dlphelper/gwshop.py diff --git a/dlphelper/sorter.py b/version-python/dlphelper/sorter.py similarity index 100% rename from dlphelper/sorter.py rename to version-python/dlphelper/sorter.py diff --git a/dlphelper/state.py b/version-python/dlphelper/state.py similarity index 100% rename from dlphelper/state.py rename to version-python/dlphelper/state.py diff --git a/dlphelper/warhammer_data.py b/version-python/dlphelper/warhammer_data.py similarity index 100% rename from dlphelper/warhammer_data.py rename to version-python/dlphelper/warhammer_data.py diff --git a/dlphelper/zero807.py b/version-python/dlphelper/zero807.py similarity index 100% rename from dlphelper/zero807.py rename to version-python/dlphelper/zero807.py diff --git a/docs/SORTING.md b/version-python/docs/SORTING.md similarity index 100% rename from docs/SORTING.md rename to version-python/docs/SORTING.md diff --git a/parts/grab.sh b/version-python/parts/grab.sh similarity index 100% rename from parts/grab.sh rename to version-python/parts/grab.sh diff --git a/requirements.txt b/version-python/requirements.txt similarity index 100% rename from requirements.txt rename to version-python/requirements.txt diff --git a/sorter.py b/version-python/sorter.py similarity index 100% rename from sorter.py rename to version-python/sorter.py diff --git a/tests/test_catalogue.py b/version-python/tests/test_catalogue.py similarity index 100% rename from tests/test_catalogue.py rename to version-python/tests/test_catalogue.py diff --git a/tests/test_extract.py b/version-python/tests/test_extract.py similarity index 100% rename from tests/test_extract.py rename to version-python/tests/test_extract.py diff --git a/tests/test_sorter_units.py b/version-python/tests/test_sorter_units.py similarity index 100% rename from tests/test_sorter_units.py rename to version-python/tests/test_sorter_units.py diff --git a/tools/debug_gofile.py b/version-python/tools/debug_gofile.py similarity index 100% rename from tools/debug_gofile.py rename to version-python/tools/debug_gofile.py diff --git a/tools/extract_all.py b/version-python/tools/extract_all.py similarity index 100% rename from tools/extract_all.py rename to version-python/tools/extract_all.py diff --git a/tools/gw_browse_test.py b/version-python/tools/gw_browse_test.py similarity index 100% rename from tools/gw_browse_test.py rename to version-python/tools/gw_browse_test.py diff --git a/tools/gw_build_catalog.py b/version-python/tools/gw_build_catalog.py similarity index 100% rename from tools/gw_build_catalog.py rename to version-python/tools/gw_build_catalog.py diff --git a/tools/gw_inspect.py b/version-python/tools/gw_inspect.py similarity index 100% rename from tools/gw_inspect.py rename to version-python/tools/gw_inspect.py diff --git a/tools/new_repo.py b/version-python/tools/new_repo.py similarity index 100% rename from tools/new_repo.py rename to version-python/tools/new_repo.py diff --git a/tools/plan_diff.py b/version-python/tools/plan_diff.py similarity index 100% rename from tools/plan_diff.py rename to version-python/tools/plan_diff.py diff --git a/tools/plan_summary.py b/version-python/tools/plan_summary.py similarity index 100% rename from tools/plan_summary.py rename to version-python/tools/plan_summary.py diff --git a/version-rust/.gitignore b/version-rust/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/version-rust/.gitignore @@ -0,0 +1 @@ +/target diff --git a/version-rust/Cargo.lock b/version-rust/Cargo.lock new file mode 100644 index 0000000..b9a93a0 --- /dev/null +++ b/version-rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "dlphelper" +version = "0.1.0" diff --git a/version-rust/Cargo.toml b/version-rust/Cargo.toml new file mode 100644 index 0000000..0f777f6 --- /dev/null +++ b/version-rust/Cargo.toml @@ -0,0 +1,7 @@ +[package] +name = "dlphelper" +version = "0.1.0" +edition = "2024" +description = "Torrent-first platform for Warhammer proxy models: Discord channel export, link extraction, per-host downloaders, manifest — Rust rewrite of version-python." + +[dependencies] diff --git a/version-rust/README.md b/version-rust/README.md new file mode 100644 index 0000000..73c0c08 --- /dev/null +++ b/version-rust/README.md @@ -0,0 +1,36 @@ +# version-rust + +Rust rewrite of `version-python` (same repo, `main` branch still carries the +flat python layout). Goal: one concise binary + webapp, eventually replacing +the python tooling. + +## Why this structure + +- `main` stays untouched (flat python layout) until the rewrite reaches parity. +- This branch re-homes the python code into `version-python/` (via `git mv`, + history preserved) and starts `version-rust/` beside it. +- `version-python/AGENTS.md` still documents the python stage — its rules + (secrets, failure policy, idempotency, rate limiting) apply to the rewrite + too and are the contract the Rust version must keep. + +## Planned crate choices (initial, adjust as modules get ported) + +| Concern | Python today | Rust plan | +|---|---|---| +| 7z extraction (extract_all) | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust, LZMA2/PPMD/AES-256); zip via `zip` crate; rar likely still needs an external tool | +| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) | +| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` | +| Fuzzy matching (sorter) | difflib-style ratios | `strsim` — port thresholds exactly, add unit tests | +| CLI | argparse | `clap` | +| Webapp | — | `axum` (CLI parity first, webapp second) | +| Discord export | DCE docker container (external) | unchanged: shell out to docker | + +## Port order (suggested) + +1. `state` / `globalstore` — state.json + urls.json schemas must stay + backward compatible with the python versions. +2. hosters: `gofile` (one guest account per run!), `zero807` (UA + Referer). +3. extraction + manifest appender. +4. catalogue / sorter (hardest logic; port the matching rules documented in + `version-python/AGENTS.md` §7 with tests). +5. webapp on top. diff --git a/version-rust/src/main.rs b/version-rust/src/main.rs new file mode 100644 index 0000000..aa55047 --- /dev/null +++ b/version-rust/src/main.rs @@ -0,0 +1,3 @@ +fn main() { + println!("dlphelper {version} — Rust rewrite scaffold; modules are ported step by step, see version-rust/README.md", version = env!("CARGO_PKG_VERSION")); +}