diff --git a/AGENTS.md b/AGENTS.md index 0dcda0f..46476e4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,6 +43,8 @@ for dedupe/cross-seeding later. ``` dlp.py CLI orchestrator (entry point) catalogue.py CLI: classify the collection into system/faction/class +sorter.py CLI: fuzzysort files into the library (plan/apply, + hardlinks, default catalog data/gwshop_catalog.json) dlphelper/ discord.py DCE docker export (JSON) + base64/plain URL extraction gofile.py GoFile API client (one guest account per run) @@ -50,7 +52,11 @@ dlphelper/ state.py state.json store + manifest.jsonl appender globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json) catalogue.py scanner + keyword classifier + report writer - warhammer_data.py taxonomy: systems, factions, force-org classes + warhammer_data.py taxonomy: GW-named systems/factions, GW shop + categories as classes, GROUP_OF_FACTION map + gwshop.py GW webshop catalog dump via Algolia (see §7.3) + sorter.py library sorter: inspect/classify/fuzzy-match + against the catalog, plan + hardlink apply parts/grab.sh legacy groundstone script (kept, not used by dlp.py) tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself) tools/debug_gofile.py inspect raw gofile API response for one content id @@ -194,29 +200,45 @@ never re-fetch them again): 2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into a watch dir, cross-seeding between direct-download and torrent copies. 3. **Library organization (started)**: `catalogue.py` classifies every file - into system / faction / force-org class and can move files into - `////` with `--apply` (audited in + into system / faction / class and can move files into + `/////` with `--apply` (audited in `catalogue_moves.jsonl`). The taxonomy lives in `dlphelper/warhammer_data.py`; it is keyword-based and grows with the collection - unclassified files are reported, never guessed. Manual fixes - goes into an overrides JSON (`--overrides`). + go into an overrides JSON (`--overrides`). The library hierarchy follows + the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the + working agreement for the sorting pipeline. - **Catalog source findings (warhammer.com, verified 2026-10-04, do not - re-probe blindly):** the shop HTML and all search/listing endpoints - (`/en-WW/shop/*`, `/app/resources/catalog/search/*`, - `/app/resources/catalog/categories`) sit behind an **AWS WAF** challenge - (405/202 "goku" JS captcha) for any non-browser client, regardless of - User-Agent. Static assets, robots.txt and 404 pages are served normally. - The sitemap declared in robots.txt does not exist. Deep paths under the - allowed `/app/resources/catalog/product/*` reach a separate commerce - backend (plain `404 Not found`) but the correct resource shape is - unknown. The site's product search is **Algolia**; its app-id and - search-only key are public (they ship with the browser page) and would - allow a full catalog dump via plain `requests` — this is the most - promising path: capture one search request from the browser devtools - ("Copy as cURL") and build `dlphelper/gwshop.py` on it. Catalog sources - stay pluggable and source-agnostic regardless (miniset.net, manual JSON - overrides, GW shop). + **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`, + working, verified 2026-10-04 — how it was obtained, do not re-derive):** + scraping warhammer.com directly is blocked — the shop HTML and all + search/listing endpoints (`/en-WW/shop/*`, + `/app/resources/catalog/search/*`, `/app/resources/catalog/categories`) + sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any + non-browser client, regardless of User-Agent; the sitemap declared in + robots.txt does not exist; deep paths under the allowed + `/app/resources/catalog/product/*` reach a separate commerce backend + (plain `404 Not found`) but the resource shape is unknown. What works: + the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and + its app-id plus the **public search-only API key** ship with the browser + page; the user captured both from the browser devtools ("Copy as cURL" + on one search request) and they are stored in the gitignored + `algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the + key is search-only, never commit any other Algolia key. The full + product index `prod-lazarus-product-en-ww` (~3500 products, plus one + sibling index per locale) is dumped with plain `requests`: the + search-only key cannot use Algolia's `/browse` (403) and normal + pagination is capped at 1000 cumulative hits, so `gwshop.py` + **partitions the index on the `GameSystemsRoot` hierarchical facet** — + each branch is fetched with an explicit filter, recursing deeper only + when a branch exceeds one page. Records are normalized into the + source-agnostic catalog schema (system/group/faction/subfaction/class + = the facet path, name/sku/productType attached) into + `data/gwshop_catalog.json`, which is the sorter's default catalog and + is **committed as a snapshot**; refresh with + `python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog + sources stay pluggable and source-agnostic (miniset.net — reachable + but stale, manual JSON overrides, `data/library_catalog.json`). 4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a primitive version; a real watch mode (per-subscription schedules, tear-off notifications, discord pings on new links) builds on it.