AGENTS.md: document the working GW/Algolia catalog source
The §7.3 catalog-source notes still described the Algolia route as a future 'promising path'. Now documented as implemented, with the full method (WAF findings, search-only key in gitignored algolia.json, index name, /browse 403 + 1000-hit pagination cap, GameSystemsRoot facet partitioning, snapshot committed as data/gwshop_catalog.json and how to refresh it) and the GW-naming switch in §7.3 + the module list.
This commit is contained in:
@@ -43,6 +43,8 @@ for dedupe/cross-seeding later.
|
||||
```
|
||||
dlp.py CLI orchestrator (entry point)
|
||||
catalogue.py CLI: classify the collection into system/faction/class
|
||||
sorter.py CLI: fuzzysort files into the library (plan/apply,
|
||||
hardlinks, default catalog data/gwshop_catalog.json)
|
||||
dlphelper/
|
||||
discord.py DCE docker export (JSON) + base64/plain URL extraction
|
||||
gofile.py GoFile API client (one guest account per run)
|
||||
@@ -50,7 +52,11 @@ dlphelper/
|
||||
state.py state.json store + manifest.jsonl appender
|
||||
globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json)
|
||||
catalogue.py scanner + keyword classifier + report writer
|
||||
warhammer_data.py taxonomy: systems, factions, force-org classes
|
||||
warhammer_data.py taxonomy: GW-named systems/factions, GW shop
|
||||
categories as classes, GROUP_OF_FACTION map
|
||||
gwshop.py GW webshop catalog dump via Algolia (see §7.3)
|
||||
sorter.py library sorter: inspect/classify/fuzzy-match
|
||||
against the catalog, plan + hardlink apply
|
||||
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
|
||||
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
|
||||
tools/debug_gofile.py inspect raw gofile API response for one content id
|
||||
@@ -194,29 +200,45 @@ never re-fetch them again):
|
||||
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
|
||||
a watch dir, cross-seeding between direct-download and torrent copies.
|
||||
3. **Library organization (started)**: `catalogue.py` classifies every file
|
||||
into system / faction / force-org class and can move files into
|
||||
`<library>/<System>/<Faction>/<Class>/` with `--apply` (audited in
|
||||
into system / faction / class and can move files into
|
||||
`<library>/<System>/<Group>/<Faction>/<Class>/` with `--apply` (audited in
|
||||
`catalogue_moves.jsonl`). The taxonomy lives in
|
||||
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
|
||||
collection - unclassified files are reported, never guessed. Manual fixes
|
||||
goes into an overrides JSON (`--overrides`).
|
||||
go into an overrides JSON (`--overrides`). The library hierarchy follows
|
||||
the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
|
||||
working agreement for the sorting pipeline.
|
||||
|
||||
**Catalog source findings (warhammer.com, verified 2026-10-04, do not
|
||||
re-probe blindly):** the shop HTML and all search/listing endpoints
|
||||
(`/en-WW/shop/*`, `/app/resources/catalog/search/*`,
|
||||
`/app/resources/catalog/categories`) sit behind an **AWS WAF** challenge
|
||||
(405/202 "goku" JS captcha) for any non-browser client, regardless of
|
||||
User-Agent. Static assets, robots.txt and 404 pages are served normally.
|
||||
The sitemap declared in robots.txt does not exist. Deep paths under the
|
||||
allowed `/app/resources/catalog/product/*` reach a separate commerce
|
||||
backend (plain `404 Not found`) but the correct resource shape is
|
||||
unknown. The site's product search is **Algolia**; its app-id and
|
||||
search-only key are public (they ship with the browser page) and would
|
||||
allow a full catalog dump via plain `requests` — this is the most
|
||||
promising path: capture one search request from the browser devtools
|
||||
("Copy as cURL") and build `dlphelper/gwshop.py` on it. Catalog sources
|
||||
stay pluggable and source-agnostic regardless (miniset.net, manual JSON
|
||||
overrides, GW shop).
|
||||
**Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
|
||||
working, verified 2026-10-04 — how it was obtained, do not re-derive):**
|
||||
scraping warhammer.com directly is blocked — the shop HTML and all
|
||||
search/listing endpoints (`/en-WW/shop/*`,
|
||||
`/app/resources/catalog/search/*`, `/app/resources/catalog/categories`)
|
||||
sit behind an **AWS WAF** challenge (405/202 "goku" JS captcha) for any
|
||||
non-browser client, regardless of User-Agent; the sitemap declared in
|
||||
robots.txt does not exist; deep paths under the allowed
|
||||
`/app/resources/catalog/product/*` reach a separate commerce backend
|
||||
(plain `404 Not found`) but the resource shape is unknown. What works:
|
||||
the shop's product search runs on **Algolia** (app-id `M5ZIQZNQ2H`) and
|
||||
its app-id plus the **public search-only API key** ship with the browser
|
||||
page; the user captured both from the browser devtools ("Copy as cURL"
|
||||
on one search request) and they are stored in the gitignored
|
||||
`algolia.json` (or `ALGOLIA_APP_ID`/`ALGOLIA_API_KEY` env vars) — the
|
||||
key is search-only, never commit any other Algolia key. The full
|
||||
product index `prod-lazarus-product-en-ww` (~3500 products, plus one
|
||||
sibling index per locale) is dumped with plain `requests`: the
|
||||
search-only key cannot use Algolia's `/browse` (403) and normal
|
||||
pagination is capped at 1000 cumulative hits, so `gwshop.py`
|
||||
**partitions the index on the `GameSystemsRoot` hierarchical facet** —
|
||||
each branch is fetched with an explicit filter, recursing deeper only
|
||||
when a branch exceeds one page. Records are normalized into the
|
||||
source-agnostic catalog schema (system/group/faction/subfaction/class
|
||||
= the facet path, name/sku/productType attached) into
|
||||
`data/gwshop_catalog.json`, which is the sorter's default catalog and
|
||||
is **committed as a snapshot**; refresh with
|
||||
`python3 -m dlphelper.gwshop data/gwshop_catalog.json`. Other catalog
|
||||
sources stay pluggable and source-agnostic (miniset.net — reachable
|
||||
but stale, manual JSON overrides, `data/library_catalog.json`).
|
||||
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
|
||||
primitive version; a real watch mode (per-subscription schedules, tear-off
|
||||
notifications, discord pings on new links) builds on it.
|
||||
|
||||
Reference in New Issue
Block a user