Add GW webshop catalog source (Algolia): dlphelper.gwshop + full dump

- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202
  goku challenge) for non-browser clients; findings recorded in AGENTS.md
- the shop search runs on Algolia; public search-only key lets us dump the
  whole product index with plain requests (no browser, no captcha)
- gwshop.py: credential loading (env or gitignored algolia.json), index
  listing, facet-partitioned full retrieval (search-only key cannot /browse
  and pagination caps at 1000, so the index is partitioned on the
  GameSystemsRoot hierarchy facet), normalization into the sorter's
  source-agnostic catalog schema (system/group/faction/subfaction/class)
- data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww,
  3491 unique products, every product with full hierarchy path
- tools/: one-shot browse + catalog builder + shape inspector
This commit is contained in:
2026-10-04 21:48:58 +02:00
parent cccf10a308
commit 5fd5dacaf0
7 changed files with 51545 additions and 1 deletions
+18 -1
View File
@@ -199,7 +199,24 @@ never re-fetch them again):
`catalogue_moves.jsonl`). The taxonomy lives in
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
collection - unclassified files are reported, never guessed. Manual fixes
go into an overrides JSON (`--overrides`).
goes into an overrides JSON (`--overrides`).
**Catalog source findings (warhammer.com, verified 2026-10-04, do not
re-probe blindly):** the shop HTML and all search/listing endpoints
(`/en-WW/shop/*`, `/app/resources/catalog/search/*`,
`/app/resources/catalog/categories`) sit behind an **AWS WAF** challenge
(405/202 "goku" JS captcha) for any non-browser client, regardless of
User-Agent. Static assets, robots.txt and 404 pages are served normally.
The sitemap declared in robots.txt does not exist. Deep paths under the
allowed `/app/resources/catalog/product/*` reach a separate commerce
backend (plain `404 Not found`) but the correct resource shape is
unknown. The site's product search is **Algolia**; its app-id and
search-only key are public (they ship with the browser page) and would
allow a full catalog dump via plain `requests` — this is the most
promising path: capture one search request from the browser devtools
("Copy as cURL") and build `dlphelper/gwshop.py` on it. Catalog sources
stay pluggable and source-agnostic regardless (miniset.net, manual JSON
overrides, GW shop).
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
primitive version; a real watch mode (per-subscription schedules, tear-off
notifications, discord pings on new links) builds on it.