Add GW webshop catalog source (Algolia): dlphelper.gwshop + full dump
- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202 goku challenge) for non-browser clients; findings recorded in AGENTS.md - the shop search runs on Algolia; public search-only key lets us dump the whole product index with plain requests (no browser, no captcha) - gwshop.py: credential loading (env or gitignored algolia.json), index listing, facet-partitioned full retrieval (search-only key cannot /browse and pagination caps at 1000, so the index is partitioned on the GameSystemsRoot hierarchy facet), normalization into the sorter's source-agnostic catalog schema (system/group/faction/subfaction/class) - data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww, 3491 unique products, every product with full hierarchy path - tools/: one-shot browse + catalog builder + shape inspector
This commit is contained in:
@@ -199,7 +199,24 @@ never re-fetch them again):
|
||||
`catalogue_moves.jsonl`). The taxonomy lives in
|
||||
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
|
||||
collection - unclassified files are reported, never guessed. Manual fixes
|
||||
go into an overrides JSON (`--overrides`).
|
||||
goes into an overrides JSON (`--overrides`).
|
||||
|
||||
**Catalog source findings (warhammer.com, verified 2026-10-04, do not
|
||||
re-probe blindly):** the shop HTML and all search/listing endpoints
|
||||
(`/en-WW/shop/*`, `/app/resources/catalog/search/*`,
|
||||
`/app/resources/catalog/categories`) sit behind an **AWS WAF** challenge
|
||||
(405/202 "goku" JS captcha) for any non-browser client, regardless of
|
||||
User-Agent. Static assets, robots.txt and 404 pages are served normally.
|
||||
The sitemap declared in robots.txt does not exist. Deep paths under the
|
||||
allowed `/app/resources/catalog/product/*` reach a separate commerce
|
||||
backend (plain `404 Not found`) but the correct resource shape is
|
||||
unknown. The site's product search is **Algolia**; its app-id and
|
||||
search-only key are public (they ship with the browser page) and would
|
||||
allow a full catalog dump via plain `requests` — this is the most
|
||||
promising path: capture one search request from the browser devtools
|
||||
("Copy as cURL") and build `dlphelper/gwshop.py` on it. Catalog sources
|
||||
stay pluggable and source-agnostic regardless (miniset.net, manual JSON
|
||||
overrides, GW shop).
|
||||
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
|
||||
primitive version; a real watch mode (per-subscription schedules, tear-off
|
||||
notifications, discord pings on new links) builds on it.
|
||||
|
||||
Reference in New Issue
Block a user