- tools/extract_all.py: recursive extraction of all archives (incl.
nested) into a plain-file tree; nested archives are deleted after
successful extraction, plain files are hardlinked, failures go to
.broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
the archive's name (files of one archive are one unit; inner
structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
and a same-system faction jump below 0.75 is rejected. Fixes 40k
files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
quick_ratio pruning; keywords: night lord, csm.
Summed nbHits across facet branches double-counted products listed in
several branches; coverage is now per unique objectID, with a warning
about records left out of the dump. AGENTS.md documents the live
count (4183 records, 3491 via facet).
The §7.3 catalog-source notes still described the Algolia route as a
future 'promising path'. Now documented as implemented, with the full
method (WAF findings, search-only key in gitignored algolia.json, index
name, /browse 403 + 1000-hit pagination cap, GameSystemsRoot facet
partitioning, snapshot committed as data/gwshop_catalog.json and how to
refresh it) and the GW-naming switch in §7.3 + the module list.
- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202
goku challenge) for non-browser clients; findings recorded in AGENTS.md
- the shop search runs on Algolia; public search-only key lets us dump the
whole product index with plain requests (no browser, no captcha)
- gwshop.py: credential loading (env or gitignored algolia.json), index
listing, facet-partitioned full retrieval (search-only key cannot /browse
and pagination caps at 1000, so the index is partitioned on the
GameSystemsRoot hierarchy facet), normalization into the sorter's
source-agnostic catalog schema (system/group/faction/subfaction/class)
- data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww,
3491 unique products, every product with full hierarchy path
- tools/: one-shot browse + catalog builder + shape inspector
- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every
successfully downloaded URL once, regardless of OUT_DIR - reposts in other
channels and re-runs of moved channels no longer re-download
- --subs subscriptions.json: batch mode over a configurable channel list;
re-exports each channel by default (new links get picked up), --no-fresh
reuses cached exports
- --assume-done <file>: manual import step that marks old URL lists as done
in the global store (deliberate, unverified, never automatic)
- lazy downloader startup: no gofile guest account / no curl runs when all
URLs are already done; token prompt skipped in --urls-file mode
- write_failed_urls creates OUT_DIR if needed
Ben_Kosytorz