Bugs:
- extract: pick the OUTERMOST archive suffix (sfx.first(), not last())
so x.tar.gz uses the gz->tar path
- gwshop: truncate the error snippet by chars, not bytes (UTF-8 panic)
- sorter: py_stem_split now matches pathlib (0 < i < len-1) so a
trailing dot is not treated as a suffix
- sorter::apply_plan: a per-file link/copy/mkdir failure is logged and
skipped instead of aborting the whole apply (never-abort contract)
- orchestrate/state: add State::mark_logged and use it for every
per-URL record so a state-save error cannot abort a run
Cleanups:
- remove dead is_archive_name and unused SourceInfo.manifest field
- mark kept-but-unused capabilities with allow(dead_code) + rationale
- let-chains / is_ok_and / is_multiple_of / replace([..]) / sort_by_key
and other idiomatic simplifications across modules
- difflib: SequenceMatcher owns its char vecs (bit-identical parity kept)
- --only now enforces gofile|0807 via value_parser
Docs:
- SORTING.md: drop phantom sorter_plan.jsonl, os.link -> std::fs::hard_link,
zipfile -> the zip crate
- README: add --since to the dlp synopsis, clarify --subs/--assume-done
- .gitignore: ignore catalogue/sorter/_logs/subscriptions run artifacts
Validation: cargo fmt --check clean, cargo clippy --all-targets 0 warnings,
cargo test 87 passed / 0 failed.
The Rust rewrite has replaced the original python implementation, so the transition layout is no longer needed:
- move version-rust/{Cargo.toml,Cargo.lock,README.md,.gitignore,src,data} to the repo root; the crate is now the repo.
- move docs/SORTING.md and data/gwshop_catalog.json to the root.
- remove version-python/ entirely, plus the dev-only version-rust/tools/dump_warhammer_data.py (its only job was regenerating data/warhammer_data.json from python).
- rewrite AGENTS.md and README.md for the single-project layout; strip all version-python/version-rust references from code comments.
- fix the gwshop credential lookup: algolia.json now sits at CARGO_MANIFEST_DIR (crate root == repo root), not its parent.
- restore the credentials/run-output ignores in .gitignore (algolia.json, .env, *.token, downloads/, state.json, manifest.jsonl, ...).
- docs/SORTING.md: retarget the extraction/sorter/dlp examples at the dlphelper binary instead of the deleted python scripts.
cargo test: 86/86 green.
- git mv of all existing files into version-python/ (history preserved)
- version-rust/: minimal cargo scaffold (edition 2024), no deps yet
- root AGENTS.md documents the transition rules and artifact/schema contract
- main is untouched; the python layout there stays as-is
- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202
goku challenge) for non-browser clients; findings recorded in AGENTS.md
- the shop search runs on Algolia; public search-only key lets us dump the
whole product index with plain requests (no browser, no captcha)
- gwshop.py: credential loading (env or gitignored algolia.json), index
listing, facet-partitioned full retrieval (search-only key cannot /browse
and pagination caps at 1000, so the index is partitioned on the
GameSystemsRoot hierarchy facet), normalization into the sorter's
source-agnostic catalog schema (system/group/faction/subfaction/class)
- data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww,
3491 unique products, every product with full hierarchy path
- tools/: one-shot browse + catalog builder + shape inspector
Ben_Kosytorz