Real 7z archives named .zip (seen live: 'Heresy Lab - HL068 Utela Justina.zip')
were quarantined because dispatch trusted the extension. python's 7z CLI
sniffs content; now so do we for 7z/zip/rar, keeping extension logic for
gz/bz2/xz/tar where the tar-inside decision needs the name.
User policy: the channel hosts user-created torrents (one ~2.5 TB); this
tool is for direct-download packs. Torrent links are recorded as done with
a note so they never re-download.
Rust-only extension (no python counterpart): --since 5d / 12h / 2026-10-04
keeps only URLs posted at/after the cutoff. URLs without a timestamp
(e.g. --urls-file lists) are kept with a warning.
- src/sorter.rs: inspect (zip/7z listing via crates, HTML-page detection),
keyword classification with inner-name votes, catalog load with dropped
systems, fuzzy matching on the CPython-difflib port (containment boost,
generic-token stoplist, tokenless bonus, weak tier, cross-system gates),
deepest-dir re-homing incl. provenance unit mode, build_target,
hardlink-apply with audit + low-confidence doc, format_plan
- all 9 python test_sorter_units.py cases ported and passing
- parity fixes found while porting: inner-name votes REPLACE (not max-keep)
the outer score below threshold; python max() keeps first maximal vote;
ratio cache keyed with allow_single; 512-byte head read for HTML sniff
- git mv of all existing files into version-python/ (history preserved)
- version-rust/: minimal cargo scaffold (edition 2024), no deps yet
- root AGENTS.md documents the transition rules and artifact/schema contract
- main is untouched; the python layout there stays as-is
The library may be a seeded torrent whose file set is frozen; the
sorter_applied.jsonl / sorter_low_confidence.txt bookkeeping now goes
to <library>/../_logs/ by default. Explicit --audit still wins.
The collection is warhammer-only; everything else stays in _unsorted
even when names match the catalog. The catalog snapshot keeps all
systems - the filter is a sorter default (DROPPED_SYSTEMS, applied in
load_catalog and after keyword classification), adjustable with
--drop-system/--keep-system.
Inner directory names now compete with the pack/file name: a subtree
under a dir whose name matches the catalog better than the pack-level
resolution is re-homed there (fixes multi-unit sculptor packs like
'Retro Elvez' dumping everything into one keyword class). Junk dir
names (files/images/HEADS/2/...) never match. Hits between 0.50 and the
threshold are accepted as low-confidence: [w] in the plan, confidence
in the audit, listed in sorter_low_confidence.txt. Works in both
provenance and manual-tree mode.
- tools/extract_all.py: recursive extraction of all archives (incl.
nested) into a plain-file tree; nested archives are deleted after
successful extraction, plain files are hardlinked, failures go to
.broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
the archive's name (files of one archive are one unit; inner
structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
and a same-system faction jump below 0.75 is rejected. Fixes 40k
files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
quick_ratio pruning; keywords: night lord, csm.
Summed nbHits across facet branches double-counted products listed in
several branches; coverage is now per unique objectID, with a warning
about records left out of the dump. AGENTS.md documents the live
count (4183 records, 3491 via facet).
Best-pool scoring used to always blend the archive's inner file names
into the fuzzy score (0.95 discount), which dragged clean-named files
into wrong units (e.g. 'Keeper of Secrets & Shalaxi Helbane.7z' ended
in Orks/Boyz). Now the outer name decides when it scores >= threshold;
inner names only vote as fallback (hash-named gofile archives rely on
that). Also: url/failed-list run artifacts are excluded from sorting,
and two plan-inspection helpers under tools/.
The §7.3 catalog-source notes still described the Algolia route as a
future 'promising path'. Now documented as implemented, with the full
method (WAF findings, search-only key in gitignored algolia.json, index
name, /browse 403 + 1000-hit pagination cap, GameSystemsRoot facet
partitioning, snapshot committed as data/gwshop_catalog.json and how to
refresh it) and the GW-naming switch in §7.3 + the module list.
- warhammer_data.py: GW naming throughout - Space Marine chapters are
their own factions under a 'Space Marines' group, systems match the
shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
Middle-Earth), classes are GW shop categories (Battleline, Infantry,
Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
class (the shop's last level is the class layer), depth-5 sources keep
a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
faction when GW moved a unit or keyword tables are stale); generic GW
branches ('Unit Type') and a Battleline default never degrade a
confident keyword classification; whole-catalog retry when the faction
pool has no confident match (handles units GW moved between factions);
whole-catalog matching also runs when keyword classification found no
faction; invalid (HTML) items never fuzzy-match; default catalog is
now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
current placement) instead of the stale Drukhari keyword
- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202
goku challenge) for non-browser clients; findings recorded in AGENTS.md
- the shop search runs on Algolia; public search-only key lets us dump the
whole product index with plain requests (no browser, no captcha)
- gwshop.py: credential loading (env or gitignored algolia.json), index
listing, facet-partitioned full retrieval (search-only key cannot /browse
and pagination caps at 1000, so the index is partitioned on the
GameSystemsRoot hierarchy facet), normalization into the sorter's
source-agnostic catalog schema (system/group/faction/subfaction/class)
- data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww,
3491 unique products, every product with full hierarchy path
- tools/: one-shot browse + catalog builder + shape inspector
- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every
successfully downloaded URL once, regardless of OUT_DIR - reposts in other
channels and re-runs of moved channels no longer re-download
- --subs subscriptions.json: batch mode over a configurable channel list;
re-exports each channel by default (new links get picked up), --no-fresh
reuses cached exports
- --assume-done <file>: manual import step that marks old URL lists as done
in the global store (deliberate, unverified, never automatic)
- lazy downloader startup: no gofile guest account / no curl runs when all
URLs are already done; token prompt skipped in --urls-file mode
- write_failed_urls creates OUT_DIR if needed
Ben_Kosytorz