- tools/extract_all.py: recursive extraction of all archives (incl.
nested) into a plain-file tree; nested archives are deleted after
successful extraction, plain files are hardlinked, failures go to
.broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
the archive's name (files of one archive are one unit; inner
structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
and a same-system faction jump below 0.75 is rejected. Fixes 40k
files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
quick_ratio pruning; keywords: night lord, csm.
Best-pool scoring used to always blend the archive's inner file names
into the fuzzy score (0.95 discount), which dragged clean-named files
into wrong units (e.g. 'Keeper of Secrets & Shalaxi Helbane.7z' ended
in Orks/Boyz). Now the outer name decides when it scores >= threshold;
inner names only vote as fallback (hash-named gofile archives rely on
that). Also: url/failed-list run artifacts are excluded from sorting,
and two plan-inspection helpers under tools/.
- warhammer_data.py: GW naming throughout - Space Marine chapters are
their own factions under a 'Space Marines' group, systems match the
shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
Middle-Earth), classes are GW shop categories (Battleline, Infantry,
Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
class (the shop's last level is the class layer), depth-5 sources keep
a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
faction when GW moved a unit or keyword tables are stale); generic GW
branches ('Unit Type') and a Battleline default never degrade a
confident keyword classification; whole-catalog retry when the faction
pool has no confident match (handles units GW moved between factions);
whole-catalog matching also runs when keyword classification found no
faction; invalid (HTML) items never fuzzy-match; default catalog is
now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
current placement) instead of the stale Drukhari keyword
Ben_Kosytorz