Switch library hierarchy to GW webshop taxonomy

- warhammer_data.py: GW naming throughout - Space Marine chapters are
  their own factions under a 'Space Marines' group, systems match the
  shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
  Middle-Earth), classes are GW shop categories (Battleline, Infantry,
  Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
  extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
  class (the shop's last level is the class layer), depth-5 sources keep
  a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
  faction when GW moved a unit or keyword tables are stale); generic GW
  branches ('Unit Type') and a Battleline default never degrade a
  confident keyword classification; whole-catalog retry when the faction
  pool has no confident match (handles units GW moved between factions);
  whole-catalog matching also runs when keyword classification found no
  faction; invalid (HTML) items never fuzzy-match; default catalog is
  now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
  as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
  e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
  current placement) instead of the stale Drukhari keyword
This commit is contained in:
2026-10-04 22:05:31 +02:00
parent 5fd5dacaf0
commit 4bbaefb45a
8 changed files with 4678 additions and 4428 deletions
+32 -29
View File
@@ -4,47 +4,50 @@ This document defines the target structure of the collection ("the library")
and how the sorter places files into it. It is the working agreement for all
sorting-related code; change it here first, then in code.
## Target hierarchy (miniset.net-aligned)
## Target hierarchy (Games Workshop webshop-aligned)
The hierarchy mirrors miniset.net ("Miniatures Collectors Guide"), extended
with the 40k force-organization class, which miniset itself uses as a
"specification" tag on set pages:
The hierarchy follows the official GW webshop taxonomy (dumped from the
shop's Algolia index by `dlphelper/gwshop.py` into
`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
hierarchy treated as sub-factions are full GW factions, and the class layer
holds GW shop categories (unit types / product lines) instead of the retired
40k force-organization slots:
```
<library>/
Warhammer 40,000/ # game
Armies of the Imperium/ # faction group (miniset naming)
Adeptus Astartes/ # faction
Space Wolves/ # sub-faction (optional)
Fast Attack/ # force-org class
Thunderwolf Cavalry/ # canonical unit name
Thunderwolf Cavalry.7z # the file(s), or unpacked contents
Warhammer 40,000/ # game (GW system)
Space Marines/ # GW group
Raven Guard/ # GW faction (chapter)
Walker/ # GW class (unit type / category)
Redemptor Dreadnought/ # canonical unit name (catalog)
Redemptor Dreadnought.zip # the file(s), or unpacked contents
```
Layer rules:
| Layer | Source of truth | Example | May be omitted? |
|-------|-----------------|---------|-----------------|
| Game | `warhammer_data.SYSTEMS` | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Faction group | Imperium/Chaos/Xenos/etc. | `Armies of Chaos` | no (derived from faction) |
| Faction | miniset.net faction names | `Death Guard` | no (else `_unsorted/` in game) |
| Sub-faction | miniset sub-factions | `Space Wolves` | **yes** — only when known |
| Class | force-org slots | `Fast Attack` | **yes** — only when known |
| Unit | canonical model name (lexicanum spelling) | `Thunderwolf Cavalry` | yes (files drop directly into the class dir) |
| Game | `gwshop_catalog.json` (system) | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Group | GW `GameSystemsRoot` lvl1 | `Space Marines` | no (from catalog or `warhammer_data.GROUP_OF_FACTION`) |
| Faction | GW factions (incl. chapters) | `Raven Guard` | no (else `_unsorted/` in game) |
| Sub-faction | deeper sources only | — | **yes** — GW has none |
| Class | GW shop category | `Walker`, `Battleline`, `Rules & Accessories` | **yes** — only when known |
| Unit | canonical product name (GW catalog) | `Redemptor Dreadnought` | yes (files drop directly into the class dir) |
- Files without a faction but with a game land in `<game>/_unsorted/`.
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- Everything in between is optional and filled in as knowledge grows.
- Sub-faction folders are only created when the file name states it
(`Thunderwolf` → Space Wolves); otherwise files stay in the faction dir so
we don't invent hierarchy.
- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
system/faction/class signal; a catalog match then overrides the hierarchy
from the catalog entry, which is authoritative.
- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
still usable via `sorter.py --catalog`; its sub-faction entries map into
the same schema. For GW-sourced entries `subfaction` stays empty.
## Canonical names
Canonical unit names follow Lexicanum/GW spelling (`Keeper of Secrets`,
`Death Korps of Krieg`), because that is what cross-seeding, wikis and
miniset.net all agree on. Release names stay untouched inside the unit
folder.
Canonical unit names follow the GW webshop spelling (`Keeper of Secrets`,
`Thunderwolf Cavalry`), because that is what the catalog, the shop and
wikis all agree on. Release names stay untouched inside the unit folder.
## Sorting pipeline
@@ -97,10 +100,10 @@ graph LR
```json
{
"system": "Warhammer 40,000",
"group": "Armies of the Imperium",
"faction": "Adeptus Astartes",
"subfaction": "Space Wolves",
"class": "Fast Attack",
"group": "Space Marines",
"faction": "Space Wolves",
"subfaction": null,
"class": "Mounted",
"name": "Thunderwolf Cavalry",
"aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}