Warhammer_DLP_Helper/docs/SORTING.md

Library structure & sorting pipeline

This document defines the target structure of the collection ("the library") and how the sorter places files into it. It is the working agreement for all sorting-related code; change it here first, then in code.

Target hierarchy (Games Workshop webshop-aligned)

The hierarchy follows the official GW webshop taxonomy (dumped from the shop's Algolia index by dlphelper/gwshop.py into data/gwshop_catalog.json). Chapters and sub-lines that the old miniset hierarchy treated as sub-factions are full GW factions, and the class layer holds GW shop categories (unit types / product lines) instead of the retired 40k force-organization slots:

<library>/
  Warhammer 40,000/                       # game (GW system)
    Space Marines/                        # GW group
      Raven Guard/                        # GW faction (chapter)
        Walker/                           # GW class (unit type / category)
          Redemptor Dreadnought/          # canonical unit name (catalog)
            Redemptor Dreadnought.zip     # the file(s), or unpacked contents

Layer rules:

Layer	Source of truth	Example	May be omitted?
Game	gwshop_catalog.json (system)	Warhammer 40,000	no (else: _unsorted/)
Group	GW GameSystemsRoot lvl1	Space Marines	no (from catalog or warhammer_data.GROUP_OF_FACTION)
Faction	GW factions (incl. chapters)	Raven Guard	no (else _unsorted/ in game)
Sub-faction	deeper sources only	—	yes — GW has none
Class	GW shop category	Walker, Battleline, Rules & Accessories	yes — only when known
Unit	canonical product name (GW catalog)	Redemptor Dreadnought	yes (files drop directly into the class dir)

• Files without a faction but with a game land in <game>/_unsorted/.
• Fully unclassifiable files land in <library>/_unsorted/ — never guessed.
• The keyword tables in warhammer_data.py (GW naming) provide the initial system/faction/class signal; a catalog match then overrides the hierarchy from the catalog entry, which is authoritative.
• data/library_catalog.json (the older miniset-aligned hand catalog) is still usable via sorter.py --catalog; its sub-faction entries map into the same schema. For GW-sourced entries subfaction stays empty.

Canonical names

Canonical unit names follow the GW webshop spelling (Keeper of Secrets, Thunderwolf Cavalry), because that is what the catalog, the shop and wikis all agree on. Release names stay untouched inside the unit folder.

Sorting pipeline

graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]

1. Inspect — read the file name; for archives, also read the inner file list (zip via stdlib zipfile, 7z via the installed 7z l binary — no Python dependency). Inner names improve classification for hash-named archives and are recorded in the report.
2. Classify — game/faction/class from the keyword tables (warhammer_data.py), same as catalogue.py.
3. Match — fuzzy-match the (inner) name against data/library_catalog.json entries within the same faction; best sequence-match above --threshold (default 0.62) wins. The catalog provides canonical names, sub-factions and classes.
4. Plan — default output: a dry-run table source → target plus sorter_plan.jsonl. Nothing is touched.
5. Apply — create the target and hardlink the file there (os.link); if the filesystems differ, fall back to a real copy with a warning. Originals are never modified or deleted — that is what keeps the *arr-style workflow possible (remux/seed/sort independently). Every applied action is appended to sorter_applied.jsonl and re-runs skip targets that already exist.

Why hardlinks, and the unzip question

• Hardlinks make "same file, two positions" free (0 extra bytes) — the original stays for seeding/history, the library copy is what torrenting sees. Limitation: hardlinks need source and library on the same filesystem; the sorter detects this and copies instead (warned).
• Zip contents can be read without extracting (that is how the sorter peeks). Modifying a member in place is technically possible with zipfile but is not useful here.
• Endgame (later stage): unpack archives into the unit folder so the library is plain files, easier to sort per-model and to torrent. The unpack step will live in the sorter (--unpack, 7z binary) but stays off until the link-based flow is proven on a small batch.

Extraction stage (tools/extract_all.py)

That endgame stage now exists as a standalone tool (the sorter stays plan/apply-only):

python3 tools/extract_all.py <src_root>... --out <extracted_root>

• Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted into a folder named after the archive, preserving relative structure; nested archives are expanded recursively and deleted, so the tree ends up with no archives at all.

• Plain files are hardlinked into the tree (fallback copy) — nothing lost, no duplicate data.

• Sources are never modified. Failed extractions (HTML-page fake zips, password archives, disk full) are moved to <out>/.broken/ and the run continues; re-runs skip already-extracted destinations.

• <out>/extract_manifest.jsonl records {archive, dest, unit} per extraction. The sorter's --provenance flag reads it: all files inside an extraction directory are classified once under the archive's name (a loose Backpack.stl alone is noise; under The Twin Lance/ it belongs to that unit) and placed together, keeping the archive's inner directory structure:

  python3 sorter.py <extracted_root> --library <lib> \
      --provenance <extracted_root>/extract_manifest.jsonl --apply
  

Catalog file (data/library_catalog.json)

{
  "system": "Warhammer 40,000",
  "group": "Space Marines",
  "faction": "Space Wolves",
  "subfaction": null,
  "class": "Mounted",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}

Entries are matched case-insensitively against cleaned names. Growing the catalog is the main lever for match quality; unknown files surface in the plan report with their best score so gaps are obvious.