Warhammer_DLP_Helper/docs/SORTING.md

Library structure & sorting pipeline

This document defines the target structure of the collection ("the library") and how the sorter places files into it. It is the working agreement for all sorting-related code; change it here first, then in code.

Target hierarchy (Games Workshop webshop-aligned)

The hierarchy follows the official GW webshop taxonomy (dumped from the shop's Algolia index by dlphelper gwshop into data/gwshop_catalog.json). Chapters and sub-lines that the old miniset hierarchy treated as sub-factions are full GW factions, and the class layer holds GW shop categories (unit types / product lines) instead of the retired 40k force-organization slots:

<library>/
  Warhammer 40,000/                       # game (GW system)
    Space Marines/                        # GW group
      Raven Guard/                        # GW faction (chapter)
        Walker/                           # GW class (unit type / category)
          Redemptor Dreadnought/          # canonical unit name (catalog)
            Redemptor Dreadnought.zip     # the file(s), or unpacked contents

Layer rules:

Layer	Source of truth	Example	May be omitted?
Game	gwshop_catalog.json (system)	Warhammer 40,000	no (else: _unsorted/)
Group	GW GameSystemsRoot lvl1	Space Marines	no (from catalog or warhammer_data.GROUP_OF_FACTION)
Faction	GW factions (incl. chapters)	Raven Guard	no (else _unsorted/ in game)
Sub-faction	deeper sources only	—	yes — GW has none
Class	GW shop category	Walker, Battleline, Rules & Accessories	yes — only when known
Unit	canonical product name (GW catalog)	Redemptor Dreadnought	yes (files drop directly into the class dir)

• Files without a faction but with a game land in <game>/_unsorted/.
• Fully unclassifiable files land in <library>/_unsorted/ — never guessed.
• The keyword tables in data/warhammer_data.json (GW naming) provide the initial system/faction/class signal; a catalog match then overrides the hierarchy from the catalog entry, which is authoritative.
• The catalog is pluggable via dlphelper sorter --catalog; every entry uses the same schema. For GW-sourced entries subfaction stays empty.

Canonical names

Canonical unit names follow the GW webshop spelling (Keeper of Secrets, Thunderwolf Cavalry), because that is what the catalog, the shop and wikis all agree on. Release names stay untouched inside the unit folder.

Sorting pipeline

graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]

1. Inspect — read the file name; for archives, also read the inner file list (zip via the zip crate, 7z via sevenz-rust2 — no external binary). Inner names improve classification for hash-named archives and are recorded in the report.
2. Classify — game/faction/class from the keyword tables (data/warhammer_data.json).
3. Match — fuzzy-match the (inner) name against the catalog (data/gwshop_catalog.json) entries within the same faction; best sequence-match above --threshold (default 0.62) wins. The catalog provides canonical names, sub-factions and classes.
4. Plan — default output: a dry-run table source → target plus sorter_plan.jsonl. Nothing is touched.
5. Apply — create the target and hardlink the file there (os.link); if the filesystems differ, fall back to a real copy with a warning. Originals are never modified or deleted — that is what keeps the *arr-style workflow possible (remux/seed/sort independently). Every applied action is appended to sorter_applied.jsonl and re-runs skip targets that already exist.

Bookkeeping location

The audit (sorter_applied.jsonl) and the low-confidence review doc (sorter_low_confidence.txt) are written to <library>/../_logs/ by default — never inside the library itself, because the library may be a seeded torrent whose file set is frozen (a rewritten audit file would fail the client's recheck). --audit FILE overrides this; the low- confidence doc always lands next to the audit file.

Why hardlinks, and the unzip question

• Hardlinks make "same file, two positions" free (0 extra bytes) — the original stays for seeding/history, the library copy is what torrenting sees. Limitation: hardlinks need source and library on the same filesystem; the sorter detects this and copies instead (warned).
• Zip contents can be read without extracting (that is how the sorter peeks). Modifying a member in place is technically possible with zipfile but is not useful here.
• Endgame (later stage): unpack archives into the unit folder so the library is plain files, easier to sort per-model and to torrent. The unpack step will live in the sorter (--unpack, 7z binary) but stays off until the link-based flow is proven on a small batch.

Extraction stage (dlphelper extract)

That endgame stage now exists as a standalone command (the sorter stays plan/apply-only):

dlphelper extract <src_root>... --out <extracted_root>

• Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted into a folder named after the archive, preserving relative structure; nested archives are expanded recursively and deleted, so the tree ends up with no archives at all.

• Plain files are hardlinked into the tree (fallback copy) — nothing lost, no duplicate data.

• Sources are never modified. Failed extractions (HTML-page fake zips, password archives, disk full) are moved to <out>/.broken/ and the run continues; re-runs skip already-extracted destinations.

• <out>/extract_manifest.jsonl records {archive, dest, unit} per extraction. By default the sorter classifies each file on its own name; the optional --provenance <manifest> mode instead classifies each extraction directory once under the archive's name (a loose Backpack.stl alone is noise; under The Twin Lance/ it belongs to that unit) and keeps the archive's inner directory structure. The live library uses the default (non-provenance) mode:

  # default — what Warhammer_Library_Live uses
  dlphelper sorter <extracted_root> --library <lib> --apply
  
  # optional provenance mode (whole extraction dirs as units)
  dlphelper sorter <extracted_root> --library <lib> \
      --provenance <extracted_root>/extract_manifest.jsonl --apply
  

Operational notes from the first full run (2026-10-06, sources unsorted/1..4 → extracted ≈ 1.4 TB / 260k files / 0 archives left; library rebuilt from scratch: 218592 hardlinked, 30584 skipped as already-present duplicates, 0 copied):

• The CIFS NAS mount reports fabricated per-path inode numbers (noserverino behavior): two paths to the same hardlinked file show different st_ino, and the values are stable across calls. To verify a link, compare st_nlink (and content/size) — never inodes. Server-side links are real: a fresh os.link bumps nlink (2→3 in the checked case).
• Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2 ("Cannot open output file"); the extractor then extracts to a /var/tmp temp dir, repairs the names and moves the result over.
• __MACOSX/ and ._* AppleDouble entries are deleted as junk, not extracted.
• Files under hidden trees (.broken/) are never planned; failed archives (HTML-page fake zips, password archives, disk full) are quarantined there for inspection and the run continues.
• Non-warhammer systems are dropped (2026-10-07, user decision: ("drop anything that isn't really warhammer"). The sorter default (DROPPED_SYSTEMS in src/sorter.rs) is {Middle-Earth}: catalog entries of dropped systems are filtered at load and keyword hits for them count as unclassified, so nothing is ever placed into those sections - the files stay in _unsorted. --drop-system NAME drops more, --keep-system NAME re-enables one. The committed catalog snapshot itself is untouched.
• Probe votes on full inner paths (2026-10-07): the pack probe's inner-name votes clean the whole relative path (dir names included), not just the file stem - hash-named 0807 packs resolve their faction from folders like Catachan/Jungle Fighters/ instead of from one root file named lieutenant. The token-containment boost is disabled for single-token inner votes (one generic-ish word must not claim a pack), and full-catalog/different-faction adoptions additionally require token containment (_contained_somewhere). A matched catalog unit's class layer always wins (no keyword class survives next to a precise unit match).

Catalog file (data/gwshop_catalog.json)

{
  "system": "Warhammer 40,000",
  "group": "Space Marines",
  "faction": "Space Wolves",
  "subfaction": null,
  "class": "Mounted",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}

Entries are matched case-insensitively against cleaned names. Growing the catalog is the main lever for match quality; unknown files surface in the plan report with their best score so gaps are obvious.