Library structure & sorting pipeline

This document defines the target structure of the collection ("the library") and how the sorter places files into it. It is the working agreement for all sorting-related code; change it here first, then in code.

Target hierarchy (Games Workshop webshop-aligned)

The hierarchy follows the official GW webshop taxonomy (dumped from the shop's Algolia index by dlphelper/gwshop.py into data/gwshop_catalog.json). Chapters and sub-lines that the old miniset hierarchy treated as sub-factions are full GW factions, and the class layer holds GW shop categories (unit types / product lines) instead of the retired 40k force-organization slots:

<library>/
  Warhammer 40,000/                       # game (GW system)
    Space Marines/                        # GW group
      Raven Guard/                        # GW faction (chapter)
        Walker/                           # GW class (unit type / category)
          Redemptor Dreadnought/          # canonical unit name (catalog)
            Redemptor Dreadnought.zip     # the file(s), or unpacked contents

Layer rules:

Layer	Source of truth	Example	May be omitted?
Game	`gwshop_catalog.json` (system)	`Warhammer 40,000`	no (else: `_unsorted/`)
Group	GW `GameSystemsRoot` lvl1	`Space Marines`	no (from catalog or `warhammer_data.GROUP_OF_FACTION`)
Faction	GW factions (incl. chapters)	`Raven Guard`	no (else `_unsorted/` in game)
Sub-faction	deeper sources only	—	yes — GW has none
Class	GW shop category	`Walker`, `Battleline`, `Rules & Accessories`	yes — only when known
Unit	canonical product name (GW catalog)	`Redemptor Dreadnought`	yes (files drop directly into the class dir)

Files without a faction but with a game land in <game>/_unsorted/.
Fully unclassifiable files land in <library>/_unsorted/ — never guessed.
The keyword tables in warhammer_data.py (GW naming) provide the initial system/faction/class signal; a catalog match then overrides the hierarchy from the catalog entry, which is authoritative.
data/library_catalog.json (the older miniset-aligned hand catalog) is still usable via sorter.py --catalog; its sub-faction entries map into the same schema. For GW-sourced entries subfaction stays empty.

Canonical names

Canonical unit names follow the GW webshop spelling (Keeper of Secrets, Thunderwolf Cavalry), because that is what the catalog, the shop and wikis all agree on. Release names stay untouched inside the unit folder.

Sorting pipeline

graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]

Inspect — read the file name; for archives, also read the inner file list (zip via stdlib zipfile, 7z via the installed 7z l binary — no Python dependency). Inner names improve classification for hash-named archives and are recorded in the report.
Classify — game/faction/class from the keyword tables (warhammer_data.py), same as catalogue.py.
Match — fuzzy-match the (inner) name against data/library_catalog.json entries within the same faction; best sequence-match above --threshold (default 0.62) wins. The catalog provides canonical names, sub-factions and classes.
Plan — default output: a dry-run table source → target plus sorter_plan.jsonl. Nothing is touched.
Apply — create the target and hardlink the file there (os.link); if the filesystems differ, fall back to a real copy with a warning. Originals are never modified or deleted — that is what keeps the *arr-style workflow possible (remux/seed/sort independently). Every applied action is appended to sorter_applied.jsonl and re-runs skip targets that already exist.

Why hardlinks, and the unzip question

Hardlinks make "same file, two positions" free (0 extra bytes) — the original stays for seeding/history, the library copy is what torrenting sees. Limitation: hardlinks need source and library on the same filesystem; the sorter detects this and copies instead (warned).
Zip contents can be read without extracting (that is how the sorter peeks). Modifying a member in place is technically possible with zipfile but is not useful here.
Endgame (later stage): unpack archives into the unit folder so the library is plain files, easier to sort per-model and to torrent. The unpack step will live in the sorter (--unpack, 7z binary) but stays off until the link-based flow is proven on a small batch.

Extraction stage (`tools/extract_all.py`)

That endgame stage now exists as a standalone tool (the sorter stays plan/apply-only):

python3 tools/extract_all.py <src_root>... --out <extracted_root>

Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted into a folder named after the archive, preserving relative structure; nested archives are expanded recursively and deleted, so the tree ends up with no archives at all.
Plain files are hardlinked into the tree (fallback copy) — nothing lost, no duplicate data.
Sources are never modified. Failed extractions (HTML-page fake zips, password archives, disk full) are moved to <out>/.broken/ and the run continues; re-runs skip already-extracted destinations.
<out>/extract_manifest.jsonl records {archive, dest, unit} per extraction. The sorter's --provenance flag reads it: all files inside an extraction directory are classified once under the archive's name (a loose Backpack.stl alone is noise; under The Twin Lance/ it belongs to that unit) and placed together, keeping the archive's inner directory structure:
```
python3 sorter.py <extracted_root> --library <lib> \
    --provenance <extracted_root>/extract_manifest.jsonl --apply
```

Operational notes from the first full run (2026-10-06, sources unsorted/1..4 → extracted ≈ 1.4 TB / 260k files / 0 archives left; library rebuilt from scratch: 218592 hardlinked, 30584 skipped as already-present duplicates, 0 copied):

The CIFS NAS mount reports fabricated per-path inode numbers (noserverino behavior): two paths to the same hardlinked file show different st_ino, and the values are stable across calls. To verify a link, compare st_nlink (and content/size) — never inodes. Server-side links are real: a fresh os.link bumps nlink (2→3 in the checked case).
Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2 ("Cannot open output file"); extract_all.py then extracts to a /var/tmp temp dir, repairs the names and moves the result over.
__MACOSX/ and ._* AppleDouble entries are deleted as junk, not extracted.
Files under hidden trees (.broken/) are never planned; retries of the truncated/HTML dead-link downloads are listed in <extracted>/.broken/RETRY_URLS.txt (all 0807.st; re-run python3 dlp.py <channel> <out> --urls-file ... for them).
Non-warhammer systems are dropped (2026-10-07, user decision: "drop anything that isn't really warhammer"). The sorter default (DROPPED_SYSTEMS in dlphelper/sorter.py) is {Middle-Earth}: catalog entries of dropped systems are filtered at load and keyword hits for them count as unclassified, so nothing is ever placed into those sections - the files stay in _unsorted. --drop-system NAME drops more, --keep-system NAME re-enables one. The committed catalog snapshot itself is untouched.

Catalog file (`data/library_catalog.json`)

{
  "system": "Warhammer 40,000",
  "group": "Space Marines",
  "faction": "Space Wolves",
  "subfaction": null,
  "class": "Mounted",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}

Entries are matched case-insensitively against cleaned names. Growing the catalog is the main lever for match quality; unknown files surface in the plan report with their best score so gaps are obvious.

8.3 KiB Raw Blame History