Warhammer_DLP_Helper/docs/SORTING.md

Library structure & sorting pipeline

This document defines the target structure of the collection ("the library") and how the sorter places files into it. It is the working agreement for all sorting-related code; change it here first, then in code.

Target hierarchy (miniset.net-aligned)

The hierarchy mirrors miniset.net ("Miniatures Collectors Guide"), extended with the 40k force-organization class, which miniset itself uses as a "specification" tag on set pages:

<library>/
  Warhammer 40,000/                       # game
    Armies of the Imperium/               # faction group (miniset naming)
      Adeptus Astartes/                   # faction
        Space Wolves/                     # sub-faction (optional)
          Fast Attack/                    # force-org class
            Thunderwolf Cavalry/          # canonical unit name
              Thunderwolf Cavalry.7z      # the file(s), or unpacked contents

Layer rules:

Layer	Source of truth	Example	May be omitted?
Game	warhammer_data.SYSTEMS	Warhammer 40,000	no (else: _unsorted/)
Faction group	Imperium/Chaos/Xenos/etc.	Armies of Chaos	no (derived from faction)
Faction	miniset.net faction names	Death Guard	no (else _unsorted/ in game)
Sub-faction	miniset sub-factions	Space Wolves	yes — only when known
Class	force-org slots	Fast Attack	yes — only when known
Unit	canonical model name (lexicanum spelling)	Thunderwolf Cavalry	yes (files drop directly into the class dir)

• Files without a faction but with a game land in <game>/_unsorted/.
• Fully unclassifiable files land in <library>/_unsorted/ — never guessed.
• Everything in between is optional and filled in as knowledge grows.
• Sub-faction folders are only created when the file name states it (Thunderwolf → Space Wolves); otherwise files stay in the faction dir so we don't invent hierarchy.

Canonical names

Canonical unit names follow Lexicanum/GW spelling (Keeper of Secrets, Death Korps of Krieg), because that is what cross-seeding, wikis and miniset.net all agree on. Release names stay untouched inside the unit folder.

Sorting pipeline

graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]

1. Inspect — read the file name; for archives, also read the inner file list (zip via stdlib zipfile, 7z via the installed 7z l binary — no Python dependency). Inner names improve classification for hash-named archives and are recorded in the report.
2. Classify — game/faction/class from the keyword tables (warhammer_data.py), same as catalogue.py.
3. Match — fuzzy-match the (inner) name against data/library_catalog.json entries within the same faction; best sequence-match above --threshold (default 0.62) wins. The catalog provides canonical names, sub-factions and classes.
4. Plan — default output: a dry-run table source → target plus sorter_plan.jsonl. Nothing is touched.
5. Apply — create the target and hardlink the file there (os.link); if the filesystems differ, fall back to a real copy with a warning. Originals are never modified or deleted — that is what keeps the *arr-style workflow possible (remux/seed/sort independently). Every applied action is appended to sorter_applied.jsonl and re-runs skip targets that already exist.

Why hardlinks, and the unzip question

• Hardlinks make "same file, two positions" free (0 extra bytes) — the original stays for seeding/history, the library copy is what torrenting sees. Limitation: hardlinks need source and library on the same filesystem; the sorter detects this and copies instead (warned).
• Zip contents can be read without extracting (that is how the sorter peeks). Modifying a member in place is technically possible with zipfile but is not useful here.
• Endgame (later stage): unpack archives into the unit folder so the library is plain files, easier to sort per-model and to torrent. The unpack step will live in the sorter (--unpack, 7z binary) but stays off until the link-based flow is proven on a small batch.

Catalog file (data/library_catalog.json)

{
  "system": "Warhammer 40,000",
  "group": "Armies of the Imperium",
  "faction": "Adeptus Astartes",
  "subfaction": "Space Wolves",
  "class": "Fast Attack",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}

Entries are matched case-insensitively against cleaned names. Growing the catalog is the main lever for match quality; unknown files surface in the plan report with their best score so gaps are obvious.