# Library structure & sorting pipeline

This document defines the target structure of the collection ("the library")
and how the sorter places files into it. It is the working agreement for all
sorting-related code; change it here first, then in code.

## Target hierarchy (miniset.net-aligned)

The hierarchy mirrors miniset.net ("Miniatures Collectors Guide"), extended
with the 40k force-organization class, which miniset itself uses as a
"specification" tag on set pages:

```
<library>/
  Warhammer 40,000/                       # game
    Armies of the Imperium/               # faction group (miniset naming)
      Adeptus Astartes/                   # faction
        Space Wolves/                     # sub-faction (optional)
          Fast Attack/                    # force-org class
            Thunderwolf Cavalry/          # canonical unit name
              Thunderwolf Cavalry.7z      # the file(s), or unpacked contents
```

Layer rules:

| Layer | Source of truth | Example | May be omitted? |
|-------|-----------------|---------|-----------------|
| Game | `warhammer_data.SYSTEMS` | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Faction group | Imperium/Chaos/Xenos/etc. | `Armies of Chaos` | no (derived from faction) |
| Faction | miniset.net faction names | `Death Guard` | no (else `_unsorted/` in game) |
| Sub-faction | miniset sub-factions | `Space Wolves` | **yes** — only when known |
| Class | force-org slots | `Fast Attack` | **yes** — only when known |
| Unit | canonical model name (lexicanum spelling) | `Thunderwolf Cavalry` | yes (files drop directly into the class dir) |

- Files without a faction but with a game land in `<game>/_unsorted/`.
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- Everything in between is optional and filled in as knowledge grows.
- Sub-faction folders are only created when the file name states it
  (`Thunderwolf` → Space Wolves); otherwise files stay in the faction dir so
  we don't invent hierarchy.

## Canonical names

Canonical unit names follow Lexicanum/GW spelling (`Keeper of Secrets`,
`Death Korps of Krieg`), because that is what cross-seeding, wikis and
miniset.net all agree on. Release names stay untouched inside the unit
folder.

## Sorting pipeline

```mermaid
graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]
```

1. **Inspect** — read the file name; for archives, also read the *inner*
   file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
   no Python dependency). Inner names improve classification for hash-named
   archives and are recorded in the report.
2. **Classify** — game/faction/class from the keyword tables
   (`warhammer_data.py`), same as `catalogue.py`.
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
   entries within the same faction; best sequence-match above
   `--threshold` (default 0.62) wins. The catalog provides canonical names,
   sub-factions and classes.
4. **Plan** — default output: a dry-run table `source → target` plus
   `sorter_plan.jsonl`. Nothing is touched.
5. **Apply** — create the target and **hardlink** the file there
   (`os.link`); if the filesystems differ, fall back to a real copy with a
   warning. Originals are never modified or deleted — that is what keeps the
   *arr-style workflow possible (remux/seed/sort independently). Every
   applied action is appended to `sorter_applied.jsonl` and re-runs skip
   targets that already exist.

## Why hardlinks, and the unzip question

- Hardlinks make "same file, two positions" free (0 extra bytes) — the
  original stays for seeding/history, the library copy is what torrenting
  sees. Limitation: hardlinks need source and library on the same
  filesystem; the sorter detects this and copies instead (warned).
- Zip contents **can** be read without extracting (that is how the sorter
  peeks). Modifying a member in place is technically possible with
  `zipfile` but is not useful here.
- Endgame (later stage): unpack archives into the unit folder so the
  library is plain files, easier to sort per-model and to torrent. The
  unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
  until the link-based flow is proven on a small batch.

## Catalog file (`data/library_catalog.json`)

```json
{
  "system": "Warhammer 40,000",
  "group": "Armies of the Imperium",
  "faction": "Adeptus Astartes",
  "subfaction": "Space Wolves",
  "class": "Fast Attack",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}
```

Entries are matched case-insensitively against cleaned names. Growing the
catalog is the main lever for match quality; unknown files surface in the
plan report with their best score so gaps are obvious.