# Library structure & sorting pipeline

This document defines the target structure of the collection ("the library")
and how the sorter places files into it. It is the working agreement for all
sorting-related code; change it here first, then in code.

## Target hierarchy (Games Workshop webshop-aligned)

The hierarchy follows the official GW webshop taxonomy (dumped from the
shop's Algolia index by `dlphelper/gwshop.py` into
`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
hierarchy treated as sub-factions are full GW factions, and the class layer
holds GW shop categories (unit types / product lines) instead of the retired
40k force-organization slots:

```
<library>/
  Warhammer 40,000/                       # game (GW system)
    Space Marines/                        # GW group
      Raven Guard/                        # GW faction (chapter)
        Walker/                           # GW class (unit type / category)
          Redemptor Dreadnought/          # canonical unit name (catalog)
            Redemptor Dreadnought.zip     # the file(s), or unpacked contents
```

Layer rules:

| Layer | Source of truth | Example | May be omitted? |
|-------|-----------------|---------|-----------------|
| Game | `gwshop_catalog.json` (system) | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Group | GW `GameSystemsRoot` lvl1 | `Space Marines` | no (from catalog or `warhammer_data.GROUP_OF_FACTION`) |
| Faction | GW factions (incl. chapters) | `Raven Guard` | no (else `_unsorted/` in game) |
| Sub-faction | deeper sources only | — | **yes** — GW has none |
| Class | GW shop category | `Walker`, `Battleline`, `Rules & Accessories` | **yes** — only when known |
| Unit | canonical product name (GW catalog) | `Redemptor Dreadnought` | yes (files drop directly into the class dir) |

- Files without a faction but with a game land in `<game>/_unsorted/`.
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
  system/faction/class signal; a catalog match then overrides the hierarchy
  from the catalog entry, which is authoritative.
- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
  still usable via `sorter.py --catalog`; its sub-faction entries map into
  the same schema. For GW-sourced entries `subfaction` stays empty.

## Canonical names

Canonical unit names follow the GW webshop spelling (`Keeper of Secrets`,
`Thunderwolf Cavalry`), because that is what the catalog, the shop and
wikis all agree on. Release names stay untouched inside the unit folder.

## Sorting pipeline

```mermaid
graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]
```

1. **Inspect** — read the file name; for archives, also read the *inner*
   file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
   no Python dependency). Inner names improve classification for hash-named
   archives and are recorded in the report.
2. **Classify** — game/faction/class from the keyword tables
   (`warhammer_data.py`), same as `catalogue.py`.
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
   entries within the same faction; best sequence-match above
   `--threshold` (default 0.62) wins. The catalog provides canonical names,
   sub-factions and classes.
4. **Plan** — default output: a dry-run table `source → target` plus
   `sorter_plan.jsonl`. Nothing is touched.
5. **Apply** — create the target and **hardlink** the file there
   (`os.link`); if the filesystems differ, fall back to a real copy with a
   warning. Originals are never modified or deleted — that is what keeps the
   *arr-style workflow possible (remux/seed/sort independently). Every
   applied action is appended to `sorter_applied.jsonl` and re-runs skip
   targets that already exist.

## Why hardlinks, and the unzip question

- Hardlinks make "same file, two positions" free (0 extra bytes) — the
  original stays for seeding/history, the library copy is what torrenting
  sees. Limitation: hardlinks need source and library on the same
  filesystem; the sorter detects this and copies instead (warned).
- Zip contents **can** be read without extracting (that is how the sorter
  peeks). Modifying a member in place is technically possible with
  `zipfile` but is not useful here.
- Endgame (later stage): unpack archives into the unit folder so the
  library is plain files, easier to sort per-model and to torrent. The
  unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
  until the link-based flow is proven on a small batch.

## Catalog file (`data/library_catalog.json`)

```json
{
  "system": "Warhammer 40,000",
  "group": "Space Marines",
  "faction": "Space Wolves",
  "subfaction": null,
  "class": "Mounted",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}
```

Entries are matched case-insensitively against cleaned names. Growing the
catalog is the main lever for match quality; unknown files surface in the
plan report with their best score so gaps are obvious.