# Library structure & sorting pipeline

This document defines the target structure of the collection ("the library")
and how the sorter places files into it. It is the working agreement for all
sorting-related code; change it here first, then in code.

## Target hierarchy (Games Workshop webshop-aligned)

The hierarchy follows the official GW webshop taxonomy (dumped from the
shop's Algolia index by `dlphelper/gwshop.py` into
`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
hierarchy treated as sub-factions are full GW factions, and the class layer
holds GW shop categories (unit types / product lines) instead of the retired
40k force-organization slots:

```
<library>/
  Warhammer 40,000/                       # game (GW system)
    Space Marines/                        # GW group
      Raven Guard/                        # GW faction (chapter)
        Walker/                           # GW class (unit type / category)
          Redemptor Dreadnought/          # canonical unit name (catalog)
            Redemptor Dreadnought.zip     # the file(s), or unpacked contents
```

Layer rules:

| Layer | Source of truth | Example | May be omitted? |
|-------|-----------------|---------|-----------------|
| Game | `gwshop_catalog.json` (system) | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Group | GW `GameSystemsRoot` lvl1 | `Space Marines` | no (from catalog or `warhammer_data.GROUP_OF_FACTION`) |
| Faction | GW factions (incl. chapters) | `Raven Guard` | no (else `_unsorted/` in game) |
| Sub-faction | deeper sources only | — | **yes** — GW has none |
| Class | GW shop category | `Walker`, `Battleline`, `Rules & Accessories` | **yes** — only when known |
| Unit | canonical product name (GW catalog) | `Redemptor Dreadnought` | yes (files drop directly into the class dir) |

- Files without a faction but with a game land in `<game>/_unsorted/`.
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
  system/faction/class signal; a catalog match then overrides the hierarchy
  from the catalog entry, which is authoritative.
- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
  still usable via `sorter.py --catalog`; its sub-faction entries map into
  the same schema. For GW-sourced entries `subfaction` stays empty.

## Canonical names

Canonical unit names follow the GW webshop spelling (`Keeper of Secrets`,
`Thunderwolf Cavalry`), because that is what the catalog, the shop and
wikis all agree on. Release names stay untouched inside the unit folder.

## Sorting pipeline

```mermaid
graph LR
    A[archive/dir] --> B[read name + zip/7z listing if possible]
    B --> C[classify: system + faction + class<br/>keyword tables]
    C --> D[fuzzy match unit name<br/>against library catalog]
    D -->|score >= threshold| E[target: catalog entry]
    D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
    E & F --> G[plan: dry-run report]
    G -->|apply| H[hardlink into library<br/>fallback: copy]
```

1. **Inspect** — read the file name; for archives, also read the *inner*
   file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
   no Python dependency). Inner names improve classification for hash-named
   archives and are recorded in the report.
2. **Classify** — game/faction/class from the keyword tables
   (`warhammer_data.py`), same as `catalogue.py`.
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
   entries within the same faction; best sequence-match above
   `--threshold` (default 0.62) wins. The catalog provides canonical names,
   sub-factions and classes.
4. **Plan** — default output: a dry-run table `source → target` plus
   `sorter_plan.jsonl`. Nothing is touched.
5. **Apply** — create the target and **hardlink** the file there
   (`os.link`); if the filesystems differ, fall back to a real copy with a
   warning. Originals are never modified or deleted — that is what keeps the
   *arr-style workflow possible (remux/seed/sort independently). Every
   applied action is appended to `sorter_applied.jsonl` and re-runs skip
   targets that already exist.

## Bookkeeping location

The audit (`sorter_applied.jsonl`) and the low-confidence review doc
(`sorter_low_confidence.txt`) are written to `<library>/../_logs/` by
default — **never inside the library itself**, because the library may be
a seeded torrent whose file set is frozen (a rewritten audit file would
fail the client's recheck). `--audit FILE` overrides this; the low-
confidence doc always lands next to the audit file.

## Why hardlinks, and the unzip question

- Hardlinks make "same file, two positions" free (0 extra bytes) — the
  original stays for seeding/history, the library copy is what torrenting
  sees. Limitation: hardlinks need source and library on the same
  filesystem; the sorter detects this and copies instead (warned).
- Zip contents **can** be read without extracting (that is how the sorter
  peeks). Modifying a member in place is technically possible with
  `zipfile` but is not useful here.
- Endgame (later stage): unpack archives into the unit folder so the
  library is plain files, easier to sort per-model and to torrent. The
  unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
  until the link-based flow is proven on a small batch.

## Extraction stage (`tools/extract_all.py`)

That endgame stage now exists as a standalone tool (the sorter stays
plan/apply-only):

    python3 tools/extract_all.py <src_root>... --out <extracted_root>

- Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
  into a folder named after the archive, preserving relative structure;
  nested archives are expanded recursively and deleted, so the tree ends
  up with **no archives at all**.
- Plain files are hardlinked into the tree (fallback copy) — nothing lost,
  no duplicate data.
- Sources are never modified. Failed extractions (HTML-page fake zips,
  password archives, disk full) are moved to `<out>/.broken/` and the run
  continues; re-runs skip already-extracted destinations.
- `<out>/extract_manifest.jsonl` records `{archive, dest, unit}` per
  extraction. The sorter's `--provenance` flag reads it: all files inside
  an extraction directory are classified **once under the archive's
  name** (a loose `Backpack.stl` alone is noise; under
  `The Twin Lance/` it belongs to that unit) and placed together, keeping
  the archive's inner directory structure:

      python3 sorter.py <extracted_root> --library <lib> \
          --provenance <extracted_root>/extract_manifest.jsonl --apply

Operational notes from the first full run (2026-10-06, sources
`unsorted/1..4` → `extracted` ≈ 1.4 TB / 260k files / 0 archives left;
library rebuilt from scratch: 218592 hardlinked, 30584 skipped as
already-present duplicates, 0 copied):

- The CIFS NAS mount reports **fabricated per-path inode numbers**
  (`noserverino` behavior): two paths to the same hardlinked file show
  different `st_ino`, and the values are stable across calls. To verify a
  link, compare `st_nlink` (and content/size) — never inodes. Server-side
  links are real: a fresh `os.link` bumps nlink (2→3 in the checked case).
- Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2
  ("Cannot open output file"); `extract_all.py` then extracts to a
  `/var/tmp` temp dir, repairs the names and moves the result over.
- `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not
  extracted.
- Files under hidden trees (`.broken/`) are never planned; retries of the
  truncated/HTML dead-link downloads are listed in
  `<extracted>/.broken/RETRY_URLS.txt` (all 0807.st; re-run
  `python3 dlp.py <channel> <out> --urls-file ...` for them).
- **Non-warhammer systems are dropped** (2026-10-07, user decision:
  "drop anything that isn't really warhammer"). The sorter default
  (`DROPPED_SYSTEMS` in `dlphelper/sorter.py`) is `{Middle-Earth}`:
  catalog entries of dropped systems are filtered at load and keyword
  hits for them count as unclassified, so nothing is ever placed into
  those sections - the files stay in `_unsorted`. `--drop-system NAME`
  drops more, `--keep-system NAME` re-enables one. The committed catalog
  snapshot itself is untouched.
- **Probe votes on full inner paths** (2026-10-07): the pack probe's
  inner-name votes clean the whole relative path (dir names included),
  not just the file stem - hash-named 0807 packs resolve their faction
  from folders like `Catachan/Jungle Fighters/` instead of from one
  root file named `lieutenant`. The token-containment boost is disabled
  for single-token inner votes (one generic-ish word must not claim a
  pack), and full-catalog/different-faction adoptions additionally
  require token containment (`_contained_somewhere`). A matched catalog
  unit's class layer always wins (no keyword class survives next to a
  precise unit match).

## Catalog file (`data/library_catalog.json`)

```json
{
  "system": "Warhammer 40,000",
  "group": "Space Marines",
  "faction": "Space Wolves",
  "subfaction": null,
  "class": "Mounted",
  "name": "Thunderwolf Cavalry",
  "aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}
```

Entries are matched case-insensitively against cleaned names. Growing the
catalog is the main lever for match quality; unknown files surface in the
plan report with their best score so gaps are obvious.