Sorter: fuzzysort files into the miniset-aligned library via hardlinks

- docs/SORTING.md: target hierarchy (game > faction group > faction >
  subfaction > class > unit, miniset.net naming), canonical-name policy,
  hardlink policy, unzip outlook
- data/library_catalog.json: canonical catalog (name/aliases/subfaction/class)
- dlphelper/sorter.py: inspect (zip via zipfile, 7z via 7z l), keyword
  classification incl. inner-name voting for hash-named archives, fuzzy
  matching (difflib), plan/apply split, hardlink preferred with copy
  fallback, append-only audit, existing targets skipped, run artifacts
  filtered, HTML-page detection for bad downloads
- sorter.py CLI (plan by default, --apply to act)
This commit is contained in:
2026-10-04 21:17:52 +02:00
parent acb3415878
commit cccf10a308
4 changed files with 684 additions and 0 deletions
+111
View File
@@ -0,0 +1,111 @@
# Library structure & sorting pipeline
This document defines the target structure of the collection ("the library")
and how the sorter places files into it. It is the working agreement for all
sorting-related code; change it here first, then in code.
## Target hierarchy (miniset.net-aligned)
The hierarchy mirrors miniset.net ("Miniatures Collectors Guide"), extended
with the 40k force-organization class, which miniset itself uses as a
"specification" tag on set pages:
```
<library>/
Warhammer 40,000/ # game
Armies of the Imperium/ # faction group (miniset naming)
Adeptus Astartes/ # faction
Space Wolves/ # sub-faction (optional)
Fast Attack/ # force-org class
Thunderwolf Cavalry/ # canonical unit name
Thunderwolf Cavalry.7z # the file(s), or unpacked contents
```
Layer rules:
| Layer | Source of truth | Example | May be omitted? |
|-------|-----------------|---------|-----------------|
| Game | `warhammer_data.SYSTEMS` | `Warhammer 40,000` | no (else: `_unsorted/`) |
| Faction group | Imperium/Chaos/Xenos/etc. | `Armies of Chaos` | no (derived from faction) |
| Faction | miniset.net faction names | `Death Guard` | no (else `_unsorted/` in game) |
| Sub-faction | miniset sub-factions | `Space Wolves` | **yes** — only when known |
| Class | force-org slots | `Fast Attack` | **yes** — only when known |
| Unit | canonical model name (lexicanum spelling) | `Thunderwolf Cavalry` | yes (files drop directly into the class dir) |
- Files without a faction but with a game land in `<game>/_unsorted/`.
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
- Everything in between is optional and filled in as knowledge grows.
- Sub-faction folders are only created when the file name states it
(`Thunderwolf` → Space Wolves); otherwise files stay in the faction dir so
we don't invent hierarchy.
## Canonical names
Canonical unit names follow Lexicanum/GW spelling (`Keeper of Secrets`,
`Death Korps of Krieg`), because that is what cross-seeding, wikis and
miniset.net all agree on. Release names stay untouched inside the unit
folder.
## Sorting pipeline
```mermaid
graph LR
A[archive/dir] --> B[read name + zip/7z listing if possible]
B --> C[classify: system + faction + class<br/>keyword tables]
C --> D[fuzzy match unit name<br/>against library catalog]
D -->|score >= threshold| E[target: catalog entry]
D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
E & F --> G[plan: dry-run report]
G -->|apply| H[hardlink into library<br/>fallback: copy]
```
1. **Inspect** — read the file name; for archives, also read the *inner*
file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
no Python dependency). Inner names improve classification for hash-named
archives and are recorded in the report.
2. **Classify** — game/faction/class from the keyword tables
(`warhammer_data.py`), same as `catalogue.py`.
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
entries within the same faction; best sequence-match above
`--threshold` (default 0.62) wins. The catalog provides canonical names,
sub-factions and classes.
4. **Plan** — default output: a dry-run table `source → target` plus
`sorter_plan.jsonl`. Nothing is touched.
5. **Apply** — create the target and **hardlink** the file there
(`os.link`); if the filesystems differ, fall back to a real copy with a
warning. Originals are never modified or deleted — that is what keeps the
*arr-style workflow possible (remux/seed/sort independently). Every
applied action is appended to `sorter_applied.jsonl` and re-runs skip
targets that already exist.
## Why hardlinks, and the unzip question
- Hardlinks make "same file, two positions" free (0 extra bytes) — the
original stays for seeding/history, the library copy is what torrenting
sees. Limitation: hardlinks need source and library on the same
filesystem; the sorter detects this and copies instead (warned).
- Zip contents **can** be read without extracting (that is how the sorter
peeks). Modifying a member in place is technically possible with
`zipfile` but is not useful here.
- Endgame (later stage): unpack archives into the unit folder so the
library is plain files, easier to sort per-model and to torrent. The
unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
until the link-based flow is proven on a small batch.
## Catalog file (`data/library_catalog.json`)
```json
{
"system": "Warhammer 40,000",
"group": "Armies of the Imperium",
"faction": "Adeptus Astartes",
"subfaction": "Space Wolves",
"class": "Fast Attack",
"name": "Thunderwolf Cavalry",
"aliases": ["thunder wolf", "thunderwolf cavalry squad"]
}
```
Entries are matched case-insensitively against cleaned names. Growing the
catalog is the main lever for match quality; unknown files surface in the
plan report with their best score so gaps are obvious.