Sorter: fuzzysort files into the miniset-aligned library via hardlinks
- docs/SORTING.md: target hierarchy (game > faction group > faction > subfaction > class > unit, miniset.net naming), canonical-name policy, hardlink policy, unzip outlook - data/library_catalog.json: canonical catalog (name/aliases/subfaction/class) - dlphelper/sorter.py: inspect (zip via zipfile, 7z via 7z l), keyword classification incl. inner-name voting for hash-named archives, fuzzy matching (difflib), plan/apply split, hardlink preferred with copy fallback, append-only audit, existing targets skipped, run artifacts filtered, HTML-page detection for bad downloads - sorter.py CLI (plan by default, --apply to act)
This commit is contained in:
+111
@@ -0,0 +1,111 @@
|
||||
# Library structure & sorting pipeline
|
||||
|
||||
This document defines the target structure of the collection ("the library")
|
||||
and how the sorter places files into it. It is the working agreement for all
|
||||
sorting-related code; change it here first, then in code.
|
||||
|
||||
## Target hierarchy (miniset.net-aligned)
|
||||
|
||||
The hierarchy mirrors miniset.net ("Miniatures Collectors Guide"), extended
|
||||
with the 40k force-organization class, which miniset itself uses as a
|
||||
"specification" tag on set pages:
|
||||
|
||||
```
|
||||
<library>/
|
||||
Warhammer 40,000/ # game
|
||||
Armies of the Imperium/ # faction group (miniset naming)
|
||||
Adeptus Astartes/ # faction
|
||||
Space Wolves/ # sub-faction (optional)
|
||||
Fast Attack/ # force-org class
|
||||
Thunderwolf Cavalry/ # canonical unit name
|
||||
Thunderwolf Cavalry.7z # the file(s), or unpacked contents
|
||||
```
|
||||
|
||||
Layer rules:
|
||||
|
||||
| Layer | Source of truth | Example | May be omitted? |
|
||||
|-------|-----------------|---------|-----------------|
|
||||
| Game | `warhammer_data.SYSTEMS` | `Warhammer 40,000` | no (else: `_unsorted/`) |
|
||||
| Faction group | Imperium/Chaos/Xenos/etc. | `Armies of Chaos` | no (derived from faction) |
|
||||
| Faction | miniset.net faction names | `Death Guard` | no (else `_unsorted/` in game) |
|
||||
| Sub-faction | miniset sub-factions | `Space Wolves` | **yes** — only when known |
|
||||
| Class | force-org slots | `Fast Attack` | **yes** — only when known |
|
||||
| Unit | canonical model name (lexicanum spelling) | `Thunderwolf Cavalry` | yes (files drop directly into the class dir) |
|
||||
|
||||
- Files without a faction but with a game land in `<game>/_unsorted/`.
|
||||
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
|
||||
- Everything in between is optional and filled in as knowledge grows.
|
||||
- Sub-faction folders are only created when the file name states it
|
||||
(`Thunderwolf` → Space Wolves); otherwise files stay in the faction dir so
|
||||
we don't invent hierarchy.
|
||||
|
||||
## Canonical names
|
||||
|
||||
Canonical unit names follow Lexicanum/GW spelling (`Keeper of Secrets`,
|
||||
`Death Korps of Krieg`), because that is what cross-seeding, wikis and
|
||||
miniset.net all agree on. Release names stay untouched inside the unit
|
||||
folder.
|
||||
|
||||
## Sorting pipeline
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[archive/dir] --> B[read name + zip/7z listing if possible]
|
||||
B --> C[classify: system + faction + class<br/>keyword tables]
|
||||
C --> D[fuzzy match unit name<br/>against library catalog]
|
||||
D -->|score >= threshold| E[target: catalog entry]
|
||||
D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
|
||||
E & F --> G[plan: dry-run report]
|
||||
G -->|apply| H[hardlink into library<br/>fallback: copy]
|
||||
```
|
||||
|
||||
1. **Inspect** — read the file name; for archives, also read the *inner*
|
||||
file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
|
||||
no Python dependency). Inner names improve classification for hash-named
|
||||
archives and are recorded in the report.
|
||||
2. **Classify** — game/faction/class from the keyword tables
|
||||
(`warhammer_data.py`), same as `catalogue.py`.
|
||||
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
|
||||
entries within the same faction; best sequence-match above
|
||||
`--threshold` (default 0.62) wins. The catalog provides canonical names,
|
||||
sub-factions and classes.
|
||||
4. **Plan** — default output: a dry-run table `source → target` plus
|
||||
`sorter_plan.jsonl`. Nothing is touched.
|
||||
5. **Apply** — create the target and **hardlink** the file there
|
||||
(`os.link`); if the filesystems differ, fall back to a real copy with a
|
||||
warning. Originals are never modified or deleted — that is what keeps the
|
||||
*arr-style workflow possible (remux/seed/sort independently). Every
|
||||
applied action is appended to `sorter_applied.jsonl` and re-runs skip
|
||||
targets that already exist.
|
||||
|
||||
## Why hardlinks, and the unzip question
|
||||
|
||||
- Hardlinks make "same file, two positions" free (0 extra bytes) — the
|
||||
original stays for seeding/history, the library copy is what torrenting
|
||||
sees. Limitation: hardlinks need source and library on the same
|
||||
filesystem; the sorter detects this and copies instead (warned).
|
||||
- Zip contents **can** be read without extracting (that is how the sorter
|
||||
peeks). Modifying a member in place is technically possible with
|
||||
`zipfile` but is not useful here.
|
||||
- Endgame (later stage): unpack archives into the unit folder so the
|
||||
library is plain files, easier to sort per-model and to torrent. The
|
||||
unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
|
||||
until the link-based flow is proven on a small batch.
|
||||
|
||||
## Catalog file (`data/library_catalog.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"system": "Warhammer 40,000",
|
||||
"group": "Armies of the Imperium",
|
||||
"faction": "Adeptus Astartes",
|
||||
"subfaction": "Space Wolves",
|
||||
"class": "Fast Attack",
|
||||
"name": "Thunderwolf Cavalry",
|
||||
"aliases": ["thunder wolf", "thunderwolf cavalry squad"]
|
||||
}
|
||||
```
|
||||
|
||||
Entries are matched case-insensitively against cleaned names. Growing the
|
||||
catalog is the main lever for match quality; unknown files surface in the
|
||||
plan report with their best score so gaps are obvious.
|
||||
Reference in New Issue
Block a user