chore: start transition layout — python into version-python/, scaffold version-rust/
- git mv of all existing files into version-python/ (history preserved) - version-rust/: minimal cargo scaffold (edition 2024), no deps yet - root AGENTS.md documents the transition rules and artifact/schema contract - main is untouched; the python layout there stays as-is
This commit is contained in:
@@ -0,0 +1,187 @@
|
||||
# Library structure & sorting pipeline
|
||||
|
||||
This document defines the target structure of the collection ("the library")
|
||||
and how the sorter places files into it. It is the working agreement for all
|
||||
sorting-related code; change it here first, then in code.
|
||||
|
||||
## Target hierarchy (Games Workshop webshop-aligned)
|
||||
|
||||
The hierarchy follows the official GW webshop taxonomy (dumped from the
|
||||
shop's Algolia index by `dlphelper/gwshop.py` into
|
||||
`data/gwshop_catalog.json`). Chapters and sub-lines that the old miniset
|
||||
hierarchy treated as sub-factions are full GW factions, and the class layer
|
||||
holds GW shop categories (unit types / product lines) instead of the retired
|
||||
40k force-organization slots:
|
||||
|
||||
```
|
||||
<library>/
|
||||
Warhammer 40,000/ # game (GW system)
|
||||
Space Marines/ # GW group
|
||||
Raven Guard/ # GW faction (chapter)
|
||||
Walker/ # GW class (unit type / category)
|
||||
Redemptor Dreadnought/ # canonical unit name (catalog)
|
||||
Redemptor Dreadnought.zip # the file(s), or unpacked contents
|
||||
```
|
||||
|
||||
Layer rules:
|
||||
|
||||
| Layer | Source of truth | Example | May be omitted? |
|
||||
|-------|-----------------|---------|-----------------|
|
||||
| Game | `gwshop_catalog.json` (system) | `Warhammer 40,000` | no (else: `_unsorted/`) |
|
||||
| Group | GW `GameSystemsRoot` lvl1 | `Space Marines` | no (from catalog or `warhammer_data.GROUP_OF_FACTION`) |
|
||||
| Faction | GW factions (incl. chapters) | `Raven Guard` | no (else `_unsorted/` in game) |
|
||||
| Sub-faction | deeper sources only | — | **yes** — GW has none |
|
||||
| Class | GW shop category | `Walker`, `Battleline`, `Rules & Accessories` | **yes** — only when known |
|
||||
| Unit | canonical product name (GW catalog) | `Redemptor Dreadnought` | yes (files drop directly into the class dir) |
|
||||
|
||||
- Files without a faction but with a game land in `<game>/_unsorted/`.
|
||||
- Fully unclassifiable files land in `<library>/_unsorted/` — never guessed.
|
||||
- The keyword tables in `warhammer_data.py` (GW naming) provide the initial
|
||||
system/faction/class signal; a catalog match then overrides the hierarchy
|
||||
from the catalog entry, which is authoritative.
|
||||
- `data/library_catalog.json` (the older miniset-aligned hand catalog) is
|
||||
still usable via `sorter.py --catalog`; its sub-faction entries map into
|
||||
the same schema. For GW-sourced entries `subfaction` stays empty.
|
||||
|
||||
## Canonical names
|
||||
|
||||
Canonical unit names follow the GW webshop spelling (`Keeper of Secrets`,
|
||||
`Thunderwolf Cavalry`), because that is what the catalog, the shop and
|
||||
wikis all agree on. Release names stay untouched inside the unit folder.
|
||||
|
||||
## Sorting pipeline
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[archive/dir] --> B[read name + zip/7z listing if possible]
|
||||
B --> C[classify: system + faction + class<br/>keyword tables]
|
||||
C --> D[fuzzy match unit name<br/>against library catalog]
|
||||
D -->|score >= threshold| E[target: catalog entry]
|
||||
D -->|no match| F[implicit entry: own cleaned name<br/>flagged for review]
|
||||
E & F --> G[plan: dry-run report]
|
||||
G -->|apply| H[hardlink into library<br/>fallback: copy]
|
||||
```
|
||||
|
||||
1. **Inspect** — read the file name; for archives, also read the *inner*
|
||||
file list (zip via stdlib `zipfile`, 7z via the installed `7z l` binary —
|
||||
no Python dependency). Inner names improve classification for hash-named
|
||||
archives and are recorded in the report.
|
||||
2. **Classify** — game/faction/class from the keyword tables
|
||||
(`warhammer_data.py`), same as `catalogue.py`.
|
||||
3. **Match** — fuzzy-match the (inner) name against `data/library_catalog.json`
|
||||
entries within the same faction; best sequence-match above
|
||||
`--threshold` (default 0.62) wins. The catalog provides canonical names,
|
||||
sub-factions and classes.
|
||||
4. **Plan** — default output: a dry-run table `source → target` plus
|
||||
`sorter_plan.jsonl`. Nothing is touched.
|
||||
5. **Apply** — create the target and **hardlink** the file there
|
||||
(`os.link`); if the filesystems differ, fall back to a real copy with a
|
||||
warning. Originals are never modified or deleted — that is what keeps the
|
||||
*arr-style workflow possible (remux/seed/sort independently). Every
|
||||
applied action is appended to `sorter_applied.jsonl` and re-runs skip
|
||||
targets that already exist.
|
||||
|
||||
## Bookkeeping location
|
||||
|
||||
The audit (`sorter_applied.jsonl`) and the low-confidence review doc
|
||||
(`sorter_low_confidence.txt`) are written to `<library>/../_logs/` by
|
||||
default — **never inside the library itself**, because the library may be
|
||||
a seeded torrent whose file set is frozen (a rewritten audit file would
|
||||
fail the client's recheck). `--audit FILE` overrides this; the low-
|
||||
confidence doc always lands next to the audit file.
|
||||
|
||||
## Why hardlinks, and the unzip question
|
||||
|
||||
- Hardlinks make "same file, two positions" free (0 extra bytes) — the
|
||||
original stays for seeding/history, the library copy is what torrenting
|
||||
sees. Limitation: hardlinks need source and library on the same
|
||||
filesystem; the sorter detects this and copies instead (warned).
|
||||
- Zip contents **can** be read without extracting (that is how the sorter
|
||||
peeks). Modifying a member in place is technically possible with
|
||||
`zipfile` but is not useful here.
|
||||
- Endgame (later stage): unpack archives into the unit folder so the
|
||||
library is plain files, easier to sort per-model and to torrent. The
|
||||
unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
|
||||
until the link-based flow is proven on a small batch.
|
||||
|
||||
## Extraction stage (`tools/extract_all.py`)
|
||||
|
||||
That endgame stage now exists as a standalone tool (the sorter stays
|
||||
plan/apply-only):
|
||||
|
||||
python3 tools/extract_all.py <src_root>... --out <extracted_root>
|
||||
|
||||
- Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
|
||||
into a folder named after the archive, preserving relative structure;
|
||||
nested archives are expanded recursively and deleted, so the tree ends
|
||||
up with **no archives at all**.
|
||||
- Plain files are hardlinked into the tree (fallback copy) — nothing lost,
|
||||
no duplicate data.
|
||||
- Sources are never modified. Failed extractions (HTML-page fake zips,
|
||||
password archives, disk full) are moved to `<out>/.broken/` and the run
|
||||
continues; re-runs skip already-extracted destinations.
|
||||
- `<out>/extract_manifest.jsonl` records `{archive, dest, unit}` per
|
||||
extraction. The sorter's `--provenance` flag reads it: all files inside
|
||||
an extraction directory are classified **once under the archive's
|
||||
name** (a loose `Backpack.stl` alone is noise; under
|
||||
`The Twin Lance/` it belongs to that unit) and placed together, keeping
|
||||
the archive's inner directory structure:
|
||||
|
||||
python3 sorter.py <extracted_root> --library <lib> \
|
||||
--provenance <extracted_root>/extract_manifest.jsonl --apply
|
||||
|
||||
Operational notes from the first full run (2026-10-06, sources
|
||||
`unsorted/1..4` → `extracted` ≈ 1.4 TB / 260k files / 0 archives left;
|
||||
library rebuilt from scratch: 218592 hardlinked, 30584 skipped as
|
||||
already-present duplicates, 0 copied):
|
||||
|
||||
- The CIFS NAS mount reports **fabricated per-path inode numbers**
|
||||
(`noserverino` behavior): two paths to the same hardlinked file show
|
||||
different `st_ino`, and the values are stable across calls. To verify a
|
||||
link, compare `st_nlink` (and content/size) — never inodes. Server-side
|
||||
links are real: a fresh `os.link` bumps nlink (2→3 in the checked case).
|
||||
- Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2
|
||||
("Cannot open output file"); `extract_all.py` then extracts to a
|
||||
`/var/tmp` temp dir, repairs the names and moves the result over.
|
||||
- `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not
|
||||
extracted.
|
||||
- Files under hidden trees (`.broken/`) are never planned; retries of the
|
||||
truncated/HTML dead-link downloads are listed in
|
||||
`<extracted>/.broken/RETRY_URLS.txt` (all 0807.st; re-run
|
||||
`python3 dlp.py <channel> <out> --urls-file ...` for them).
|
||||
- **Non-warhammer systems are dropped** (2026-10-07, user decision:
|
||||
"drop anything that isn't really warhammer"). The sorter default
|
||||
(`DROPPED_SYSTEMS` in `dlphelper/sorter.py`) is `{Middle-Earth}`:
|
||||
catalog entries of dropped systems are filtered at load and keyword
|
||||
hits for them count as unclassified, so nothing is ever placed into
|
||||
those sections - the files stay in `_unsorted`. `--drop-system NAME`
|
||||
drops more, `--keep-system NAME` re-enables one. The committed catalog
|
||||
snapshot itself is untouched.
|
||||
- **Probe votes on full inner paths** (2026-10-07): the pack probe's
|
||||
inner-name votes clean the whole relative path (dir names included),
|
||||
not just the file stem - hash-named 0807 packs resolve their faction
|
||||
from folders like `Catachan/Jungle Fighters/` instead of from one
|
||||
root file named `lieutenant`. The token-containment boost is disabled
|
||||
for single-token inner votes (one generic-ish word must not claim a
|
||||
pack), and full-catalog/different-faction adoptions additionally
|
||||
require token containment (`_contained_somewhere`). A matched catalog
|
||||
unit's class layer always wins (no keyword class survives next to a
|
||||
precise unit match).
|
||||
|
||||
## Catalog file (`data/library_catalog.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"system": "Warhammer 40,000",
|
||||
"group": "Space Marines",
|
||||
"faction": "Space Wolves",
|
||||
"subfaction": null,
|
||||
"class": "Mounted",
|
||||
"name": "Thunderwolf Cavalry",
|
||||
"aliases": ["thunder wolf", "thunderwolf cavalry squad"]
|
||||
}
|
||||
```
|
||||
|
||||
Entries are matched case-insensitively against cleaned names. Growing the
|
||||
catalog is the main lever for match quality; unknown files surface in the
|
||||
plan report with their best score so gaps are obvious.
|
||||
Reference in New Issue
Block a user