extract stage + system-locked matching + provenance unit mode

- tools/extract_all.py: recursive extraction of all archives (incl.
  nested) into a plain-file tree; nested archives are deleted after
  successful extraction, plain files are hardlinked, failures go to
  .broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
  the archive's name (files of one archive are one unit; inner
  structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
  system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
  and a same-system faction jump below 0.75 is rejected. Fixes 40k
  files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
  quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
2026-10-05 08:09:22 +02:00
parent 7e28518a52
commit 86fcb3aecc
7 changed files with 455 additions and 26 deletions
+26
View File
@@ -95,6 +95,32 @@ graph LR
unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
until the link-based flow is proven on a small batch.
## Extraction stage (`tools/extract_all.py`)
That endgame stage now exists as a standalone tool (the sorter stays
plan/apply-only):
python3 tools/extract_all.py <src_root>... --out <extracted_root>
- Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
into a folder named after the archive, preserving relative structure;
nested archives are expanded recursively and deleted, so the tree ends
up with **no archives at all**.
- Plain files are hardlinked into the tree (fallback copy) — nothing lost,
no duplicate data.
- Sources are never modified. Failed extractions (HTML-page fake zips,
password archives, disk full) are moved to `<out>/.broken/` and the run
continues; re-runs skip already-extracted destinations.
- `<out>/extract_manifest.jsonl` records `{archive, dest, unit}` per
extraction. The sorter's `--provenance` flag reads it: all files inside
an extraction directory are classified **once under the archive's
name** (a loose `Backpack.stl` alone is noise; under
`The Twin Lance/` it belongs to that unit) and placed together, keeping
the archive's inner directory structure:
python3 sorter.py <extracted_root> --library <lib> \
--provenance <extracted_root>/extract_manifest.jsonl --apply
## Catalog file (`data/library_catalog.json`)
```json