extract stage + system-locked matching + provenance unit mode
- tools/extract_all.py: recursive extraction of all archives (incl. nested) into a plain-file tree; nested archives are deleted after successful extraction, plain files are hardlinked, failures go to .broken/, extract_manifest.jsonl records archive->dest->unit. - sorter --provenance: classify each extraction directory once under the archive's name (files of one archive are one unit; inner structure kept), so loose STLs sort with their model set. - matching discipline: a fuzzy match never leaves the file's system; system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit, and a same-system faction jump below 0.75 is rejected. Fixes 40k files landing in Middle-earth/Blood Bowl/AoS buckets. - speed for big trees: lru_cache on keyword matchers and pair scores, quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
@@ -95,6 +95,32 @@ graph LR
|
||||
unpack step will live in the sorter (`--unpack`, 7z binary) but stays off
|
||||
until the link-based flow is proven on a small batch.
|
||||
|
||||
## Extraction stage (`tools/extract_all.py`)
|
||||
|
||||
That endgame stage now exists as a standalone tool (the sorter stays
|
||||
plan/apply-only):
|
||||
|
||||
python3 tools/extract_all.py <src_root>... --out <extracted_root>
|
||||
|
||||
- Every archive (zip/7z/rar/tar/gz/bz2/xz, also nested ones) is extracted
|
||||
into a folder named after the archive, preserving relative structure;
|
||||
nested archives are expanded recursively and deleted, so the tree ends
|
||||
up with **no archives at all**.
|
||||
- Plain files are hardlinked into the tree (fallback copy) — nothing lost,
|
||||
no duplicate data.
|
||||
- Sources are never modified. Failed extractions (HTML-page fake zips,
|
||||
password archives, disk full) are moved to `<out>/.broken/` and the run
|
||||
continues; re-runs skip already-extracted destinations.
|
||||
- `<out>/extract_manifest.jsonl` records `{archive, dest, unit}` per
|
||||
extraction. The sorter's `--provenance` flag reads it: all files inside
|
||||
an extraction directory are classified **once under the archive's
|
||||
name** (a loose `Backpack.stl` alone is noise; under
|
||||
`The Twin Lance/` it belongs to that unit) and placed together, keeping
|
||||
the archive's inner directory structure:
|
||||
|
||||
python3 sorter.py <extracted_root> --library <lib> \
|
||||
--provenance <extracted_root>/extract_manifest.jsonl --apply
|
||||
|
||||
## Catalog file (`data/library_catalog.json`)
|
||||
|
||||
```json
|
||||
|
||||
Reference in New Issue
Block a user