extract stage + system-locked matching + provenance unit mode

- tools/extract_all.py: recursive extraction of all archives (incl.
  nested) into a plain-file tree; nested archives are deleted after
  successful extraction, plain files are hardlinked, failures go to
  .broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
  the archive's name (files of one archive are one unit; inner
  structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
  system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
  and a same-system faction jump below 0.75 is rejected. Fixes 40k
  files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
  quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
2026-10-05 08:09:22 +02:00
parent 7e28518a52
commit 86fcb3aecc
7 changed files with 455 additions and 26 deletions
+14 -1
View File
@@ -52,6 +52,9 @@ def main() -> None:
parser.add_argument("--apply", action="store_true",
help="create the targets (hardlink, fallback copy)")
parser.add_argument("--audit", type=Path, default=None, metavar="FILE")
parser.add_argument("--provenance", type=Path, default=None, metavar="FILE",
help="extract_manifest.jsonl from tools/extract_all.py: "
"sort whole extraction directories as units")
parser.add_argument("--json", action="store_true",
help="print the plan as JSON instead of text")
args = parser.parse_args()
@@ -65,7 +68,17 @@ def main() -> None:
catalog = load_catalog(args.catalog)
audit = args.audit or (args.library / "sorter_applied.jsonl")
items = sort_plan(files, args.library, catalog, threshold=args.threshold)
provenance = None
if args.provenance:
provenance = {}
with args.provenance.open(encoding="utf-8") as fh:
for line in fh:
rec = json.loads(line)
provenance[rec["dest"]] = rec["unit"]
log.info("Provenance: %d extraction directories", len(provenance))
items = sort_plan(files, args.library, catalog, threshold=args.threshold,
provenance=provenance, root=source)
if args.json:
print(json.dumps([