extract stage + system-locked matching + provenance unit mode
- tools/extract_all.py: recursive extraction of all archives (incl. nested) into a plain-file tree; nested archives are deleted after successful extraction, plain files are hardlinked, failures go to .broken/, extract_manifest.jsonl records archive->dest->unit. - sorter --provenance: classify each extraction directory once under the archive's name (files of one archive are one unit; inner structure kept), so loose STLs sort with their model set. - matching discipline: a fuzzy match never leaves the file's system; system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit, and a same-system faction jump below 0.75 is rejected. Fixes 40k files landing in Middle-earth/Blood Bowl/AoS buckets. - speed for big trees: lru_cache on keyword matchers and pair scores, quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
@@ -52,6 +52,9 @@ def main() -> None:
|
||||
parser.add_argument("--apply", action="store_true",
|
||||
help="create the targets (hardlink, fallback copy)")
|
||||
parser.add_argument("--audit", type=Path, default=None, metavar="FILE")
|
||||
parser.add_argument("--provenance", type=Path, default=None, metavar="FILE",
|
||||
help="extract_manifest.jsonl from tools/extract_all.py: "
|
||||
"sort whole extraction directories as units")
|
||||
parser.add_argument("--json", action="store_true",
|
||||
help="print the plan as JSON instead of text")
|
||||
args = parser.parse_args()
|
||||
@@ -65,7 +68,17 @@ def main() -> None:
|
||||
catalog = load_catalog(args.catalog)
|
||||
audit = args.audit or (args.library / "sorter_applied.jsonl")
|
||||
|
||||
items = sort_plan(files, args.library, catalog, threshold=args.threshold)
|
||||
provenance = None
|
||||
if args.provenance:
|
||||
provenance = {}
|
||||
with args.provenance.open(encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
rec = json.loads(line)
|
||||
provenance[rec["dest"]] = rec["unit"]
|
||||
log.info("Provenance: %d extraction directories", len(provenance))
|
||||
|
||||
items = sort_plan(files, args.library, catalog, threshold=args.threshold,
|
||||
provenance=provenance, root=source)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps([
|
||||
|
||||
Reference in New Issue
Block a user