extract stage + system-locked matching + provenance unit mode

- tools/extract_all.py: recursive extraction of all archives (incl.
  nested) into a plain-file tree; nested archives are deleted after
  successful extraction, plain files are hardlinked, failures go to
  .broken/, extract_manifest.jsonl records archive->dest->unit.
- sorter --provenance: classify each extraction directory once under
  the archive's name (files of one archive are one unit; inner
  structure kept), so loose STLs sort with their model set.
- matching discipline: a fuzzy match never leaves the file's system;
  system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit,
  and a same-system faction jump below 0.75 is rejected. Fixes 40k
  files landing in Middle-earth/Blood Bowl/AoS buckets.
- speed for big trees: lru_cache on keyword matchers and pair scores,
  quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
2026-10-05 08:09:22 +02:00
parent 7e28518a52
commit 86fcb3aecc
7 changed files with 455 additions and 26 deletions
+18 -1
View File
@@ -56,10 +56,14 @@ dlphelper/
categories as classes, GROUP_OF_FACTION map
gwshop.py GW webshop catalog dump via Algolia (see §7.3)
sorter.py library sorter: inspect/classify/fuzzy-match
against the catalog, plan + hardlink apply
against the catalog, plan + hardlink apply,
optional --provenance unit mode for extracted trees
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py inspect raw gofile API response for one content id
tools/extract_all.py recursive archive extraction into a plain-file tree
(extract_manifest.jsonl feeds the sorter's
--provenance unit mode; failures go to .broken/)
tests/test_extract.py offline extraction test (plain python3, no deps)
tests/test_catalogue.py offline classifier test
```
@@ -209,6 +213,19 @@ never re-fetch them again):
the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
working agreement for the sorting pipeline.
**Sorter matching rules (fixed 2026-10-05 after the first full run):**
the outer file name wins when it scores >= threshold; inner archive
names only vote as fallback (hash-named gofile archives need that). A
fuzzy match never jumps into another game's section: keyword-identified
systems constrain the catalog pool, and a file with neither system nor
faction keyword only adopts a catalog unit on a strong hit
(`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`.
Token-containment boosts exist for short names vs long catalog names
("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token
stoplist. The catalog deliberately covers all GW systems (needed for
catalog-agnostic use) - wrong-system placement is prevented by matching
discipline, not by shrinking the catalog.
**Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
working, verified 2026-10-04 — how it was obtained, do not re-derive):**
scraping warhammer.com directly is blocked — the shop HTML and all