extract stage + system-locked matching + provenance unit mode
- tools/extract_all.py: recursive extraction of all archives (incl. nested) into a plain-file tree; nested archives are deleted after successful extraction, plain files are hardlinked, failures go to .broken/, extract_manifest.jsonl records archive->dest->unit. - sorter --provenance: classify each extraction directory once under the archive's name (files of one archive are one unit; inner structure kept), so loose STLs sort with their model set. - matching discipline: a fuzzy match never leaves the file's system; system-less files need CROSS_SYSTEM_MIN (0.75) for a catalog unit, and a same-system faction jump below 0.75 is rejected. Fixes 40k files landing in Middle-earth/Blood Bowl/AoS buckets. - speed for big trees: lru_cache on keyword matchers and pair scores, quick_ratio pruning; keywords: night lord, csm.
This commit is contained in:
@@ -56,10 +56,14 @@ dlphelper/
|
||||
categories as classes, GROUP_OF_FACTION map
|
||||
gwshop.py GW webshop catalog dump via Algolia (see §7.3)
|
||||
sorter.py library sorter: inspect/classify/fuzzy-match
|
||||
against the catalog, plan + hardlink apply
|
||||
against the catalog, plan + hardlink apply,
|
||||
optional --provenance unit mode for extracted trees
|
||||
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
|
||||
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
|
||||
tools/debug_gofile.py inspect raw gofile API response for one content id
|
||||
tools/extract_all.py recursive archive extraction into a plain-file tree
|
||||
(extract_manifest.jsonl feeds the sorter's
|
||||
--provenance unit mode; failures go to .broken/)
|
||||
tests/test_extract.py offline extraction test (plain python3, no deps)
|
||||
tests/test_catalogue.py offline classifier test
|
||||
```
|
||||
@@ -209,6 +213,19 @@ never re-fetch them again):
|
||||
the **GW webshop taxonomy** (see below); `docs/SORTING.md` is the
|
||||
working agreement for the sorting pipeline.
|
||||
|
||||
**Sorter matching rules (fixed 2026-10-05 after the first full run):**
|
||||
the outer file name wins when it scores >= threshold; inner archive
|
||||
names only vote as fallback (hash-named gofile archives need that). A
|
||||
fuzzy match never jumps into another game's section: keyword-identified
|
||||
systems constrain the catalog pool, and a file with neither system nor
|
||||
faction keyword only adopts a catalog unit on a strong hit
|
||||
(`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`.
|
||||
Token-containment boosts exist for short names vs long catalog names
|
||||
("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token
|
||||
stoplist. The catalog deliberately covers all GW systems (needed for
|
||||
catalog-agnostic use) - wrong-system placement is prevented by matching
|
||||
discipline, not by shrinking the catalog.
|
||||
|
||||
**Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`,
|
||||
working, verified 2026-10-04 — how it was obtained, do not re-derive):**
|
||||
scraping warhammer.com directly is blocked — the shop HTML and all
|
||||
|
||||
Reference in New Issue
Block a user