Catalogue: classify collection into system/faction/force-org class

- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and
  Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW)
- dlphelper/catalogue.py: manifest+directory scanner, conservative keyword
  classifier (plural-tolerant word boundaries, longest match wins), markdown
  and jsonl reports, overrides support, audited --apply moves
- catalogue.py CLI; tests/test_catalogue.py offline classifier test
- real-data check on the user's dirs: 1624/3333 files classified so far
This commit is contained in:
2026-10-04 20:25:21 +02:00
parent 64abd7126f
commit acb3415878
6 changed files with 848 additions and 7 deletions
+23
View File
@@ -52,6 +52,29 @@ then run (e.g. from a daily cron/systemd timer):
Subs mode re-exports each channel to pick up new links; `state.json` and the
global store make sure only genuinely new URLs are downloaded.
## Cataloguing the collection
`catalogue.py` classifies every downloaded file into game system, faction and
force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
taxonomy in `dlphelper/warhammer_data.py`:
# preview a dlp.py run dir (manifest + downloads)
python3 catalogue.py /path/to/run_dir
# include the manual-era directories
python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
# catalogue arbitrary dirs only
python3 catalogue.py --scan /big/storage/manual
# actually reorganize into a library tree (audited, only classified files)
python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
Outputs `catalogue.md` (readable) and `catalogue.jsonl` (machine, with source
attribution per file). Unclassified files are listed, never guessed; fix them
with an overrides JSON (`--overrides overrides.json`).
## Already downloaded the old way?
`--assume-done` imports a plain URL list into the global store so those links