Catalogue: classify collection into system/faction/force-org class

- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and
  Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW)
- dlphelper/catalogue.py: manifest+directory scanner, conservative keyword
  classifier (plural-tolerant word boundaries, longest match wins), markdown
  and jsonl reports, overrides support, audited --apply moves
- catalogue.py CLI; tests/test_catalogue.py offline classifier test
- real-data check on the user's dirs: 1624/3333 files classified so far
This commit is contained in:
2026-10-04 20:25:21 +02:00
parent 64abd7126f
commit acb3415878
6 changed files with 848 additions and 7 deletions
+16 -7
View File
@@ -42,15 +42,20 @@ for dedupe/cross-seeding later.
```
dlp.py CLI orchestrator (entry point)
catalogue.py CLI: classify the collection into system/faction/class
dlphelper/
discord.py DCE docker export (JSON) + base64/plain URL extraction
gofile.py GoFile API client (one guest account per run)
zero807.py curl-based downloader for 0807.st direct links
state.py state.json store + manifest.jsonl appender
globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json)
catalogue.py scanner + keyword classifier + report writer
warhammer_data.py taxonomy: systems, factions, force-org classes
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py inspect raw gofile API response for one content id
tests/test_extract.py offline extraction test (plain python3, no deps)
tests/test_catalogue.py offline classifier test
```
Data flow:
@@ -166,9 +171,10 @@ never re-fetch them again):
15. `dlphelper/` modules must stay import-safe (no side effects at import).
### Validation
16. Minimum validation for any change: `python3 -m py_compile dlp.py
dlphelper/*.py`, plus the synthetic extraction test (below) if
`discord.py` changed. Document what you could *not* test (e.g. live
15. Minimum validation for any change: `python3 -m py_compile dlp.py
catalogue.py dlphelper/*.py`, plus the offline tests (`python3
tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
modules changed. Document what you could *not* test (e.g. live
gofile/0807 downloads need real links and tokens).
17. Full live runs need a Discord token from the user — ask for it, never
ask for it to be pasted into a file or committed.
@@ -187,10 +193,13 @@ never re-fetch them again):
comes with hashing.
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
a watch dir, cross-seeding between direct-download and torrent copies.
3. **Library organization**: the long-term goal is folders per game system,
faction and army-list class (fast attack, heavy support, ...) as described
in the wiki. Per-file metadata (source message, timestamp, url) is already
captured in `manifest.jsonl` so this can happen later without re-downloads.
3. **Library organization (started)**: `catalogue.py` classifies every file
into system / faction / force-org class and can move files into
`<library>/<System>/<Faction>/<Class>/` with `--apply` (audited in
`catalogue_moves.jsonl`). The taxonomy lives in
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
collection - unclassified files are reported, never guessed. Manual fixes
go into an overrides JSON (`--overrides`).
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
primitive version; a real watch mode (per-subscription schedules, tear-off
notifications, discord pings on new links) builds on it.