Catalogue: classify collection into system/faction/force-org class
- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW) - dlphelper/catalogue.py: manifest+directory scanner, conservative keyword classifier (plural-tolerant word boundaries, longest match wins), markdown and jsonl reports, overrides support, audited --apply moves - catalogue.py CLI; tests/test_catalogue.py offline classifier test - real-data check on the user's dirs: 1624/3333 files classified so far
This commit is contained in:
@@ -42,15 +42,20 @@ for dedupe/cross-seeding later.
|
||||
|
||||
```
|
||||
dlp.py CLI orchestrator (entry point)
|
||||
catalogue.py CLI: classify the collection into system/faction/class
|
||||
dlphelper/
|
||||
discord.py DCE docker export (JSON) + base64/plain URL extraction
|
||||
gofile.py GoFile API client (one guest account per run)
|
||||
zero807.py curl-based downloader for 0807.st direct links
|
||||
state.py state.json store + manifest.jsonl appender
|
||||
globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json)
|
||||
catalogue.py scanner + keyword classifier + report writer
|
||||
warhammer_data.py taxonomy: systems, factions, force-org classes
|
||||
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
|
||||
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
|
||||
tools/debug_gofile.py inspect raw gofile API response for one content id
|
||||
tests/test_extract.py offline extraction test (plain python3, no deps)
|
||||
tests/test_catalogue.py offline classifier test
|
||||
```
|
||||
|
||||
Data flow:
|
||||
@@ -166,9 +171,10 @@ never re-fetch them again):
|
||||
15. `dlphelper/` modules must stay import-safe (no side effects at import).
|
||||
|
||||
### Validation
|
||||
16. Minimum validation for any change: `python3 -m py_compile dlp.py
|
||||
dlphelper/*.py`, plus the synthetic extraction test (below) if
|
||||
`discord.py` changed. Document what you could *not* test (e.g. live
|
||||
15. Minimum validation for any change: `python3 -m py_compile dlp.py
|
||||
catalogue.py dlphelper/*.py`, plus the offline tests (`python3
|
||||
tests/test_extract.py`, `python3 tests/test_catalogue.py`) if the touched
|
||||
modules changed. Document what you could *not* test (e.g. live
|
||||
gofile/0807 downloads need real links and tokens).
|
||||
17. Full live runs need a Discord token from the user — ask for it, never
|
||||
ask for it to be pasted into a file or committed.
|
||||
@@ -187,10 +193,13 @@ never re-fetch them again):
|
||||
comes with hashing.
|
||||
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
|
||||
a watch dir, cross-seeding between direct-download and torrent copies.
|
||||
3. **Library organization**: the long-term goal is folders per game system,
|
||||
faction and army-list class (fast attack, heavy support, ...) as described
|
||||
in the wiki. Per-file metadata (source message, timestamp, url) is already
|
||||
captured in `manifest.jsonl` so this can happen later without re-downloads.
|
||||
3. **Library organization (started)**: `catalogue.py` classifies every file
|
||||
into system / faction / force-org class and can move files into
|
||||
`<library>/<System>/<Faction>/<Class>/` with `--apply` (audited in
|
||||
`catalogue_moves.jsonl`). The taxonomy lives in
|
||||
`dlphelper/warhammer_data.py`; it is keyword-based and grows with the
|
||||
collection - unclassified files are reported, never guessed. Manual fixes
|
||||
go into an overrides JSON (`--overrides`).
|
||||
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
|
||||
primitive version; a real watch mode (per-subscription schedules, tear-off
|
||||
notifications, discord pings on new links) builds on it.
|
||||
|
||||
Reference in New Issue
Block a user