Commit Graph
3 Commits
Author SHA1 Message Date
Ben_Kosytorz 4bbaefb45a Switch library hierarchy to GW webshop taxonomy
- warhammer_data.py: GW naming throughout - Space Marine chapters are
  their own factions under a 'Space Marines' group, systems match the
  shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
  Middle-Earth), classes are GW shop categories (Battleline, Infantry,
  Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
  extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
  class (the shop's last level is the class layer), depth-5 sources keep
  a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
  faction when GW moved a unit or keyword tables are stale); generic GW
  branches ('Unit Type') and a Battleline default never degrade a
  confident keyword classification; whole-catalog retry when the faction
  pool has no confident match (handles units GW moved between factions);
  whole-catalog matching also runs when keyword classification found no
  faction; invalid (HTML) items never fuzzy-match; default catalog is
  now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
  as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
  e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
  current placement) instead of the stale Drukhari keyword
2026-10-04 22:05:31 +02:00
Ben_Kosytorz acb3415878 Catalogue: classify collection into system/faction/force-org class
- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and
  Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW)
- dlphelper/catalogue.py: manifest+directory scanner, conservative keyword
  classifier (plural-tolerant word boundaries, longest match wins), markdown
  and jsonl reports, overrides support, audited --apply moves
- catalogue.py CLI; tests/test_catalogue.py offline classifier test
- real-data check on the user's dirs: 1624/3333 files classified so far
2026-10-04 20:25:21 +02:00
Ben_Kosytorz db8a1c1b52 Initial version: dlp.py orchestrates Discord export, link extraction and host-aware downloads
- dlphelper/discord.py: DCE docker export (JSON) + base64/plain URL extraction
  (extraction regex kept compatible with parts/grab.sh)
- dlphelper/gofile.py: GoFile client adapted from Lysagxra/GoFileDownloader's
  API scheme, but ONE guest account per run and per-file failure tracking
  instead of sys.exit - fixes the rate-limit death after ~6 downloads
- dlphelper/zero807.py: curl downloader with the UA+Referer contract 0807.st
  requires; .part files for resumable downloads
- dlphelper/state.py: state.json (resume/skip) + manifest.jsonl (torrent stage
  later) + failed_urls.txt
- tools/new_repo.py: Gitea repo helper reading ~/.git-credentials itself
- AGENTS.md with agent rules, README, offline extraction test
2026-10-04 14:06:46 +02:00