Commit Graph
6 Commits
Author SHA1 Message Date
Ben_Kosytorz e13412a5fe sorter: weak tier and tokenless adoption need a shared rare token; pool scan takes best accepted entry 2026-10-07 06:08:40 +02:00
Ben_Kosytorz a995937c4b sorter: cross-system name adoption needs token containment; sys-retry keeps the higher score 2026-10-06 23:44:00 +02:00
Ben_Kosytorz 7ccaf75a41 sorter: deepest-matching-dir re-homing + weak 0.50 confidence tier
Inner directory names now compete with the pack/file name: a subtree
under a dir whose name matches the catalog better than the pack-level
resolution is re-homed there (fixes multi-unit sculptor packs like
'Retro Elvez' dumping everything into one keyword class). Junk dir
names (files/images/HEADS/2/...) never match. Hits between 0.50 and the
threshold are accepted as low-confidence: [w] in the plan, confidence
in the audit, listed in sorter_low_confidence.txt. Works in both
provenance and manual-tree mode.
2026-10-06 23:07:26 +02:00
Ben_Kosytorz 4bbaefb45a Switch library hierarchy to GW webshop taxonomy
- warhammer_data.py: GW naming throughout - Space Marine chapters are
  their own factions under a 'Space Marines' group, systems match the
  shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
  Middle-Earth), classes are GW shop categories (Battleline, Infantry,
  Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
  extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
  class (the shop's last level is the class layer), depth-5 sources keep
  a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
  faction when GW moved a unit or keyword tables are stale); generic GW
  branches ('Unit Type') and a Battleline default never degrade a
  confident keyword classification; whole-catalog retry when the faction
  pool has no confident match (handles units GW moved between factions);
  whole-catalog matching also runs when keyword classification found no
  faction; invalid (HTML) items never fuzzy-match; default catalog is
  now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
  as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
  e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
  current placement) instead of the stale Drukhari keyword
2026-10-04 22:05:31 +02:00
Ben_Kosytorz acb3415878 Catalogue: classify collection into system/faction/force-org class
- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and
  Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW)
- dlphelper/catalogue.py: manifest+directory scanner, conservative keyword
  classifier (plural-tolerant word boundaries, longest match wins), markdown
  and jsonl reports, overrides support, audited --apply moves
- catalogue.py CLI; tests/test_catalogue.py offline classifier test
- real-data check on the user's dirs: 1624/3333 files classified so far
2026-10-04 20:25:21 +02:00
Ben_Kosytorz db8a1c1b52 Initial version: dlp.py orchestrates Discord export, link extraction and host-aware downloads
- dlphelper/discord.py: DCE docker export (JSON) + base64/plain URL extraction
  (extraction regex kept compatible with parts/grab.sh)
- dlphelper/gofile.py: GoFile client adapted from Lysagxra/GoFileDownloader's
  API scheme, but ONE guest account per run and per-file failure tracking
  instead of sys.exit - fixes the rate-limit death after ~6 downloads
- dlphelper/zero807.py: curl downloader with the UA+Referer contract 0807.st
  requires; .part files for resumable downloads
- dlphelper/state.py: state.json (resume/skip) + manifest.jsonl (torrent stage
  later) + failed_urls.txt
- tools/new_repo.py: Gitea repo helper reading ~/.git-credentials itself
- AGENTS.md with agent rules, README, offline extraction test
2026-10-04 14:06:46 +02:00