Commit Graph
4 Commits
Author SHA1 Message Date
Ben_Kosytorz da5f1f7315 sorter: outer file name wins over inner archive names in fuzzy match
Best-pool scoring used to always blend the archive's inner file names
into the fuzzy score (0.95 discount), which dragged clean-named files
into wrong units (e.g. 'Keeper of Secrets & Shalaxi Helbane.7z' ended
in Orks/Boyz). Now the outer name decides when it scores >= threshold;
inner names only vote as fallback (hash-named gofile archives rely on
that). Also: url/failed-list run artifacts are excluded from sorting,
and two plan-inspection helpers under tools/.
2026-10-05 07:34:29 +02:00
Ben_Kosytorz 5fd5dacaf0 Add GW webshop catalog source (Algolia): dlphelper.gwshop + full dump
- warhammer.com shop HTML/listing endpoints are behind AWS WAF (405/202
  goku challenge) for non-browser clients; findings recorded in AGENTS.md
- the shop search runs on Algolia; public search-only key lets us dump the
  whole product index with plain requests (no browser, no captcha)
- gwshop.py: credential loading (env or gitignored algolia.json), index
  listing, facet-partitioned full retrieval (search-only key cannot /browse
  and pagination caps at 1000, so the index is partitioned on the
  GameSystemsRoot hierarchy facet), normalization into the sorter's
  source-agnostic catalog schema (system/group/faction/subfaction/class)
- data/gwshop_catalog.json: snapshot of prod-lazarus-product-en-ww,
  3491 unique products, every product with full hierarchy path
- tools/: one-shot browse + catalog builder + shape inspector
2026-10-04 21:48:58 +02:00
Ben_Kosytorz 64abd7126f GoFileEmpty: no retries for content that exists but is empty; add tools/debug_gofile.py for raw content-API inspection 2026-10-04 19:16:08 +02:00
Ben_Kosytorz db8a1c1b52 Initial version: dlp.py orchestrates Discord export, link extraction and host-aware downloads
- dlphelper/discord.py: DCE docker export (JSON) + base64/plain URL extraction
  (extraction regex kept compatible with parts/grab.sh)
- dlphelper/gofile.py: GoFile client adapted from Lysagxra/GoFileDownloader's
  API scheme, but ONE guest account per run and per-file failure tracking
  instead of sys.exit - fixes the rate-limit death after ~6 downloads
- dlphelper/zero807.py: curl downloader with the UA+Referer contract 0807.st
  requires; .part files for resumable downloads
- dlphelper/state.py: state.json (resume/skip) + manifest.jsonl (torrent stage
  later) + failed_urls.txt
- tools/new_repo.py: Gitea repo helper reading ~/.git-credentials itself
- AGENTS.md with agent rules, README, offline extraction test
2026-10-04 14:06:46 +02:00