Ben_Kosytorz
acb3415878
Catalogue: classify collection into system/faction/force-org class
...
- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and
Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW)
- dlphelper/catalogue.py: manifest+directory scanner, conservative keyword
classifier (plural-tolerant word boundaries, longest match wins), markdown
and jsonl reports, overrides support, audited --apply moves
- catalogue.py CLI; tests/test_catalogue.py offline classifier test
- real-data check on the user's dirs: 1624/3333 files classified so far
2026-10-04 20:25:21 +02:00
Ben_Kosytorz
64abd7126f
GoFileEmpty: no retries for content that exists but is empty; add tools/debug_gofile.py for raw content-API inspection
2026-10-04 19:16:08 +02:00
Ben_Kosytorz
4f4270fd28
Match gofile 'error-notFound' status (API prefixes dead-link status with error-)
2026-10-04 19:03:54 +02:00
Ben_Kosytorz
413672f114
Dead links (gofile notFound/404, 0807 404/410) fail immediately without retry
2026-10-04 18:54:08 +02:00
Ben_Kosytorz
9d47c29495
Fix gofile website-token crash: int(time()) -> int(time.time())
2026-10-04 18:49:50 +02:00
Ben_Kosytorz
edf6d46cd2
Add --limit (per host class, on pending URLs) and --shuffle for test runs
2026-10-04 18:45:13 +02:00
Ben_Kosytorz
c8bb882d9d
Fix DCE token env var: v2.48 reads DISCORD_TOKEN, not TOKEN (both now set)
2026-10-04 18:36:47 +02:00
Ben_Kosytorz
72ef7bc465
Add subscription mode, global cross-run URL store and --assume-done import
...
- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every
successfully downloaded URL once, regardless of OUT_DIR - reposts in other
channels and re-runs of moved channels no longer re-download
- --subs subscriptions.json: batch mode over a configurable channel list;
re-exports each channel by default (new links get picked up), --no-fresh
reuses cached exports
- --assume-done <file>: manual import step that marks old URL lists as done
in the global store (deliberate, unverified, never automatic)
- lazy downloader startup: no gofile guest account / no curl runs when all
URLs are already done; token prompt skipped in --urls-file mode
- write_failed_urls creates OUT_DIR if needed
2026-10-04 17:19:11 +02:00
Ben_Kosytorz
db8a1c1b52
Initial version: dlp.py orchestrates Discord export, link extraction and host-aware downloads
...
- dlphelper/discord.py: DCE docker export (JSON) + base64/plain URL extraction
(extraction regex kept compatible with parts/grab.sh)
- dlphelper/gofile.py: GoFile client adapted from Lysagxra/GoFileDownloader's
API scheme, but ONE guest account per run and per-file failure tracking
instead of sys.exit - fixes the rate-limit death after ~6 downloads
- dlphelper/zero807.py: curl downloader with the UA+Referer contract 0807.st
requires; .part files for resumable downloads
- dlphelper/state.py: state.json (resume/skip) + manifest.jsonl (torrent stage
later) + failed_urls.txt
- tools/new_repo.py: Gitea repo helper reading ~/.git-credentials itself
- AGENTS.md with agent rules, README, offline extraction test
2026-10-04 14:06:46 +02:00
Ben_Kosytorz