Ben_Kosytorz
4bbaefb45a
- warhammer_data.py: GW naming throughout - Space Marine chapters are
their own factions under a 'Space Marines' group, systems match the
shop (Age of Sigmar, The Horus Heresy, Other Games, The Old World,
Middle-Earth), classes are GW shop categories (Battleline, Infantry,
Vehicle, Walker, Flyer, Monster, Mounted, Hero, Titanic), group map
extended to AoS grand alliances / Other Games / Old World / Middle-Earth
- gwshop.py: hierarchy remap - GW depth-4 paths are system/group/faction/
class (the shop's last level is the class layer), depth-5 sources keep
a subfaction before it; catalog snapshot regenerated accordingly
- sorter.py: catalog entry is hierarchy-authoritative (group, class, and
faction when GW moved a unit or keyword tables are stale); generic GW
branches ('Unit Type') and a Battleline default never degrade a
confident keyword classification; whole-catalog retry when the faction
pool has no confident match (handles units GW moved between factions);
whole-catalog matching also runs when keyword classification found no
faction; invalid (HTML) items never fuzzy-match; default catalog is
now data/gwshop_catalog.json
- catalogue.py: --apply moves include the group layer, same structure
as the sorter
- docs/SORTING.md: hierarchy rewritten to GW naming with examples
- tests: expectations updated to GW naming (chapters, unit types)
- verified live: 4/4 real files in the test batch now catalog-match,
e.g. The Twin Lance -> 40k/Xenos Armies/T'au Empire/Epic Hero (GW's
current placement) instead of the stale Drukhari keyword
Warhammer DLP Helper
A small "radarr for Warhammer proxy files" tool: point it at a Discord channel/thread and an output directory, and it exports the channel, extracts the encoded download links, and downloads everything - automatically choosing the right downloader per host:
| Host | Downloader | Notes |
|---|---|---|
| gofile.io | built-in GoFile API client | one guest account per run (rate-limit friendly) |
| 0807.st | curl | needs browser-like headers (-A, -e https://0807.st/) |
Install
pip install -r requirements.txt # just `requests`
docker pull tyrrrz/discordchatexporter:stable
Usage
export DISCORD_TOKEN="..."
python3 dlp.py <channel_or_thread_id> /path/to/output
The token can also be passed with --token (or you'll be prompted, input
hidden). Useful options:
--urls-file failed_urls.txt retry a previous failure list, no re-export
--password <pw> for password-protected gofile links
--only gofile|0807 process only one host
--extract unzip archives into per-archive folders
--hash record sha256 per file in the manifest
--dry-run just show what would be downloaded
--fresh force a fresh channel export
Several channels / watching for updates
Put your channels into subscriptions.json:
{
"subscriptions": [
{"name": "some-eldar",
"channel_id": "1455993943471820842",
"out_dir": "/big/storage/warhammer/some-eldar"}
]
}
then run (e.g. from a daily cron/systemd timer):
python3 dlp.py --subs subscriptions.json
Subs mode re-exports each channel to pick up new links; state.json and the
global store make sure only genuinely new URLs are downloaded.
Cataloguing the collection
catalogue.py classifies every downloaded file into game system, faction and
force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
taxonomy in dlphelper/warhammer_data.py:
# preview a dlp.py run dir (manifest + downloads)
python3 catalogue.py /path/to/run_dir
# include the manual-era directories
python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
# catalogue arbitrary dirs only
python3 catalogue.py --scan /big/storage/manual
# actually reorganize into a library tree (audited, only classified files)
python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
Outputs catalogue.md (readable) and catalogue.jsonl (machine, with source
attribution per file). Unclassified files are listed, never guessed; fix them
with an overrides JSON (--overrides overrides.json).
Already downloaded the old way?
--assume-done imports a plain URL list into the global store so those links
are never fetched again (no verification — use deliberately):
python3 dlp.py --assume-done old_urls.txt
Output layout
OUT_DIR/
export.json Discord channel export (cached between runs)
state.json per-URL status; re-runs skip everything marked "ok"
failed_urls.txt URLs that failed after all retries
manifest.jsonl one line per downloaded file (url, source message, path, size)
downloads/
gofile/<content>/<folders>/<files>
0807/<id>.zip
| downloads/ | actual content; .part files mean "interrupted, will be resumed" |
| ~/.local/share/dlphelper/urls.json | global cross-run store of finished URLs (never re-downloaded, in any channel) |
Runs are resumable and idempotent: interrupted downloads use .part files and
finished ones are skipped on the next run. Individual failures never stop the
run; after 3 consecutive failures it pauses for a cooldown (rate-limit
protection) and continues.
Why not use Lysagxra/GoFileDownloader directly?
That tool creates a new GoFile guest account per URL and calls
sys.exit(1) on the first failure, which is exactly why runs died after ~6
downloads. This project reimplements its (working) API scheme - headers,
X-Website-Token, download links with the accountToken cookie - but creates
one guest account per run, records failures per file, and keeps going.
parts/grab.sh is the original groundstone script, kept for reference.