Ben_Kosytorz
acb3415878
- dlphelper/warhammer_data.py: keyword taxonomy (40k factions incl. Chaos and Xenos super-faction units, plus AoS/Heresy/Kill Team/Necromunda/Epic/OW) - dlphelper/catalogue.py: manifest+directory scanner, conservative keyword classifier (plural-tolerant word boundaries, longest match wins), markdown and jsonl reports, overrides support, audited --apply moves - catalogue.py CLI; tests/test_catalogue.py offline classifier test - real-data check on the user's dirs: 1624/3333 files classified so far
112 lines
4.3 KiB
Markdown
112 lines
4.3 KiB
Markdown
# Warhammer DLP Helper
|
|
|
|
A small "radarr for Warhammer proxy files" tool: point it at a Discord
|
|
channel/thread and an output directory, and it exports the channel, extracts
|
|
the encoded download links, and downloads everything - automatically choosing
|
|
the right downloader per host:
|
|
|
|
| Host | Downloader | Notes |
|
|
|-------------|------------|-------|
|
|
| gofile.io | built-in GoFile API client | one guest account per run (rate-limit friendly) |
|
|
| 0807.st | curl | needs browser-like headers (`-A`, `-e https://0807.st/`) |
|
|
|
|
## Install
|
|
|
|
pip install -r requirements.txt # just `requests`
|
|
docker pull tyrrrz/discordchatexporter:stable
|
|
|
|
## Usage
|
|
|
|
export DISCORD_TOKEN="..."
|
|
python3 dlp.py <channel_or_thread_id> /path/to/output
|
|
|
|
The token can also be passed with `--token` (or you'll be prompted, input
|
|
hidden). Useful options:
|
|
|
|
--urls-file failed_urls.txt retry a previous failure list, no re-export
|
|
--password <pw> for password-protected gofile links
|
|
--only gofile|0807 process only one host
|
|
--extract unzip archives into per-archive folders
|
|
--hash record sha256 per file in the manifest
|
|
--dry-run just show what would be downloaded
|
|
--fresh force a fresh channel export
|
|
|
|
## Several channels / watching for updates
|
|
|
|
Put your channels into `subscriptions.json`:
|
|
|
|
```json
|
|
{
|
|
"subscriptions": [
|
|
{"name": "some-eldar",
|
|
"channel_id": "1455993943471820842",
|
|
"out_dir": "/big/storage/warhammer/some-eldar"}
|
|
]
|
|
}
|
|
```
|
|
|
|
then run (e.g. from a daily cron/systemd timer):
|
|
|
|
python3 dlp.py --subs subscriptions.json
|
|
|
|
Subs mode re-exports each channel to pick up new links; `state.json` and the
|
|
global store make sure only genuinely new URLs are downloaded.
|
|
|
|
## Cataloguing the collection
|
|
|
|
`catalogue.py` classifies every downloaded file into game system, faction and
|
|
force-organization class (HQ / Troops / Elites / Fast Attack / Heavy Support /
|
|
Flyer / Dedicated Transport / Lords of War / Fortification) using a keyword
|
|
taxonomy in `dlphelper/warhammer_data.py`:
|
|
|
|
# preview a dlp.py run dir (manifest + downloads)
|
|
python3 catalogue.py /path/to/run_dir
|
|
|
|
# include the manual-era directories
|
|
python3 catalogue.py /path/to/run_dir --scan /big/storage/manual --out-md catalogue.md
|
|
|
|
# catalogue arbitrary dirs only
|
|
python3 catalogue.py --scan /big/storage/manual
|
|
|
|
# actually reorganize into a library tree (audited, only classified files)
|
|
python3 catalogue.py <run_dir> --apply --library /big/storage/warhammer_library
|
|
|
|
Outputs `catalogue.md` (readable) and `catalogue.jsonl` (machine, with source
|
|
attribution per file). Unclassified files are listed, never guessed; fix them
|
|
with an overrides JSON (`--overrides overrides.json`).
|
|
|
|
## Already downloaded the old way?
|
|
|
|
`--assume-done` imports a plain URL list into the global store so those links
|
|
are never fetched again (no verification — use deliberately):
|
|
|
|
python3 dlp.py --assume-done old_urls.txt
|
|
|
|
## Output layout
|
|
|
|
OUT_DIR/
|
|
export.json Discord channel export (cached between runs)
|
|
state.json per-URL status; re-runs skip everything marked "ok"
|
|
failed_urls.txt URLs that failed after all retries
|
|
manifest.jsonl one line per downloaded file (url, source message, path, size)
|
|
downloads/
|
|
gofile/<content>/<folders>/<files>
|
|
0807/<id>.zip
|
|
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
|
|
| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
|
|
|
|
Runs are resumable and idempotent: interrupted downloads use `.part` files and
|
|
finished ones are skipped on the next run. Individual failures never stop the
|
|
run; after 3 consecutive failures it pauses for a cooldown (rate-limit
|
|
protection) and continues.
|
|
|
|
## Why not use Lysagxra/GoFileDownloader directly?
|
|
|
|
That tool creates a new GoFile guest account **per URL** and calls
|
|
`sys.exit(1)` on the first failure, which is exactly why runs died after ~6
|
|
downloads. This project reimplements its (working) API scheme - headers,
|
|
`X-Website-Token`, download links with the `accountToken` cookie - but creates
|
|
one guest account per run, records failures per file, and keeps going.
|
|
|
|
`parts/grab.sh` is the original groundstone script, kept for reference.
|