Files

T

Ben_Kosytorz 72ef7bc465 Add subscription mode, global cross-run URL store and --assume-done import

- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every
  successfully downloaded URL once, regardless of OUT_DIR - reposts in other
  channels and re-runs of moved channels no longer re-download
- --subs subscriptions.json: batch mode over a configurable channel list;
  re-exports each channel by default (new links get picked up), --no-fresh
  reuses cached exports
- --assume-done <file>: manual import step that marks old URL lists as done
  in the global store (deliberate, unverified, never automatic)
- lazy downloader startup: no gofile guest account / no curl runs when all
  URLs are already done; token prompt skipped in --urls-file mode
- write_failed_urls creates OUT_DIR if needed

2026-10-04 17:19:11 +02:00

3.3 KiB

Raw Blame History

Warhammer DLP Helper

A small "radarr for Warhammer proxy files" tool: point it at a Discord channel/thread and an output directory, and it exports the channel, extracts the encoded download links, and downloads everything - automatically choosing the right downloader per host:

Host	Downloader	Notes
gofile.io	built-in GoFile API client	one guest account per run (rate-limit friendly)
0807.st	curl	needs browser-like headers (`-A`, `-e https://0807.st/`)

Install

pip install -r requirements.txt   # just `requests`
docker pull tyrrrz/discordchatexporter:stable

Usage

export DISCORD_TOKEN="..."
python3 dlp.py <channel_or_thread_id> /path/to/output

The token can also be passed with --token (or you'll be prompted, input hidden). Useful options:

--urls-file failed_urls.txt   retry a previous failure list, no re-export
--password <pw>               for password-protected gofile links
--only gofile|0807            process only one host
--extract                     unzip archives into per-archive folders
--hash                        record sha256 per file in the manifest
--dry-run                     just show what would be downloaded
--fresh                       force a fresh channel export

Several channels / watching for updates

Put your channels into subscriptions.json:

{
  "subscriptions": [
    {"name": "some-eldar",
     "channel_id": "1455993943471820842",
     "out_dir": "/big/storage/warhammer/some-eldar"}
  ]
}

then run (e.g. from a daily cron/systemd timer):

python3 dlp.py --subs subscriptions.json

Subs mode re-exports each channel to pick up new links; state.json and the global store make sure only genuinely new URLs are downloaded.

Already downloaded the old way?

--assume-done imports a plain URL list into the global store so those links are never fetched again (no verification — use deliberately):

python3 dlp.py --assume-done old_urls.txt

Output layout

OUT_DIR/
  export.json           Discord channel export (cached between runs)
  state.json            per-URL status; re-runs skip everything marked "ok"
  failed_urls.txt       URLs that failed after all retries
  manifest.jsonl        one line per downloaded file (url, source message, path, size)
  downloads/
    gofile/<content>/<folders>/<files>
    0807/<id>.zip

| downloads/ | actual content; .part files mean "interrupted, will be resumed" | | ~/.local/share/dlphelper/urls.json | global cross-run store of finished URLs (never re-downloaded, in any channel) |

Runs are resumable and idempotent: interrupted downloads use .part files and finished ones are skipped on the next run. Individual failures never stop the run; after 3 consecutive failures it pauses for a cooldown (rate-limit protection) and continues.

Why not use Lysagxra/GoFileDownloader directly?

That tool creates a new GoFile guest account per URL and calls sys.exit(1) on the first failure, which is exactly why runs died after ~6 downloads. This project reimplements its (working) API scheme - headers, X-Website-Token, download links with the accountToken cookie - but creates one guest account per run, records failures per file, and keeps going.

parts/grab.sh is the original groundstone script, kept for reference.

3.3 KiB Raw Blame History