Initial version: dlp.py orchestrates Discord export, link extraction and host-aware downloads
- dlphelper/discord.py: DCE docker export (JSON) + base64/plain URL extraction (extraction regex kept compatible with parts/grab.sh) - dlphelper/gofile.py: GoFile client adapted from Lysagxra/GoFileDownloader's API scheme, but ONE guest account per run and per-file failure tracking instead of sys.exit - fixes the rate-limit death after ~6 downloads - dlphelper/zero807.py: curl downloader with the UA+Referer contract 0807.st requires; .part files for resumable downloads - dlphelper/state.py: state.json (resume/skip) + manifest.jsonl (torrent stage later) + failed_urls.txt - tools/new_repo.py: Gitea repo helper reading ~/.git-credentials itself - AGENTS.md with agent rules, README, offline extraction test
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# Warhammer DLP Helper
|
||||
|
||||
A small "radarr for Warhammer proxy files" tool: point it at a Discord
|
||||
channel/thread and an output directory, and it exports the channel, extracts
|
||||
the encoded download links, and downloads everything - automatically choosing
|
||||
the right downloader per host:
|
||||
|
||||
| Host | Downloader | Notes |
|
||||
|-------------|------------|-------|
|
||||
| gofile.io | built-in GoFile API client | one guest account per run (rate-limit friendly) |
|
||||
| 0807.st | curl | needs browser-like headers (`-A`, `-e https://0807.st/`) |
|
||||
|
||||
## Install
|
||||
|
||||
pip install -r requirements.txt # just `requests`
|
||||
docker pull tyrrrz/discordchatexporter:stable
|
||||
|
||||
## Usage
|
||||
|
||||
export DISCORD_TOKEN="..."
|
||||
python3 dlp.py <channel_or_thread_id> /path/to/output
|
||||
|
||||
The token can also be passed with `--token` (or you'll be prompted, input
|
||||
hidden). Useful options:
|
||||
|
||||
--urls-file failed_urls.txt retry a previous failure list, no re-export
|
||||
--password <pw> for password-protected gofile links
|
||||
--only gofile|0807 process only one host
|
||||
--extract unzip archives into per-archive folders
|
||||
--hash record sha256 per file in the manifest
|
||||
--dry-run just show what would be downloaded
|
||||
--fresh force a fresh channel export
|
||||
|
||||
## Output layout
|
||||
|
||||
OUT_DIR/
|
||||
export.json Discord channel export (cached between runs)
|
||||
state.json per-URL status; re-runs skip everything marked "ok"
|
||||
failed_urls.txt URLs that failed after all retries
|
||||
manifest.jsonl one line per downloaded file (url, source message, path, size)
|
||||
downloads/
|
||||
gofile/<content>/<folders>/<files>
|
||||
0807/<id>.zip
|
||||
|
||||
Runs are resumable and idempotent: interrupted downloads use `.part` files and
|
||||
finished ones are skipped on the next run. Individual failures never stop the
|
||||
run; after 3 consecutive failures it pauses for a cooldown (rate-limit
|
||||
protection) and continues.
|
||||
|
||||
## Why not use Lysagxra/GoFileDownloader directly?
|
||||
|
||||
That tool creates a new GoFile guest account **per URL** and calls
|
||||
`sys.exit(1)` on the first failure, which is exactly why runs died after ~6
|
||||
downloads. This project reimplements its (working) API scheme - headers,
|
||||
`X-Website-Token`, download links with the `accountToken` cookie - but creates
|
||||
one guest account per run, records failures per file, and keeps going.
|
||||
|
||||
`parts/grab.sh` is the original groundstone script, kept for reference.
|
||||
Reference in New Issue
Block a user