Add subscription mode, global cross-run URL store and --assume-done import

- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every
  successfully downloaded URL once, regardless of OUT_DIR - reposts in other
  channels and re-runs of moved channels no longer re-download
- --subs subscriptions.json: batch mode over a configurable channel list;
  re-exports each channel by default (new links get picked up), --no-fresh
  reuses cached exports
- --assume-done <file>: manual import step that marks old URL lists as done
  in the global store (deliberate, unverified, never automatic)
- lazy downloader startup: no gofile guest account / no curl runs when all
  URLs are already done; token prompt skipped in --urls-file mode
- write_failed_urls creates OUT_DIR if needed
This commit is contained in:
2026-10-04 17:19:11 +02:00
parent db8a1c1b52
commit 72ef7bc465
4 changed files with 357 additions and 105 deletions
+30
View File
@@ -31,6 +31,34 @@ hidden). Useful options:
--dry-run just show what would be downloaded
--fresh force a fresh channel export
## Several channels / watching for updates
Put your channels into `subscriptions.json`:
```json
{
"subscriptions": [
{"name": "some-eldar",
"channel_id": "1455993943471820842",
"out_dir": "/big/storage/warhammer/some-eldar"}
]
}
```
then run (e.g. from a daily cron/systemd timer):
python3 dlp.py --subs subscriptions.json
Subs mode re-exports each channel to pick up new links; `state.json` and the
global store make sure only genuinely new URLs are downloaded.
## Already downloaded the old way?
`--assume-done` imports a plain URL list into the global store so those links
are never fetched again (no verification — use deliberately):
python3 dlp.py --assume-done old_urls.txt
## Output layout
OUT_DIR/
@@ -41,6 +69,8 @@ hidden). Useful options:
downloads/
gofile/<content>/<folders>/<files>
0807/<id>.zip
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
Runs are resumable and idempotent: interrupted downloads use `.part` files and
finished ones are skipped on the next run. Individual failures never stop the