Add subscription mode, global cross-run URL store and --assume-done import
- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every successfully downloaded URL once, regardless of OUT_DIR - reposts in other channels and re-runs of moved channels no longer re-download - --subs subscriptions.json: batch mode over a configurable channel list; re-exports each channel by default (new links get picked up), --no-fresh reuses cached exports - --assume-done <file>: manual import step that marks old URL lists as done in the global store (deliberate, unverified, never automatic) - lazy downloader startup: no gofile guest account / no curl runs when all URLs are already done; token prompt skipped in --urls-file mode - write_failed_urls creates OUT_DIR if needed
This commit is contained in:
@@ -31,6 +31,34 @@ hidden). Useful options:
|
||||
--dry-run just show what would be downloaded
|
||||
--fresh force a fresh channel export
|
||||
|
||||
## Several channels / watching for updates
|
||||
|
||||
Put your channels into `subscriptions.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"subscriptions": [
|
||||
{"name": "some-eldar",
|
||||
"channel_id": "1455993943471820842",
|
||||
"out_dir": "/big/storage/warhammer/some-eldar"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
then run (e.g. from a daily cron/systemd timer):
|
||||
|
||||
python3 dlp.py --subs subscriptions.json
|
||||
|
||||
Subs mode re-exports each channel to pick up new links; `state.json` and the
|
||||
global store make sure only genuinely new URLs are downloaded.
|
||||
|
||||
## Already downloaded the old way?
|
||||
|
||||
`--assume-done` imports a plain URL list into the global store so those links
|
||||
are never fetched again (no verification — use deliberately):
|
||||
|
||||
python3 dlp.py --assume-done old_urls.txt
|
||||
|
||||
## Output layout
|
||||
|
||||
OUT_DIR/
|
||||
@@ -41,6 +69,8 @@ hidden). Useful options:
|
||||
downloads/
|
||||
gofile/<content>/<folders>/<files>
|
||||
0807/<id>.zip
|
||||
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
|
||||
| `~/.local/share/dlphelper/urls.json` | global cross-run store of finished URLs (never re-downloaded, in any channel) |
|
||||
|
||||
Runs are resumable and idempotent: interrupted downloads use `.part` files and
|
||||
finished ones are skipped on the next run. Individual failures never stop the
|
||||
|
||||
Reference in New Issue
Block a user