Add subscription mode, global cross-run URL store and --assume-done import
- dlphelper/globalstore.py: ~/.local/share/dlphelper/urls.json records every successfully downloaded URL once, regardless of OUT_DIR - reposts in other channels and re-runs of moved channels no longer re-download - --subs subscriptions.json: batch mode over a configurable channel list; re-exports each channel by default (new links get picked up), --no-fresh reuses cached exports - --assume-done <file>: manual import step that marks old URL lists as done in the global store (deliberate, unverified, never automatic) - lazy downloader startup: no gofile guest account / no curl runs when all URLs are already done; token prompt skipped in --urls-file mode - write_failed_urls creates OUT_DIR if needed
This commit is contained in:
@@ -47,8 +47,10 @@ dlphelper/
|
||||
gofile.py GoFile API client (one guest account per run)
|
||||
zero807.py curl-based downloader for 0807.st direct links
|
||||
state.py state.json store + manifest.jsonl appender
|
||||
globalstore.py cross-run URL store (~/.local/share/dlphelper/urls.json)
|
||||
parts/grab.sh legacy groundstone script (kept, not used by dlp.py)
|
||||
tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
|
||||
tests/test_extract.py offline extraction test (plain python3, no deps)
|
||||
```
|
||||
|
||||
Data flow:
|
||||
@@ -64,6 +66,7 @@ graph LR
|
||||
E --> H[downloads/gofile/...]
|
||||
F --> I[downloads/0807/...]
|
||||
E & F --> J[state.json + manifest.jsonl + failed_urls.txt]
|
||||
J --> K[globalstore: urls.json, marks URL done forever]
|
||||
```
|
||||
|
||||
Key facts about the hosters (learned the hard way, do not re-learn):
|
||||
@@ -89,6 +92,20 @@ Key facts about the hosters (learned the hard way, do not re-learn):
|
||||
Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.txt`
|
||||
(or just re-run — state.json skips everything already marked ok).
|
||||
|
||||
Several channels at once (`subscriptions.json`: list of {name, channel_id,
|
||||
out_dir}):
|
||||
|
||||
python3 dlp.py --subs subscriptions.json
|
||||
|
||||
Subs mode re-exports every channel by default (that is how new links get
|
||||
picked up) and relies on state.json + the global store to only download
|
||||
what is actually new. Use a cron/systemd timer for watch-like behavior.
|
||||
|
||||
Manual import of the pre-dlp.py era (URLs you know are fully downloaded,
|
||||
never re-fetch them again):
|
||||
|
||||
python3 dlp.py --assume-done old_urls.txt
|
||||
|
||||
## 5. Run artifacts (all in OUT_DIR, gitignored)
|
||||
|
||||
| File | Purpose |
|
||||
@@ -98,6 +115,7 @@ Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.tx
|
||||
| `failed_urls.txt` | rewritten each run: every non-ok URL, usable as `--urls-file` |
|
||||
| `manifest.jsonl` | append-only, one JSON line per downloaded file (channel, url, message_id, timestamp, author, path, size, optional sha256) |
|
||||
| `downloads/` | actual content; `.part` files mean "interrupted, will be resumed" |
|
||||
| `~/.local/share/dlphelper/urls.json` | **global** cross-run store: every successfully downloaded URL is recorded once, regardless of OUT_DIR, so reposts in other channels are not re-downloaded. `--global-state` moves it, `--no-global` disables it for a run. |
|
||||
|
||||
## 6. Rules
|
||||
|
||||
@@ -122,51 +140,60 @@ Retry only failures: `python3 dlp.py <id> <out> --urls-file <out>/failed_urls.tx
|
||||
Password-required gofile links are not retried (it can't help).
|
||||
|
||||
### Idempotency
|
||||
8. Re-runs must be safe and cheap: skip URLs marked `ok`, skip existing
|
||||
non-empty files, write partial downloads to `*.part` and rename on success.
|
||||
8. Re-runs must be safe and cheap: skip URLs marked `ok` (in `state.json` **or**
|
||||
the global store), skip existing non-empty files, write partial downloads
|
||||
to `*.part` and rename on success.
|
||||
9. `state.json` is the source of truth for resume; keep its schema backward
|
||||
compatible when touching `state.py` (unknown keys are preserved, missing
|
||||
keys defaulted).
|
||||
keys defaulted). Same for the global store schema in `globalstore.py`.
|
||||
10. `--assume-done` is a deliberate, manual import step for old URL lists.
|
||||
Never invoke it automatically, and never let the tool mark URLs done
|
||||
without a verified successful download.
|
||||
|
||||
### Rate limiting / network etiquette
|
||||
10. One gofile guest account per run; `ensure_token()` backs off linearly
|
||||
11. One gofile guest account per run; `ensure_token()` backs off linearly
|
||||
(20s × attempt). Do not raise the guest-account frequency "to make tests
|
||||
pass". Testing account creation more than a couple times per hour is a
|
||||
bug, not a test strategy.
|
||||
11. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
|
||||
12. Keep the 0807 referer/UA contract. Don't "optimize" it to aria2c without
|
||||
re-verifying the 403 behavior.
|
||||
|
||||
### Code style
|
||||
12. Python 3.12+, type hints on public functions, `logging` not `print` in
|
||||
13. Python 3.12+, type hints on public functions, `logging` not `print` in
|
||||
library code (prints are fine only for `--dry-run` output).
|
||||
13. Stay on stdlib + `requests`. New dependencies need a real justification
|
||||
14. Stay on stdlib + `requests`. New dependencies need a real justification
|
||||
and belong in `requirements.txt`.
|
||||
14. `dlphelper/` modules must stay import-safe (no side effects at import).
|
||||
15. `dlphelper/` modules must stay import-safe (no side effects at import).
|
||||
|
||||
### Validation
|
||||
15. Minimum validation for any change: `python3 -m py_compile dlp.py
|
||||
16. Minimum validation for any change: `python3 -m py_compile dlp.py
|
||||
dlphelper/*.py`, plus the synthetic extraction test (below) if
|
||||
`discord.py` changed. Document what you could *not* test (e.g. live
|
||||
gofile/0807 downloads need real links and tokens).
|
||||
16. Full live runs need a Discord token from the user — ask for it, never
|
||||
17. Full live runs need a Discord token from the user — ask for it, never
|
||||
ask for it to be pasted into a file or committed.
|
||||
|
||||
### Git
|
||||
17. Branch `main`, descriptive commit messages, no force-push, no history
|
||||
18. Branch `main`, descriptive commit messages, no force-push, no history
|
||||
rewriting on the shared remote.
|
||||
18. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
|
||||
19. When creating a new repo use `tools/new_repo.py` (Gitea API); creation is
|
||||
fine, deletion/renaming of existing repos requires the user.
|
||||
|
||||
## 7. Roadmap (future stages, keep compatible)
|
||||
|
||||
1. **Dedupe**: hash-based (sha256) duplicate detection across threads via
|
||||
`manifest.jsonl`.
|
||||
`manifest.jsonl` (the `--hash` flag feeds this). URL-level dedupe already
|
||||
exists via the global store; content-level (same file, different link)
|
||||
comes with hashing.
|
||||
2. **Torrent stage**: `mktorrent` per downloaded folder, .torrent files into
|
||||
a watch dir, cross-seeding between direct-download and torrent copies.
|
||||
3. **Library organization**: naming conventions per game system / faction so
|
||||
the collection is browsable like a media library.
|
||||
4. **Watch mode**: periodic re-export of subscribed channels, notify on new
|
||||
links.
|
||||
3. **Library organization**: the long-term goal is folders per game system,
|
||||
faction and army-list class (fast attack, heavy support, ...) as described
|
||||
in the wiki. Per-file metadata (source message, timestamp, url) is already
|
||||
captured in `manifest.jsonl` so this can happen later without re-downloads.
|
||||
4. **Watch mode**: `--subs subscriptions.json` + a cron/timer already gives a
|
||||
primitive version; a real watch mode (per-subscription schedules, tear-off
|
||||
notifications, discord pings on new links) builds on it.
|
||||
|
||||
## 8. Testing notes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user