Ben_Kosytorz
55d32379ed
- orchestrate.rs: process_channel, run_gofile (re-auth on AuthError, no-retry branches for dead/password/empty, cooldowns), run_0807, subscriptions, --assume-done (with note), failed_urls export - main.rs: dlp / catalogue / sorter / extract / gwshop subcommands, token resolution (--token -> $DISCORD_TOKEN -> hidden prompt) - globalstore: mark() gains a note field (python note= kwarg) - README: port status, CLI, deviations, untested live paths 77 tests green.
124 lines
6.3 KiB
Markdown
124 lines
6.3 KiB
Markdown
# version-rust
|
|
|
|
Rust rewrite of `version-python` (same repo, `main` branch still carries the
|
|
flat python layout). Goal: one concise binary + webapp, eventually replacing
|
|
the python tooling.
|
|
|
|
## Why this structure
|
|
|
|
- `main` stays untouched (flat python layout) until the rewrite reaches parity.
|
|
- This branch re-homes the python code into `version-python/` (via `git mv`,
|
|
history preserved) and starts `version-rust/` beside it.
|
|
- `version-python/AGENTS.md` still documents the python stage — its rules
|
|
(secrets, failure policy, idempotency, rate limiting) apply to the rewrite
|
|
too and are the contract the Rust version must keep.
|
|
|
|
## Crate choices (as ported)
|
|
|
|
| Concern | Python today | Rust |
|
|
|---|---|---|
|
|
| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
|
|
| zip extraction | 7z CLI | `zip` crate |
|
|
| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
|
|
| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
|
|
| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
|
|
| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
|
|
| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
|
|
| CLI | argparse | `clap` (subcommands below) |
|
|
| Webapp | — | `axum` (CLI parity first, webapp second) |
|
|
| Discord export | DCE docker container (external) | unchanged: shell out to docker |
|
|
|
|
## Port status
|
|
|
|
All python entry points are ported; live network paths are tested offline
|
|
only (see "Untested live paths").
|
|
|
|
| Rust module | Python source | Tests |
|
|
|---|---|---|
|
|
| `src/state.rs` | state.py | ✓ |
|
|
| `src/globalstore.rs` | globalstore.py | ✓ |
|
|
| `src/manifest.rs` | state.py append_manifest | ✓ |
|
|
| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
|
|
| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
|
|
| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
|
|
| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
|
|
| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
|
|
| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
|
|
| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
|
|
| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
|
|
| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
|
|
| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
|
|
| `src/main.rs` | all CLIs | smoke-tested (see below) |
|
|
|
|
## CLI
|
|
|
|
```
|
|
dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
|
|
[--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
|
|
[--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
|
|
[--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
|
|
[--global-state FILE] [--no-global] [--assume-done FILE]
|
|
|
|
dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
|
|
[--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]
|
|
|
|
dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
|
|
[--apply] [--audit FILE] [--provenance FILE] [--json]
|
|
[--drop-system NAME]... [--keep-system NAME]...
|
|
|
|
dlphelper extract <SOURCES>... --out DIR
|
|
|
|
dlphelper gwshop <out> [--index NAME] [--raw FILE]
|
|
```
|
|
|
|
Notes:
|
|
|
|
- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
|
|
only ever passed to the DCE container via env, never argv.
|
|
- `sorter --catalog` defaults to `version-python/data/gwshop_catalog.json`
|
|
(falling back to `data/gwshop_catalog.json`), both relative to the current
|
|
working directory — pass `--catalog` when running elsewhere.
|
|
- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
|
|
(bookkeeping stays out of the library, which may be a seeded torrent).
|
|
- Retry loops, cooldowns and the "one gofile guest account per run" rule are
|
|
ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
|
|
- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
|
|
"interrupted" record on the in-flight URL is not replicated).
|
|
|
|
## Deviations from the python version (known, accepted)
|
|
|
|
- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
|
|
(7z CLI in python); 7z/zip/rar use native Rust.
|
|
- `fs::copy` fallback in the sorter does not preserve mtime
|
|
(python `shutil.copy2` does).
|
|
- state.json omits `"files": []` for records without files
|
|
(python writes the empty list); readers of both versions accept both.
|
|
- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
|
|
python printed them in dataclass order. Content is identical.
|
|
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
|
them with odd semantics).
|
|
|
|
## Untested live paths (validated offline only, per the failure-policy rules)
|
|
|
|
- Actual Discord channel export via the DCE container (needs a token) — the
|
|
docker invocation and export.json parsing are unit-tested, the network path
|
|
is not.
|
|
- Live gofile downloads (guest account creation, `ensure_token`) — website
|
|
token derivation is golden-tested against python; the HTTP calls are not.
|
|
- Live 0807 downloads — the `curl -A "Mozilla/5.0" -e "https://0807.st/"`
|
|
contract is kept, timeouts are per python (`--max-time 1800`), but no live
|
|
fetch was made.
|
|
- Real `.rar` extraction — the extractor wiring is in place (`unrar` crate,
|
|
unrar runtime library present), but no real RAR archive was available for a
|
|
live test.
|
|
- `gwshop` was smoke-tested for request shaping only; a full Algolia browse
|
|
needs valid credentials (`algolia.json` or env).
|
|
|
|
## Build/test on NFS
|
|
|
|
The repo lives on NFS; rustc SIGBUSes there. Use:
|
|
|
|
```
|
|
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
|
|
```
|