Ben_Kosytorz
f631e351f8
Bugs: - extract: pick the OUTERMOST archive suffix (sfx.first(), not last()) so x.tar.gz uses the gz->tar path - gwshop: truncate the error snippet by chars, not bytes (UTF-8 panic) - sorter: py_stem_split now matches pathlib (0 < i < len-1) so a trailing dot is not treated as a suffix - sorter::apply_plan: a per-file link/copy/mkdir failure is logged and skipped instead of aborting the whole apply (never-abort contract) - orchestrate/state: add State::mark_logged and use it for every per-URL record so a state-save error cannot abort a run Cleanups: - remove dead is_archive_name and unused SourceInfo.manifest field - mark kept-but-unused capabilities with allow(dead_code) + rationale - let-chains / is_ok_and / is_multiple_of / replace([..]) / sort_by_key and other idiomatic simplifications across modules - difflib: SequenceMatcher owns its char vecs (bit-identical parity kept) - --only now enforces gofile|0807 via value_parser Docs: - SORTING.md: drop phantom sorter_plan.jsonl, os.link -> std::fs::hard_link, zipfile -> the zip crate - README: add --since to the dlp synopsis, clarify --subs/--assume-done - .gitignore: ignore catalogue/sorter/_logs/subscriptions run artifacts Validation: cargo fmt --check clean, cargo clippy --all-targets 0 warnings, cargo test 87 passed / 0 failed.
147 lines
7.7 KiB
Markdown
147 lines
7.7 KiB
Markdown
# Warhammer DLP Helper
|
|
|
|
Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts):
|
|
export a Discord channel, extract the (often base64-encoded) download links,
|
|
download per host, extract the archives, sort the files into the library, and
|
|
record a manifest that will feed the torrent stage. One concise Rust binary +
|
|
webapp (the webapp is the next stage).
|
|
|
|
Ported from the original Python implementation, which this version replaced.
|
|
The `Rust port of …` provenance notes in the module headers name the Python
|
|
sources it was derived from; `AGENTS.md` holds the project rules and
|
|
`docs/SORTING.md` the library/sorting contract.
|
|
|
|
## Crate choices (as ported)
|
|
|
|
| Concern | Original Python | Rust |
|
|
|---|---|---|
|
|
| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
|
|
| zip extraction | 7z CLI | `zip` crate |
|
|
| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
|
|
| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
|
|
| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
|
|
| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
|
|
| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
|
|
| CLI | argparse | `clap` (subcommands below) |
|
|
| Webapp | — | `axum` (CLI parity first, webapp second) |
|
|
| Discord export | DCE docker container (external) | unchanged: shell out to docker |
|
|
|
|
## Port status
|
|
|
|
All original Python entry points are ported and live-validated (see below).
|
|
|
|
| Rust module | Python source | Tests |
|
|
|---|---|---|
|
|
| `src/state.rs` | state.py | ✓ |
|
|
| `src/globalstore.rs` | globalstore.py | ✓ |
|
|
| `src/manifest.rs` | state.py append_manifest | ✓ |
|
|
| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
|
|
| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
|
|
| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
|
|
| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
|
|
| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
|
|
| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
|
|
| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
|
|
| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
|
|
| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
|
|
| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
|
|
| `src/main.rs` | all CLIs | smoke-tested (see below) |
|
|
|
|
## CLI
|
|
|
|
```
|
|
dlphelper dlp <channel_id> <out_dir> [--since WHEN] [--urls-file FILE] [--token]
|
|
[--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
|
|
[--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
|
|
[--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
|
|
[--global-state FILE] [--no-global]
|
|
|
|
dlphelper dlp --subs FILE # alternative mode: iterate every channel
|
|
# in the list (channel_id/out_dir unused)
|
|
|
|
dlphelper dlp --assume-done FILE # alternative mode: mark URLs as done, exit
|
|
|
|
dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
|
|
[--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]
|
|
|
|
dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
|
|
[--apply] [--audit FILE] [--provenance FILE] [--json]
|
|
[--drop-system NAME]... [--keep-system NAME]...
|
|
|
|
dlphelper extract <SOURCES>... --out DIR
|
|
|
|
dlphelper gwshop <out> [--index NAME] [--raw FILE]
|
|
```
|
|
|
|
Notes:
|
|
|
|
- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
|
|
only ever passed to the DCE container via env, never argv.
|
|
- `sorter --catalog` defaults to `data/gwshop_catalog.json`, relative to the
|
|
current working directory — pass `--catalog` when running elsewhere.
|
|
- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
|
|
(bookkeeping stays out of the library, which may be a seeded torrent).
|
|
- The sorter implements the rules in `docs/SORTING.md` — that document is the
|
|
working agreement; change it first, then the code.
|
|
- `dlp --since` limits a run to recent messages (e.g. `5d`, `12h`,
|
|
`2026-10-04`); `--subs` and `--assume-done` are alternative modes that ignore
|
|
`<channel_id>`/`<out_dir>`.
|
|
- Retry loops, cooldowns and the "one gofile guest account per run" rule are
|
|
ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
|
|
- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
|
|
"interrupted" record on the in-flight URL is not replicated).
|
|
|
|
## Deviations from the python version (known, accepted)
|
|
|
|
- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
|
|
(7z CLI in python); 7z/zip/rar use native Rust.
|
|
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
|
|
suffix set never included it, so python left those files unextracted; this
|
|
is a deliberate Rust-only addition.
|
|
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
|
|
parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
|
|
(a 302 to the CDN, with the real file name in the page) into
|
|
`<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
|
|
HTML page itself; the folder `.zip` export is rejected by the site as too
|
|
large, so per-file download is the only way in.
|
|
- `fs::copy` fallback in the sorter does not preserve mtime
|
|
(python `shutil.copy2` does).
|
|
- state.json omits `"files": []` for records without files
|
|
(python writes the empty list); readers of both versions accept both.
|
|
- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
|
|
python printed them in dataclass order. Content is identical.
|
|
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
|
them with odd semantics).
|
|
|
|
## Live validation (2026-10-09/11, channel 1455993943471820842)
|
|
|
|
Initial run with `--since 5d`: DCE export via docker (token env-only), URL
|
|
extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
|
|
file-level skip-existing, retry on network errors), 0807 via curl
|
|
(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
|
|
resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
|
|
extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
|
|
82 GiB.
|
|
|
|
Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
|
|
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
|
|
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
|
|
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
|
|
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
|
|
|
|
Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
|
|
expanded; then `sorter --apply` against the production library
|
|
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
|
|
low-confidence doc beside the library).
|
|
|
|
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
|
|
runtime).
|
|
|
|
## Build/test on NFS
|
|
|
|
The repo lives on NFS; rustc SIGBUSes there. Use:
|
|
|
|
```
|
|
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
|
|
```
|