Ben_Kosytorz
6c6a3273bb
The Rust rewrite has replaced the original python implementation, so the transition layout is no longer needed:
- move version-rust/{Cargo.toml,Cargo.lock,README.md,.gitignore,src,data} to the repo root; the crate is now the repo.
- move docs/SORTING.md and data/gwshop_catalog.json to the root.
- remove version-python/ entirely, plus the dev-only version-rust/tools/dump_warhammer_data.py (its only job was regenerating data/warhammer_data.json from python).
- rewrite AGENTS.md and README.md for the single-project layout; strip all version-python/version-rust references from code comments.
- fix the gwshop credential lookup: algolia.json now sits at CARGO_MANIFEST_DIR (crate root == repo root), not its parent.
- restore the credentials/run-output ignores in .gitignore (algolia.json, .env, *.token, downloads/, state.json, manifest.jsonl, ...).
- docs/SORTING.md: retarget the extraction/sorter/dlp examples at the dlphelper binary instead of the deleted python scripts.
cargo test: 86/86 green.
139 lines
7.3 KiB
Markdown
139 lines
7.3 KiB
Markdown
# Warhammer DLP Helper
|
|
|
|
Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts):
|
|
export a Discord channel, extract the (often base64-encoded) download links,
|
|
download per host, extract the archives, sort the files into the library, and
|
|
record a manifest that will feed the torrent stage. One concise Rust binary +
|
|
webapp (the webapp is the next stage).
|
|
|
|
Ported from the original Python implementation, which this version replaced.
|
|
The `Rust port of …` provenance notes in the module headers name the Python
|
|
sources it was derived from; `AGENTS.md` holds the project rules and
|
|
`docs/SORTING.md` the library/sorting contract.
|
|
|
|
## Crate choices (as ported)
|
|
|
|
| Concern | Original Python | Rust |
|
|
|---|---|---|
|
|
| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
|
|
| zip extraction | 7z CLI | `zip` crate |
|
|
| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
|
|
| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
|
|
| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
|
|
| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
|
|
| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
|
|
| CLI | argparse | `clap` (subcommands below) |
|
|
| Webapp | — | `axum` (CLI parity first, webapp second) |
|
|
| Discord export | DCE docker container (external) | unchanged: shell out to docker |
|
|
|
|
## Port status
|
|
|
|
All original Python entry points are ported and live-validated (see below).
|
|
|
|
| Rust module | Python source | Tests |
|
|
|---|---|---|
|
|
| `src/state.rs` | state.py | ✓ |
|
|
| `src/globalstore.rs` | globalstore.py | ✓ |
|
|
| `src/manifest.rs` | state.py append_manifest | ✓ |
|
|
| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
|
|
| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
|
|
| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
|
|
| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
|
|
| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
|
|
| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
|
|
| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
|
|
| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
|
|
| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
|
|
| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
|
|
| `src/main.rs` | all CLIs | smoke-tested (see below) |
|
|
|
|
## CLI
|
|
|
|
```
|
|
dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
|
|
[--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
|
|
[--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
|
|
[--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
|
|
[--global-state FILE] [--no-global] [--assume-done FILE]
|
|
|
|
dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
|
|
[--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]
|
|
|
|
dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
|
|
[--apply] [--audit FILE] [--provenance FILE] [--json]
|
|
[--drop-system NAME]... [--keep-system NAME]...
|
|
|
|
dlphelper extract <SOURCES>... --out DIR
|
|
|
|
dlphelper gwshop <out> [--index NAME] [--raw FILE]
|
|
```
|
|
|
|
Notes:
|
|
|
|
- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
|
|
only ever passed to the DCE container via env, never argv.
|
|
- `sorter --catalog` defaults to `data/gwshop_catalog.json`, relative to the
|
|
current working directory — pass `--catalog` when running elsewhere.
|
|
- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
|
|
(bookkeeping stays out of the library, which may be a seeded torrent).
|
|
- The sorter implements the rules in `docs/SORTING.md` — that document is the
|
|
working agreement; change it first, then the code.
|
|
- Retry loops, cooldowns and the "one gofile guest account per run" rule are
|
|
ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
|
|
- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
|
|
"interrupted" record on the in-flight URL is not replicated).
|
|
|
|
## Deviations from the python version (known, accepted)
|
|
|
|
- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
|
|
(7z CLI in python); 7z/zip/rar use native Rust.
|
|
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
|
|
suffix set never included it, so python left those files unextracted; this
|
|
is a deliberate Rust-only addition.
|
|
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
|
|
parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
|
|
(a 302 to the CDN, with the real file name in the page) into
|
|
`<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
|
|
HTML page itself; the folder `.zip` export is rejected by the site as too
|
|
large, so per-file download is the only way in.
|
|
- `fs::copy` fallback in the sorter does not preserve mtime
|
|
(python `shutil.copy2` does).
|
|
- state.json omits `"files": []` for records without files
|
|
(python writes the empty list); readers of both versions accept both.
|
|
- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
|
|
python printed them in dataclass order. Content is identical.
|
|
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
|
them with odd semantics).
|
|
|
|
## Live validation (2026-10-09/11, channel 1455993943471820842)
|
|
|
|
Initial run with `--since 5d`: DCE export via docker (token env-only), URL
|
|
extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
|
|
file-level skip-existing, retry on network errors), 0807 via curl
|
|
(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
|
|
resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
|
|
extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
|
|
82 GiB.
|
|
|
|
Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
|
|
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
|
|
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
|
|
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
|
|
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
|
|
|
|
Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
|
|
expanded; then `sorter --apply` against the production library
|
|
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
|
|
low-confidence doc beside the library).
|
|
|
|
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
|
|
runtime).
|
|
|
|
## Build/test on NFS
|
|
|
|
The repo lives on NFS; rustc SIGBUSes there. Use:
|
|
|
|
```
|
|
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
|
|
```
|