Ben_Kosytorz/Warhammer_DLP_Helper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Warhammer_DLP_Helper/README.md
T
Ben_Kosytorz f631e351f8 Review pass: fix bugs, silence clippy, correct docs 
Bugs:
- extract: pick the OUTERMOST archive suffix (sfx.first(), not last())
  so x.tar.gz uses the gz->tar path
- gwshop: truncate the error snippet by chars, not bytes (UTF-8 panic)
- sorter: py_stem_split now matches pathlib (0 < i < len-1) so a
  trailing dot is not treated as a suffix
- sorter::apply_plan: a per-file link/copy/mkdir failure is logged and
  skipped instead of aborting the whole apply (never-abort contract)
- orchestrate/state: add State::mark_logged and use it for every
  per-URL record so a state-save error cannot abort a run

Cleanups:
- remove dead is_archive_name and unused SourceInfo.manifest field
- mark kept-but-unused capabilities with allow(dead_code) + rationale
- let-chains / is_ok_and / is_multiple_of / replace([..]) / sort_by_key
  and other idiomatic simplifications across modules
- difflib: SequenceMatcher owns its char vecs (bit-identical parity kept)
- --only now enforces gofile|0807 via value_parser

Docs:
- SORTING.md: drop phantom sorter_plan.jsonl, os.link -> std::fs::hard_link,
  zipfile -> the zip crate
- README: add --since to the dlp synopsis, clarify --subs/--assume-done
- .gitignore: ignore catalogue/sorter/_logs/subscriptions run artifacts

Validation: cargo fmt --check clean, cargo clippy --all-targets 0 warnings,
cargo test 87 passed / 0 failed.
2026-10-11 13:27:33 +02:00

7.7 KiB
Raw Permalink Blame History

Warhammer DLP Helper

Torrent-first platform for Warhammer proxy models (3rd-party / fan sculpts): export a Discord channel, extract the (often base64-encoded) download links, download per host, extract the archives, sort the files into the library, and record a manifest that will feed the torrent stage. One concise Rust binary + webapp (the webapp is the next stage).

Ported from the original Python implementation, which this version replaced. The Rust port of … provenance notes in the module headers name the Python sources it was derived from; AGENTS.md holds the project rules and docs/SORTING.md the library/sorting contract.

Crate choices (as ported)

Concern Original Python Rust
7z extraction 7z CLI, cp437 name repair, /var/tmp fallback sevenz-rust2 (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active)
zip extraction 7z CLI zip crate
RAR extraction (critical) 7z CLI unrar.rs (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (sevenz-rust2 ArchiveWriter) anyway
tar / gz / bz2 / xz 7z CLI deviation: system binaries (tar, gzip, bzip2, xz) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical
HTTP client requests reqwest (blocking first, async where the webapp needs it)
sha256 (gofile website token, --hash manifest) hashlib sha2
Fuzzy matching (sorter) difflib-style ratios exact port of CPython difflib.SequenceMatcher in src/difflib.rs (NOT strsim — ratios must be bit-identical for the thresholds)
CLI argparse clap (subcommands below)
Webapp — axum (CLI parity first, webapp second)
Discord export DCE docker container (external) unchanged: shell out to docker

Port status

All original Python entry points are ported and live-validated (see below).

Rust module Python source Tests
src/state.rs state.py ✓
src/globalstore.rs globalstore.py ✓
src/manifest.rs state.py append_manifest ✓
src/difflib.rs CPython difflib bit-identical on generated vectors
src/warhammer_data.rs + data/warhammer_data.json warhammer_data.py ✓ (order-sensitive, IndexMap)
src/discord.rs dlphelper/discord.py ✓ (DCE docker export, URL extraction)
src/gofile.rs dlphelper/gofile.py ✓ (website-token goldens)
src/zero807.rs dlphelper/zero807.py ✓ (curl subprocess contract)
src/extract.rs tools/extract_all.py ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, .broken/)
src/catalogue.rs dlphelper/catalogue.py ✓ (all python test cases)
src/sorter.rs dlphelper/sorter.py ✓ (all python test_sorter_units cases)
src/gwshop.rs dlphelper/gwshop.py ✓ (Algolia client, facet partition)
src/orchestrate.rs dlp.py pipeline ✓ (record/failed-urls/subs/assume-done; retry loops offline)
src/main.rs all CLIs smoke-tested (see below)

CLI

dlphelper dlp <channel_id> <out_dir> [--since WHEN] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global]

dlphelper dlp --subs FILE              # alternative mode: iterate every channel
                                       # in the list (channel_id/out_dir unused)

dlphelper dlp --assume-done FILE       # alternative mode: mark URLs as done, exit

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]

Notes:

  • The Discord token goes --token → $DISCORD_TOKEN → hidden prompt, and is only ever passed to the DCE container via env, never argv.
  • sorter --catalog defaults to data/gwshop_catalog.json, relative to the current working directory — pass --catalog when running elsewhere.
  • sorter --audit defaults to <library>/../_logs/sorter_applied.jsonl (bookkeeping stays out of the library, which may be a seeded torrent).
  • The sorter implements the rules in docs/SORTING.md — that document is the working agreement; change it first, then the code.
  • dlp --since limits a run to recent messages (e.g. 5d, 12h, 2026-10-04); --subs and --assume-done are alternative modes that ignore <channel_id>/<out_dir>.
  • Retry loops, cooldowns and the "one gofile guest account per run" rule are ported verbatim from dlp.py; a failure in one URL never aborts the run.
  • Ctrl-C: state.json is saved after every URL; re-run to resume (the python "interrupted" record on the in-flight URL is not replicated).

Deviations from the python version (known, accepted)

  • tar/gz/bz2/xz extraction shells out to the system binaries (7z CLI in python); 7z/zip/rar use native Rust.
  • The extractor also recognizes .tgz (a gzip-compressed tar). The python suffix set never included it, so python left those files unextracted; this is a deliberate Rust-only addition.
  • 0807.st folder pages (https://0807.st/f/<id>) are expanded: the page is parsed and each listed file is downloaded from https://0807.st/<id>?dl=1 (a 302 to the CDN, with the real file name in the page) into <out_dir>/downloads/0807/<page-id>/. Python (and earlier Rust) saved the HTML page itself; the folder .zip export is rejected by the site as too large, so per-file download is the only way in.
  • fs::copy fallback in the sorter does not preserve mtime (python shutil.copy2 does).
  • state.json omits "files": [] for records without files (python writes the empty list); readers of both versions accept both.
  • sorter --json prints keys alphabetically (serde_json, no preserve_order); python printed them in dataclass order. Content is identical.
  • Negative --retries/--limit values are rejected by clap (python accepted them with odd semantics).

Live validation (2026-10-09/11, channel 1455993943471820842)

Initial run with --since 5d: DCE export via docker (token env-only), URL extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account, file-level skip-existing, retry on network errors), 0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json resume, RAR extraction through unrar (1.4G archive -> STLs), nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files, 82 GiB.

Update run with --since 6d --fresh: re-exported the channel, kept 35 of 445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links were 0807 folder pages (/f/<id>): the new expansion fetched 12 files / ~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into downloads/0807/<page-id>/. 36/36 URLs ok, 144 files, 105 GiB.

Extraction of the whole run: 106 archives, 0 failed, all .tgz units expanded; then sorter --apply against the production library Warhammer_Library_Live (12k + 5.5k new hardlinks, skip-existing, 0 copied, low-confidence doc beside the library).

Still not live-tested: a full Algolia gwshop browse (needs credentials at runtime).

Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
Reference in New Issue View Git Blame Copy Permalink