# version-rust

Rust rewrite of `version-python` (same repo, `main` branch still carries the
flat python layout). Goal: one concise binary + webapp, eventually replacing
the python tooling.

## Why this structure

- `main` stays untouched (flat python layout) until the rewrite reaches parity.
- This branch re-homes the python code into `version-python/` (via `git mv`,
  history preserved) and starts `version-rust/` beside it.
- `version-python/AGENTS.md` still documents the python stage — its rules
  (secrets, failure policy, idempotency, rate limiting) apply to the rewrite
  too and are the contract the Rust version must keep.

## Crate choices (as ported)

| Concern | Python today | Rust |
|---|---|---|
| 7z extraction | 7z CLI, cp437 name repair, /var/tmp fallback | `sevenz-rust2` (pure Rust; LZMA/LZMA2/PPMD/AES-256 decrypt, v0.23, active) |
| zip extraction | 7z CLI | `zip` crate |
| RAR extraction (critical) | 7z CLI | `unrar.rs` (muja/unrar.rs — bindings to the official UnRAR library, RAR5 support, extraction only). Note: UnRAR license is freeware-not-OSI and forbids building a RAR compressor — acceptable, we only extract, and compression later is 7z-only (`sevenz-rust2` `ArchiveWriter`) anyway |
| tar / gz / bz2 / xz | 7z CLI | **deviation:** system binaries (`tar`, `gzip`, `bzip2`, `xz`) — no mature pure-Rust tar+filter combo that matches the 7z-pass behavior; contract (dest layout, quarantine, manifest) is identical |
| HTTP client | `requests` | `reqwest` (blocking first, async where the webapp needs it) |
| sha256 (gofile website token, --hash manifest) | hashlib | `sha2` |
| Fuzzy matching (sorter) | difflib-style ratios | exact port of CPython `difflib.SequenceMatcher` in `src/difflib.rs` (NOT strsim — ratios must be bit-identical for the thresholds) |
| CLI | argparse | `clap` (subcommands below) |
| Webapp | — | `axum` (CLI parity first, webapp second) |
| Discord export | DCE docker container (external) | unchanged: shell out to docker |

## Port status

All python entry points are ported; live network paths are tested offline
only (see "Untested live paths").

| Rust module | Python source | Tests |
|---|---|---|
| `src/state.rs` | state.py | ✓ |
| `src/globalstore.rs` | globalstore.py | ✓ |
| `src/manifest.rs` | state.py append_manifest | ✓ |
| `src/difflib.rs` | CPython difflib | bit-identical on generated vectors |
| `src/warhammer_data.rs` + `data/warhammer_data.json` | warhammer_data.py | ✓ (order-sensitive, IndexMap) |
| `src/discord.rs` | dlphelper/discord.py | ✓ (DCE docker export, URL extraction) |
| `src/gofile.rs` | dlphelper/gofile.py | ✓ (website-token goldens) |
| `src/zero807.rs` | dlphelper/zero807.py | ✓ (curl subprocess contract) |
| `src/extract.rs` | tools/extract_all.py | ✓ (7z/zip/rar/tar+bz2/xz, nested, sweep, `.broken/`) |
| `src/catalogue.rs` | dlphelper/catalogue.py | ✓ (all python test cases) |
| `src/sorter.rs` | dlphelper/sorter.py | ✓ (all python test_sorter_units cases) |
| `src/gwshop.rs` | dlphelper/gwshop.py | ✓ (Algolia client, facet partition) |
| `src/orchestrate.rs` | dlp.py pipeline | ✓ (record/failed-urls/subs/assume-done; retry loops offline) |
| `src/main.rs` | all CLIs | smoke-tested (see below) |

## CLI

```
dlphelper dlp <channel_id> <out_dir> [--subs FILE] [--urls-file FILE] [--token]
    [--password] [--retries N] [--retry-delay N] [--cooldown-failures N]
    [--cooldown-seconds N] [--only gofile|0807] [--limit N] [--shuffle]
    [--hash] [--extract] [--fresh|--no-fresh] [--dry-run]
    [--global-state FILE] [--no-global] [--assume-done FILE]

dlphelper catalogue [run_dir] [--scan DIR]... [--manifest FILE]
    [--overrides FILE] [--out-md FILE] [--out-jsonl FILE] [--apply --library DIR]

dlphelper sorter <source> --library DIR [--catalog FILE] [--threshold 0.62]
    [--apply] [--audit FILE] [--provenance FILE] [--json]
    [--drop-system NAME]... [--keep-system NAME]...

dlphelper extract <SOURCES>... --out DIR

dlphelper gwshop <out> [--index NAME] [--raw FILE]
```

Notes:

- The Discord token goes `--token` → `$DISCORD_TOKEN` → hidden prompt, and is
  only ever passed to the DCE container via env, never argv.
- `sorter --catalog` defaults to `version-python/data/gwshop_catalog.json`
  (falling back to `data/gwshop_catalog.json`), both relative to the current
  working directory — pass `--catalog` when running elsewhere.
- `sorter --audit` defaults to `<library>/../_logs/sorter_applied.jsonl`
  (bookkeeping stays out of the library, which may be a seeded torrent).
- Retry loops, cooldowns and the "one gofile guest account per run" rule are
  ported verbatim from `dlp.py`; a failure in one URL never aborts the run.
- Ctrl-C: state.json is saved after every URL; re-run to resume (the python
  "interrupted" record on the in-flight URL is not replicated).

## Deviations from the python version (known, accepted)

- `tar`/`gz`/`bz2`/`xz` extraction shells out to the system binaries
  (7z CLI in python); 7z/zip/rar use native Rust.
- `fs::copy` fallback in the sorter does not preserve mtime
  (python `shutil.copy2` does).
- state.json omits `"files": []` for records without files
  (python writes the empty list); readers of both versions accept both.
- `sorter --json` prints keys alphabetically (serde_json, no preserve_order);
  python printed them in dataclass order. Content is identical.
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
  them with odd semantics).

## Untested live paths (validated offline only, per the failure-policy rules)

- Actual Discord channel export via the DCE container (needs a token) — the
  docker invocation and export.json parsing are unit-tested, the network path
  is not.
- Live gofile downloads (guest account creation, `ensure_token`) — website
  token derivation is golden-tested against python; the HTTP calls are not.
- Live 0807 downloads — the `curl -A "Mozilla/5.0" -e "https://0807.st/"`
  contract is kept, timeouts are per python (`--max-time 1800`), but no live
  fetch was made.
- Real `.rar` extraction — the extractor wiring is in place (`unrar` crate,
  unrar runtime library present), but no real RAR archive was available for a
  live test.
- `gwshop` was smoke-tested for request shaping only; a full Algolia browse
  needs valid credentials (`algolia.json` or env).

## Build/test on NFS

The repo lives on NFS; rustc SIGBUSes there. Use:

```
CARGO_TARGET_DIR=/var/tmp/dlphelper-target cargo test
```