diff --git a/version-rust/README.md b/version-rust/README.md index 66e4baf..ec2f632 100644 --- a/version-rust/README.md +++ b/version-rust/README.md @@ -94,6 +94,12 @@ Notes: - The extractor also recognizes `.tgz` (a gzip-compressed tar). The python suffix set never included it, so python left those files unextracted; this is a deliberate Rust-only addition. +- 0807.st folder pages (`https://0807.st/f/`) are expanded: the page is + parsed and each listed file is downloaded from `https://0807.st/?dl=1` + (a 302 to the CDN, with the real file name in the page) into + `/downloads/0807//`. Python (and earlier Rust) saved the + HTML page itself; the folder `.zip` export is rejected by the site as too + large, so per-file download is the only way in. - `fs::copy` fallback in the sorter does not preserve mtime (python `shutil.copy2` does). - state.json omits `"files": []` for records without files @@ -103,18 +109,29 @@ Notes: - Negative `--retries`/`--limit` values are rejected by clap (python accepted them with odd semantics). -## Live validation (2026-10-09/10, channel 1455993943471820842) +## Live validation (2026-10-09/11, channel 1455993943471820842) -Tested against a real channel with `--since 5d`: DCE export via docker -(token env-only), URL extraction (449 base64 + 8 plain -> 441 unique), -gofile (guest account, file-level skip-existing, retry on network errors), -0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution, -state.json resume, RAR extraction through `unrar` (1.4G archive -> STLs), -nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, -129 files, 82 GiB. +Initial run with `--since 5d`: DCE export via docker (token env-only), URL +extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account, +file-level skip-existing, retry on network errors), 0807 via curl +(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json +resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested +extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files, +82 GiB. + +Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of +445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links +were 0807 folder pages (`/f/`): the new expansion fetched 12 files / +~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into +`downloads/0807//`. 36/36 URLs ok, 144 files, 105 GiB. + +Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units +expanded; then `sorter --apply` against the production library +`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied, +low-confidence doc beside the library). Still not live-tested: a full Algolia `gwshop` browse (needs credentials at -runtime) and `sorter --apply` against the production library. +runtime). ## Build/test on NFS