docs(rust): record the 6d update run and 0807 folder-page expansion

This commit is contained in:
2026-10-11 08:54:58 +02:00
parent 3b3c4b57ea
commit 356b9089b7
+26 -9
View File
@@ -94,6 +94,12 @@ Notes:
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python - The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
suffix set never included it, so python left those files unextracted; this suffix set never included it, so python left those files unextracted; this
is a deliberate Rust-only addition. is a deliberate Rust-only addition.
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
(a 302 to the CDN, with the real file name in the page) into
`<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
HTML page itself; the folder `.zip` export is rejected by the site as too
large, so per-file download is the only way in.
- `fs::copy` fallback in the sorter does not preserve mtime - `fs::copy` fallback in the sorter does not preserve mtime
(python `shutil.copy2` does). (python `shutil.copy2` does).
- state.json omits `"files": []` for records without files - state.json omits `"files": []` for records without files
@@ -103,18 +109,29 @@ Notes:
- Negative `--retries`/`--limit` values are rejected by clap (python accepted - Negative `--retries`/`--limit` values are rejected by clap (python accepted
them with odd semantics). them with odd semantics).
## Live validation (2026-10-09/10, channel 1455993943471820842) ## Live validation (2026-10-09/11, channel 1455993943471820842)
Tested against a real channel with `--since 5d`: DCE export via docker Initial run with `--since 5d`: DCE export via docker (token env-only), URL
(token env-only), URL extraction (449 base64 + 8 plain -> 441 unique), extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
gofile (guest account, file-level skip-existing, retry on network errors), file-level skip-existing, retry on network errors), 0807 via curl
0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution, (zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
state.json resume, RAR extraction through `unrar` (1.4G archive -> STLs), resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
129 files, 82 GiB. 82 GiB.
Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
expanded; then `sorter --apply` against the production library
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
low-confidence doc beside the library).
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
runtime) and `sorter --apply` against the production library. runtime).
## Build/test on NFS ## Build/test on NFS