docs(rust): record the 6d update run and 0807 folder-page expansion
This commit is contained in:
+26
-9
@@ -94,6 +94,12 @@ Notes:
|
||||
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
|
||||
suffix set never included it, so python left those files unextracted; this
|
||||
is a deliberate Rust-only addition.
|
||||
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
|
||||
parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
|
||||
(a 302 to the CDN, with the real file name in the page) into
|
||||
`<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
|
||||
HTML page itself; the folder `.zip` export is rejected by the site as too
|
||||
large, so per-file download is the only way in.
|
||||
- `fs::copy` fallback in the sorter does not preserve mtime
|
||||
(python `shutil.copy2` does).
|
||||
- state.json omits `"files": []` for records without files
|
||||
@@ -103,18 +109,29 @@ Notes:
|
||||
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
||||
them with odd semantics).
|
||||
|
||||
## Live validation (2026-10-09/10, channel 1455993943471820842)
|
||||
## Live validation (2026-10-09/11, channel 1455993943471820842)
|
||||
|
||||
Tested against a real channel with `--since 5d`: DCE export via docker
|
||||
(token env-only), URL extraction (449 base64 + 8 plain -> 441 unique),
|
||||
gofile (guest account, file-level skip-existing, retry on network errors),
|
||||
0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution,
|
||||
state.json resume, RAR extraction through `unrar` (1.4G archive -> STLs),
|
||||
nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok,
|
||||
129 files, 82 GiB.
|
||||
Initial run with `--since 5d`: DCE export via docker (token env-only), URL
|
||||
extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
|
||||
file-level skip-existing, retry on network errors), 0807 via curl
|
||||
(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
|
||||
resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
|
||||
extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
|
||||
82 GiB.
|
||||
|
||||
Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
|
||||
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
|
||||
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
|
||||
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
|
||||
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
|
||||
|
||||
Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
|
||||
expanded; then `sorter --apply` against the production library
|
||||
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
|
||||
low-confidence doc beside the library).
|
||||
|
||||
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
|
||||
runtime) and `sorter --apply` against the production library.
|
||||
runtime).
|
||||
|
||||
## Build/test on NFS
|
||||
|
||||
|
||||
Reference in New Issue
Block a user