docs(rust): record the 6d update run and 0807 folder-page expansion
This commit is contained in:
+26
-9
@@ -94,6 +94,12 @@ Notes:
|
|||||||
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
|
- The extractor also recognizes `.tgz` (a gzip-compressed tar). The python
|
||||||
suffix set never included it, so python left those files unextracted; this
|
suffix set never included it, so python left those files unextracted; this
|
||||||
is a deliberate Rust-only addition.
|
is a deliberate Rust-only addition.
|
||||||
|
- 0807.st folder pages (`https://0807.st/f/<id>`) are expanded: the page is
|
||||||
|
parsed and each listed file is downloaded from `https://0807.st/<id>?dl=1`
|
||||||
|
(a 302 to the CDN, with the real file name in the page) into
|
||||||
|
`<out_dir>/downloads/0807/<page-id>/`. Python (and earlier Rust) saved the
|
||||||
|
HTML page itself; the folder `.zip` export is rejected by the site as too
|
||||||
|
large, so per-file download is the only way in.
|
||||||
- `fs::copy` fallback in the sorter does not preserve mtime
|
- `fs::copy` fallback in the sorter does not preserve mtime
|
||||||
(python `shutil.copy2` does).
|
(python `shutil.copy2` does).
|
||||||
- state.json omits `"files": []` for records without files
|
- state.json omits `"files": []` for records without files
|
||||||
@@ -103,18 +109,29 @@ Notes:
|
|||||||
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
- Negative `--retries`/`--limit` values are rejected by clap (python accepted
|
||||||
them with odd semantics).
|
them with odd semantics).
|
||||||
|
|
||||||
## Live validation (2026-10-09/10, channel 1455993943471820842)
|
## Live validation (2026-10-09/11, channel 1455993943471820842)
|
||||||
|
|
||||||
Tested against a real channel with `--since 5d`: DCE export via docker
|
Initial run with `--since 5d`: DCE export via docker (token env-only), URL
|
||||||
(token env-only), URL extraction (449 base64 + 8 plain -> 441 unique),
|
extraction (449 base64 + 8 plain -> 441 unique), gofile (guest account,
|
||||||
gofile (guest account, file-level skip-existing, retry on network errors),
|
file-level skip-existing, retry on network errors), 0807 via curl
|
||||||
0807 via curl (zip/7z/rar/torrent), sha256 hashing, manifest attribution,
|
(zip/7z/rar/torrent), sha256 hashing, manifest attribution, state.json
|
||||||
state.json resume, RAR extraction through `unrar` (1.4G archive -> STLs),
|
resume, RAR extraction through `unrar` (1.4G archive -> STLs), nested
|
||||||
nested extraction, and the catalogue CLI on the run dir. 31/31 URLs ok,
|
extraction, and the catalogue CLI on the run dir. 31/31 URLs ok, 129 files,
|
||||||
129 files, 82 GiB.
|
82 GiB.
|
||||||
|
|
||||||
|
Update run with `--since 6d --fresh`: re-exported the channel, kept 35 of
|
||||||
|
445 URLs, skipped 30 already-done, downloaded 5 new. Two of the new links
|
||||||
|
were 0807 folder pages (`/f/<id>`): the new expansion fetched 12 files /
|
||||||
|
~19 GiB with their real names (HQ, Troops, Elites, Terrain, ...) into
|
||||||
|
`downloads/0807/<page-id>/`. 36/36 URLs ok, 144 files, 105 GiB.
|
||||||
|
|
||||||
|
Extraction of the whole run: 106 archives, 0 failed, all `.tgz` units
|
||||||
|
expanded; then `sorter --apply` against the production library
|
||||||
|
`Warhammer_Library_Live` (12k + 5.5k new hardlinks, skip-existing, 0 copied,
|
||||||
|
low-confidence doc beside the library).
|
||||||
|
|
||||||
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
|
Still not live-tested: a full Algolia `gwshop` browse (needs credentials at
|
||||||
runtime) and `sorter --apply` against the production library.
|
runtime).
|
||||||
|
|
||||||
## Build/test on NFS
|
## Build/test on NFS
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user