docs: first full extraction + sort run - CIFS inode gotcha, cp437 fallback, .broken/RETRY_URLS policy

This commit is contained in:
2026-10-06 18:58:55 +02:00
parent 14ffc9e918
commit c3562f63d2
2 changed files with 24 additions and 1 deletions
+4 -1
View File
@@ -63,7 +63,10 @@ tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself)
tools/debug_gofile.py inspect raw gofile API response for one content id
tools/extract_all.py recursive archive extraction into a plain-file tree
(extract_manifest.jsonl feeds the sorter's
--provenance unit mode; failures go to .broken/)
--provenance unit mode; failures go to .broken/;
7z-direct-to-CIFS failures fall back to a
/var/tmp extraction with cp437 name repair;
AppleDouble/__MACOSX junk is dropped)
tests/test_extract.py offline extraction test (plain python3, no deps)
tests/test_catalogue.py offline classifier test
```
+20
View File
@@ -121,6 +121,26 @@ plan/apply-only):
python3 sorter.py <extracted_root> --library <lib> \
--provenance <extracted_root>/extract_manifest.jsonl --apply
Operational notes from the first full run (2026-10-06, sources
`unsorted/1..4` → `extracted` ≈ 1.4 TB / 260k files / 0 archives left;
library rebuilt from scratch: 218592 hardlinked, 30584 skipped as
already-present duplicates, 0 copied):
- The CIFS NAS mount reports **fabricated per-path inode numbers**
(`noserverino` behavior): two paths to the same hardlinked file show
different `st_ino`, and the values are stable across calls. To verify a
link, compare `st_nlink` (and content/size) — never inodes. Server-side
links are real: a fresh `os.link` bumps nlink (2→3 in the checked case).
- Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2
("Cannot open output file"); `extract_all.py` then extracts to a
`/var/tmp` temp dir, repairs the names and moves the result over.
- `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not
extracted.
- Files under hidden trees (`.broken/`) are never planned; retries of the
truncated/HTML dead-link downloads are listed in
`<extracted>/.broken/RETRY_URLS.txt` (all 0807.st; re-run
`python3 dlp.py <channel> <out> --urls-file ...` for them).
## Catalog file (`data/library_catalog.json`)
```json