From c3562f63d234d36ccefd65f41a5d1140979cdd03 Mon Sep 17 00:00:00 2001 From: Ben_Kosytorz Date: Tue, 6 Oct 2026 18:58:55 +0200 Subject: [PATCH] docs: first full extraction + sort run - CIFS inode gotcha, cp437 fallback, .broken/RETRY_URLS policy --- AGENTS.md | 5 ++++- docs/SORTING.md | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index f5b4d03..b6808f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,7 +63,10 @@ tools/new_repo.py Gitea repo helper (reads ~/.git-credentials itself) tools/debug_gofile.py inspect raw gofile API response for one content id tools/extract_all.py recursive archive extraction into a plain-file tree (extract_manifest.jsonl feeds the sorter's - --provenance unit mode; failures go to .broken/) + --provenance unit mode; failures go to .broken/; + 7z-direct-to-CIFS failures fall back to a + /var/tmp extraction with cp437 name repair; + AppleDouble/__MACOSX junk is dropped) tests/test_extract.py offline extraction test (plain python3, no deps) tests/test_catalogue.py offline classifier test ``` diff --git a/docs/SORTING.md b/docs/SORTING.md index 9b41fcd..699cd11 100644 --- a/docs/SORTING.md +++ b/docs/SORTING.md @@ -121,6 +121,26 @@ plan/apply-only): python3 sorter.py --library \ --provenance /extract_manifest.jsonl --apply +Operational notes from the first full run (2026-10-06, sources +`unsorted/1..4` → `extracted` ≈ 1.4 TB / 260k files / 0 archives left; +library rebuilt from scratch: 218592 hardlinked, 30584 skipped as +already-present duplicates, 0 copied): + +- The CIFS NAS mount reports **fabricated per-path inode numbers** + (`noserverino` behavior): two paths to the same hardlinked file show + different `st_ino`, and the values are stable across calls. To verify a + link, compare `st_nlink` (and content/size) — never inodes. Server-side + links are real: a fresh `os.link` bumps nlink (2→3 in the checked case). +- Filenames with raw cp437 bytes crash 7z direct-to-CIFS with exit 2 + ("Cannot open output file"); `extract_all.py` then extracts to a + `/var/tmp` temp dir, repairs the names and moves the result over. +- `__MACOSX/` and `._*` AppleDouble entries are deleted as junk, not + extracted. +- Files under hidden trees (`.broken/`) are never planned; retries of the + truncated/HTML dead-link downloads are listed in + `/.broken/RETRY_URLS.txt` (all 0807.st; re-run + `python3 dlp.py --urls-file ...` for them). + ## Catalog file (`data/library_catalog.json`) ```json