From fcec02747359c0ab389445e18a0cc752812aceaa Mon Sep 17 00:00:00 2001 From: Ben_Kosytorz Date: Wed, 7 Oct 2026 08:04:09 +0200 Subject: [PATCH] docs: non-warhammer system drop policy (Middle-Earth default-dropped) --- AGENTS.md | 27 +++++++++++++++++++++++++-- docs/SORTING.md | 8 ++++++++ 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b6808f1..d4b3078 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -225,9 +225,32 @@ never re-fetch them again): (`CROSS_SYSTEM_MIN = 0.75`) - weaker guesses stay in `_unsorted`. Token-containment boosts exist for short names vs long catalog names ("telemon" → "Telemon Heavy Dreadnought"), guarded by a generic-token - stoplist. The catalog deliberately covers all GW systems (needed for + stoplist. + + **Refined 2026-10-07 (dir re-homing + weak tier + non-warhammer drop):** + - **Deepest-matching-dir wins**: ancestor directory names (inside the + archive in provenance mode, real dirs in manual trees) compete with + the pack/file name; the best accepted match re-homes the whole + subtree. Junk dir names (files/images/HEADS/pure numbers/lone generic + words) are never candidates. Fixes multi-unit sculptor packs (one + "Retro Elvez" archive ≈ 20 Eldar units no longer all land in one + keyword class). + - **Weak tier**: when nothing reaches the threshold, a hit >= `WEAK_MIN` + (0.50) is accepted but flagged low confidence - `[w]` in the plan, + `confidence: low` in the audit, listed in + `/sorter_low_confidence.txt`. Weak and tokenless adoptions + require a shared non-generic token; tokenless normal adoption needs + +0.08 (`TOKENLESS_BONUS`); cross-system adoption additionally needs + token containment (`_token_contained`), because char-ratio lies + ("aeldari rangers" vs "dwarf rangers" = 0.786). + - **Non-warhammer systems dropped**: `DROPPED_SYSTEMS` (default + `{Middle-Earth}`) - catalog entries filtered at load, keyword hits + count as unclassified; files stay in `_unsorted`. `--drop-system`/ + `--keep-system` adjust. The catalog snapshot keeps all systems. + + The catalog deliberately covers all GW systems (needed for catalog-agnostic use) - wrong-system placement is prevented by matching - discipline, not by shrinking the catalog. + discipline and the drop filter, not by shrinking the catalog. **Catalog source: the GW webshop via Algolia (`dlphelper/gwshop.py`, working, verified 2026-10-04 — how it was obtained, do not re-derive):** diff --git a/docs/SORTING.md b/docs/SORTING.md index 699cd11..580e204 100644 --- a/docs/SORTING.md +++ b/docs/SORTING.md @@ -140,6 +140,14 @@ already-present duplicates, 0 copied): truncated/HTML dead-link downloads are listed in `/.broken/RETRY_URLS.txt` (all 0807.st; re-run `python3 dlp.py --urls-file ...` for them). +- **Non-warhammer systems are dropped** (2026-10-07, user decision: + "drop anything that isn't really warhammer"). The sorter default + (`DROPPED_SYSTEMS` in `dlphelper/sorter.py`) is `{Middle-Earth}`: + catalog entries of dropped systems are filtered at load and keyword + hits for them count as unclassified, so nothing is ever placed into + those sections - the files stay in `_unsorted`. `--drop-system NAME` + drops more, `--keep-system NAME` re-enables one. The committed catalog + snapshot itself is untouched. ## Catalog file (`data/library_catalog.json`)