Ben_Kosytorz/Warhammer_DLP_Helper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
45 Commits 1 Branch 0 Tags
81972afd226f4ac96fa0895b05f3490fd511d7ff
Commit Graph
10 Commits
Author SHA1 Message Date
Ben_Kosytorz 81972afd22 fix(rust): sniff container magic before extension in the extractor 
Real 7z archives named .zip (seen live: 'Heresy Lab - HL068 Utela Justina.zip')
were quarantined because dispatch trusted the extension. python's 7z CLI
sniffs content; now so do we for 7z/zip/rar, keeping extension logic for
gz/bz2/xz/tar where the tar-inside decision needs the name.
 2026-10-10 20:12:32 +02:00
Ben_Kosytorz ef5cf04c52 feat(rust): never download .torrent links or torrent files in gofile content 
User policy: the channel hosts user-created torrents (one ~2.5 TB); this
tool is for direct-download packs. Torrent links are recorded as done with
a note so they never re-download.
 2026-10-10 09:17:00 +02:00
Ben_Kosytorz 53da8dc3c2 feat(rust): dlp --since — filter URLs by message timestamp 
Rust-only extension (no python counterpart): --since 5d / 12h / 2026-10-04
keeps only URLs posted at/after the cutoff. URLs without a timestamp
(e.g. --urls-file lists) are kept with a warning.
 2026-10-09 23:38:29 +02:00
Ben_Kosytorz 55d32379ed feat(rust): port dlp.py orchestration + unified CLI (clap subcommands) 
- orchestrate.rs: process_channel, run_gofile (re-auth on AuthError,
  no-retry branches for dead/password/empty, cooldowns), run_0807,
  subscriptions, --assume-done (with note), failed_urls export
- main.rs: dlp / catalogue / sorter / extract / gwshop subcommands,
  token resolution (--token -> $DISCORD_TOKEN -> hidden prompt)
- globalstore: mark() gains a note field (python note= kwarg)
- README: port status, CLI, deviations, untested live paths

77 tests green.
 2026-10-09 15:30:27 +02:00
Ben_Kosytorz 1d3f83d950 feat(rust): port gwshop.py — Algolia catalog dump 
- facet-partitioned full-index retrieval (search-only key, /query only),
  retries (2/5/15s), credentials from env or gitignored algolia.json
- normalize/build_catalog: primary = longest GameSystemsRoot path,
  system/group/faction/subfaction/class expansion, objectID dedup
- 5 offline tests (hierarchy paths verified against python)
 2026-10-09 14:48:49 +02:00
Ben_Kosytorz 964400cb1e feat(rust): port sorter.py — fuzzy matching, plan, hardlink apply 
- src/sorter.rs: inspect (zip/7z listing via crates, HTML-page detection),
  keyword classification with inner-name votes, catalog load with dropped
  systems, fuzzy matching on the CPython-difflib port (containment boost,
  generic-token stoplist, tokenless bonus, weak tier, cross-system gates),
  deepest-dir re-homing incl. provenance unit mode, build_target,
  hardlink-apply with audit + low-confidence doc, format_plan
- all 9 python test_sorter_units.py cases ported and passing
- parity fixes found while porting: inner-name votes REPLACE (not max-keep)
  the outer score below threshold; python max() keeps first maximal vote;
  ratio cache keyed with allow_single; 512-byte head read for HTML sniff
 2026-10-09 14:44:09 +02:00
Ben_Kosytorz 137efa7e6c feat(rust): port discord, gofile, zero807, extract_all and catalogue 
Parallel module ports (behavior parity with the python versions):

- src/discord.rs: DCE docker export (token via env only), base64/plain URL
  extraction with grab.sh-compatible regexes, host classification; 11 tests
- src/gofile.rs: GoFile API client — one guest account per run, linear
  backoff on account creation, sha256 website token, recursive content walk,
  streaming .part downloads; 7 tests (offline goldens from python hashlib)
- src/zero807.rs: curl downloader with UA+Referer contract, dead-link
  detection; 4 tests
- src/extract.rs: recursive extraction (sevenz-rust2 for 7z, zip crate,
  unrar for rar, system tar/gzip/bzip2/xz for tarballs), nested BFS
  expansion, .broken quarantine, extract_manifest.jsonl, idempotent re-runs;
  8 tests
- src/catalogue.rs: keyword classifier over the dumped taxonomy, manifest/
  directory scanning, overrides, md+jsonl reports, audited moves; 7 tests
  incl. all 17 python test cases
 2026-10-09 14:17:02 +02:00
Ben_Kosytorz 519e7904dc feat(rust): difflib port + taxonomy data dump 
- src/difflib.rs: faithful port of CPython difflib.SequenceMatcher
  (Ratcliff/Obershelp ratio incl. real_quick_ratio/quick_ratio and autojunk),
  validated against python-computed test vectors with exact float equality
- data/warhammer_data.json: taxonomy tables dumped from the python module
  (tools/dump_warhammer_data.py), parsed order-preserving in
  src/warhammer_data.rs (python dict insert-order + last-value-wins semantics)
- deps for the module ports: clap, reqwest, url, base64, regex, sha2, hex,
  sevenz-rust2, unrar, zip, indexmap, rand, rpassword
 2026-10-09 13:44:28 +02:00
Ben_Kosytorz 59a89fa06a feat(rust): port state.py, globalstore.py and manifest appending 
- src/state.rs: per-run state.json (status/attempts/error/files/updated),
  corrupt-file -> *.json.bak restart, atomic tmp+rename save, mark() parity
  (attempts increments, None fields never clear existing values)
- src/globalstore.rs: cross-run urls.json at XDG_DATA_HOME or
  ~/.local/share/dlphelper/urls.json, disabled-store no-op
- src/manifest.rs: append_manifest as compact JSONL, UTF-8
- unknown/extra JSON fields are preserved on round-trip so both versions can
  read each other's artifacts (schema contract per AGENTS.md)
- 14 unit tests
 2026-10-09 13:27:59 +02:00
Ben_Kosytorz 271fb3bf68 chore: start transition layout — python into version-python/, scaffold version-rust/ 
- git mv of all existing files into version-python/ (history preserved)
- version-rust/: minimal cargo scaffold (edition 2024), no deps yet
- root AGENTS.md documents the transition rules and artifact/schema contract
- main is untouched; the python layout there stays as-is
 2026-10-09 13:18:31 +02:00