Rewrote the readme to clearyfy the wibecoded nature of this script and

made it more concise. Fixed licencing. Added a wiki.
2026-08-22 16:54:05 +02:00
parent e5b7863320
commit 24a546acba
8 changed files with 377 additions and 208 deletions
@@ -0,0 +1,65 @@
+# Architecture
+
+## Source layout
+
+```
+Cargo.toml          binary + lib; features: default=["masks"], gui
+src/
+  main.rs           CLI entry, clap parsing, `all` orchestration, GUI launch
+  lib.rs            crate root, re-exports
+  project.rs        project layout (01_frames … 04_colmap) + file gathering
+  logging.rs        emit / emit_progress ([progress] line protocol)
+  error.rs          error + cancellation types
+  math.rs           face rotations, quaternions, intrinsics (pure std)
+  json.rs           minimal JSON writer (pure std)
+  selftest.rs       rig self-test (6 check groups)
+  gpu/
+    mod.rs          GpuContext (device/queue), texture/readback helpers
+    shaders/        equirect_to_cubemap.wgsl, sharpness.wgsl, c2e.wgsl
+  stages/
+    extract.rs      stage 0: GPU sharpness + selection + frame writes
+    masks.rs        stage 1: YOLO segmentation (ONNX Runtime / MIGraphX)
+    cubemap.rs      stage 2: GPU equirect → cubemap
+    rig.rs          stage 3: rig config + nested tree
+  video/
+    mod.rs          ffmpeg/ffprobe CLI decode → raw RGBA frames
+  gui.rs            egui+wgpu GUI (feature `gui`)
+offline_check/      pure-std fallback build of the rig core (no cargo/network)
+```
+
+## How a run flows
+
+1. **CLI** parses the command; `all` builds a `GpuContext` once and runs each
+   stage in order, sharing the device/queue.
+2. **Extract** pipes decoded frames from ffmpeg to the GPU `sharpness` kernel,
+   reads back tiny per-workgroup metrics, filters by quality, selects frames,
+   and writes them out.
+3. **Cubemap** uploads each equirect frame and dispatches the
+   `equirect_to_cubemap` kernel once per face, reading back square faces.
+4. **Masks** loads `yolo11x-seg.onnx` into ONNX Runtime (with the MIGraphX EP
+   when requested), runs each cube face through the network, and synthesizes a
+   binary mask (see [GPU](GPU.md)).
+5. **Rig** scans images + masks, writes `rig_config.json`, and links the nested
+   COLMAP tree (symlink/hardlink/copy).
+
+## Why Rust + WebGPU
+
+- **Parallelism**: per-pixel/per-frame work maps naturally onto a compute
+  shader; `rayon` covers CPU-side orchestration.
+- **Cross-vendor GPU**: one WGSL codebase → Vulkan / Metal / DX12 / WebGPU,
+  instead of per-vendor libraries.
+- **Safety**: Rust's ownership model + wgpu's resource lifetimes avoid
+  use-after-free and data races.
+- **Distribution**: a single binary; no Python environment.
+
+Video **decode** stays on the CPU through ffmpeg because codecs aren't
+something you hand-roll on a bindless GPU layer; everything after decode is GPU.
+
+## Feature flags
+
+| Feature | Effect |
+| --- | --- |
+| `masks` (default) | enable ONNX Runtime YOLO masks (`ort` + `ndarray`, `load-dynamic`, `migraphx`) |
+| `gui` | build the native egui+wgpu GUI into the same binary |
+
+`cargo build --no-default-features` gives a light build with no ONNX Runtime.
@@ -0,0 +1,75 @@
+# GPU
+
+This project uses **WebGPU** via the `wgpu` crate, so the same code runs on
+Vulkan, Metal, DX12, and WebGPU/wasm.
+
+## GPU selection (all stages)
+
+By default the pipeline asks wgpu for the **discrete** (high-performance)
+adapter and logs which one it picked:
+
+```
+[gpu] using AMD Radeon RX 9070 XT (RADV GFX1201) (DiscreteGpu) via Vulkan
+```
+
+If you see an integrated GPU or a software (llvmpipe) adapter, force the right
+backend/device. The `request_adapter` fallback enumerates adapters and prefers
+a discrete GPU. On dual-GPU laptops set `WGPU_POWER_PREF` / run under the
+discrete-GPU driver profile if wgpu still picks the iGPU.
+
+## YOLO masks on the GPU (ONNX Runtime → MIGraphX)
+
+The masks stage runs YOLO through **ONNX Runtime**, loaded **dynamically from
+the system libonnxruntime** (`ORT_DYLIB_PATH`) rather than a downloaded
+CPU-only build. On AMD/ROCm systems that system lib is built with the
+**MIGraphX** execution provider, so:
+
+```bash
+# GPU inference on an AMD GPU
+pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
+    --mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280
+
+# CPU (no ROCm/MIGraphX runtime needed)
+pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
+    --mask-model yolo11x-seg.onnx --mask-device cpu
+```
+
+- If the GPU EP can't be registered, the stage **falls back to CPU** and warns.
+- The binary is built with `--features masks` by default.
+- On your system the GPU-enabled `libonnxruntime` must be findable — the build
+  uses `load-dynamic`, so it `dlopen`s `libonnxruntime.so` (or whatever
+  `ORT_DYLIB_PATH` points at) at runtime, and ROCm must be installed.
+
+### Compiled-model cache (important for first run)
+
+MIGraphX compiles the ONNX graph to GPU kernels on the **first run**, which is
+slow (minutes for a large model like `yolo11x-seg` on a new GPU). It caches the
+compiled `.mxr` and loads it on later runs.
+
+The pipeline **auto-sets `ORT_MIGRAPHX_CACHE_PATH`** to
+`~/.cache/pipeline_360/migraphx` (override with `--mask-ep-cache <dir>`), so:
+
+- first run with `--mask-device rocm` → compiles + saves (slow),
+- every later run → loads the cached model in seconds.
+
+### Env vars (useful)
+
+| Variable | Meaning |
+| --- | --- |
+| `ORT_DYLIB_PATH` | path to the libonnxruntime to dlopen at runtime |
+| `ORT_MIGRAPHX_CACHE_PATH` | MIGraphX compiled-model cache dir (auto-set) |
+| `ORT_MIGRAPHX_MODEL_CACHE_PATH` | alias some MIGraphX builds read |
+
+## Troubleshooting
+
+- **`[gpu] using llvmpipe / iGPU`** — set `WGPU_POWER_PREF=high` or switch to
+  the discrete GPU.
+- **"not a segmentation model"** — you exported the detection model; export the
+  **seg** one (`yolo11x-seg.pt`), not `yolo11x.onnx`.
+- **MIGraphX crash at inference with an empty `.mxr` path** — the cache env var
+  was unset. The pipeline sets it automatically; if you run via an external
+  launcher that strips env vars, set `ORT_MIGRAPHX_CACHE_PATH` yourself.
+- **First run "very slow"** — that's the one-time MIGraphX kernel compilation.
+  Let it finish; later runs are fast.
+- **CPU-only ONNX Runtime** — if you overrode `ORT_DYLIB_PATH` to a CPU build,
+  the MIGraphX EP won't be available and it falls back to CPU.
@@ -0,0 +1,53 @@
+# Getting Started
+
+## Prerequisites
+
+- A **Rust toolchain** (`cargo`, `rustc`).
+- **`ffmpeg` / `ffprobe`** on `PATH` — video decode shells out to the CLI and
+  pipes raw frames; no native FFmpeg bindings or dev headers are needed.
+- A GPU with a **WebGPU backend** — Vulkan (Linux/Windows), Metal (macOS),
+  DX12 (Windows), or WebGPU/wasm. On Linux, RADV (Mesa) or AMDVLK works.
+- For the **masks** stage: an **ONNX** export of a YOLO segmentation model
+  (e.g. `yolo11x-seg.onnx`). The `.pt` is not loadable in Rust:
+
+  ```bash
+  yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True
+  ```
+
+## Build
+
+Masks are enabled by default, so a plain build is the full pipeline:
+
+```bash
+cargo build
+```
+
+Lighter build without ONNX Runtime (no masks stage):
+
+```bash
+cargo build --no-default-features
+```
+
+Build with the native GUI:
+
+```bash
+cargo build --features gui
+```
+
+## First run (sanity check)
+
+```bash
+cargo run -- rig --self-test    # 6 check groups; should pass
+```
+
+## First real run
+
+```bash
+cargo run -- all --project-dir out --video day1.mp4 \
+    --mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280
+```
+
+- `--mask-device rocm` runs YOLO on an AMD GPU (see [GPU](GPU.md)); use `cpu`
+  if you don't have the ROCm/MIGraphX runtime.
+- The first GPU masks run compiles the model (slow); later runs are fast (see
+  the cache note in [GPU](GPU.md)).
@@ -0,0 +1,14 @@
+# pipeline_360 — Wiki
+
+Everything beyond the short [README](../README.md), in one place.
+
+## Contents
+
+- **[Getting Started](Getting-Started.md)** — prerequisites, build, first run
+- **[Usage](Usage.md)** — CLI reference and the native GUI
+- **[Pipeline](Pipeline.md)** — stages, output layout, naming conventions
+- **[GPU](GPU.md)** — WebGPU adapter selection, YOLO-on-GPU (MIGraphX), cache,
+  environment variables, troubleshooting
+- **[Architecture](Architecture.md)** — source layout and how a run flows
+
+> The original rewrite plan is still in [`../PLAN.md`](../PLAN.md).
@@ -0,0 +1,47 @@
+# Pipeline
+
+The rewrite follows the original Python design but reorders the stages so
+**cubemap runs before masks**: masks run on the cube-face images directly,
+which avoids a wasteful equirect→cubemap→equirect round-trip.
+
+## Output layout
+
+```
+out/
+  01_frames/                    selected sharpest equirect frames (multi-video → <stem> prefix)
+  02_masks/                     (legacy; no longer written by `all`)
+  03_cubemap/
+    images/                     {stem}_{FACE}.{ext}               (equirect → 6 cube faces)
+    masks/                      {stem}_{FACE}.{ext}.png           (per-face masks)
+  04_colmap/
+    images/                     {FACE}/{stem}.{ext}               (nested COLMAP tree)
+    masks/                      {FACE}/{stem}.{ext}.png
+    rig_config.json             rig cameras / intrinsics
+```
+
+`FACE` ∈ `{F, R, B, L, U, D}`. The nested `{FACE}/…` tree is produced via
+symlink / hardlink / copy (`--rig-link-mode`).
+
+## Stage details
+
+1. **Extract** — decodes the video(s) with ffmpeg, computes per-frame GPU
+   sharpness (variance-of-Laplacian), exposure, and contrast, optionally
+   filters out blurry / badly exposed / flat frames, checks inter-frame motion
+   (parallax), then selects the sharpest frame per chunk (or spreads them out).
+   Writes `_sharpness_metadata_<stem>.csv` so the analysis is reused on
+   subsequent runs.
+2. **Cubemap** — projects each equirect frame to six square cube faces on the
+   GPU (a faithful port of `py360convert.e2c`), batched for throughput.
+3. **Masks** — runs YOLO11x-seg on each cube face, synthesizes a binary mask
+   (COLMAP convention: **black = excluded**), and writes it next to the face
+   image. See [GPU](GPU.md).
+4. **Rig** — scans the cubemap images + masks, writes `rig_config.json`
+   (PINHOLE intrinsics + per-face `cam_from_rig_rotation`), and links the
+   nested COLMAP tree.
+
+## Naming / mask convention
+
+- Masks are named `<image_filename>.png` (hence `frame_00001_F.png.png`).
+- In a mask, **black pixels are excluded** (movable objects); white pixels are
+  kept (the static scene used for reconstruction). Masks mirror the image tree
+  so COLMAP attaches them to the right cameras.
@@ -0,0 +1,62 @@
+# Usage
+
+One binary, `pipeline_360`. Invoked with a **subcommand** it runs the CLI; with
+**no subcommand** (or `--gui`) it launches the native GUI (requires the `gui`
+feature).
+
+## `all` — full pipeline
+
+Runs extract → cubemap → masks → rig on one GPU device:
+
+```bash
+pipeline_360 all --project-dir out --video v1.mp4 [v2.mp4 …] \
+    --mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280 \
+    --extract-chunk-size 30 --cubemap-face-size 1600 --rig-ref-face F
+```
+
+- Multi-video: each source is prefixed with its filename stem in `01_frames`.
+- `--start-stage extract|cubemap|masks|rig` — resume from a later stage.
+- `--skip-masks` — skip the masks stage entirely.
+
+## Individual stages
+
+```bash
+# Extract sharpest frames (GPU sharpness + quality filtering)
+pipeline_360 extract --video v1.mp4 [v2.mp4 …] --output-dir out/01_frames \
+    --min-sharpness 0 --min-exposure 0 --max-exposure 255 \
+    --min-contrast 0 --min-motion 0 --max-motion 1e12 \
+    --selection chunk|spread --target-count 0 --min-gap 0 \
+    --bad-frame-action keep|skip|warn
+
+# Cubemap projection (equirect → 6 cube faces, GPU)
+pipeline_360 cubemap --frames-dir 01_frames --output-frames-dir 03_cubemap/images \
+    --face-size 1600 --skip-faces "U,D" --resume --jpeg-quality 95
+
+# Masks on the cube-face images (GPU via --mask-device rocm)
+pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
+    --mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280 \
+    --mask-classes person,car,bus,truck,bicycle,motorcycle \
+    --mask-conf 0.35 --mask-dilate 6 --mask-skip-faces "" --mask-empty-if-none
+
+# Rig config + nested COLMAP tree
+pipeline_360 rig --images-dir 03_cubemap/images --masks-dir 03_cubemap/masks \
+    --output rig_config.json --link-tree 04_colmap/images \
+    --masks-link-tree 04_colmap/masks --face-size 1600 --link-mode symlink
+```
+
+## GUI
+
+The native egui+wgpu GUI is in the same binary:
+
+```bash
+cargo build --features gui
+./target/debug/pipeline_360          # no subcommand → GUI
+./target/debug/pipeline_360 --gui    # explicit
+```
+
+The GUI shells out to the CLI subprocess (the same binary) with
+`--gui-log-output` and streams progress into a live log panel. It exposes every
+stage's options as forms.
+
+> `--mask-projection` and `--mask-face-size` are accepted for back-compat but
+> are no-ops: masks always run on cube-face images.