SHA256
Compare commits
4
Commits
e5b7863320
..
main
| Author | SHA256 | Date | |
|---|---|---|---|
 Ben_Kosytorz
|
cf0576c082 | ||
|
Ben_Kosytorz
|
b9dd04077a | ||
|
Ben_Kosytorz
|
1166ad15c7 | ||
|
Ben_Kosytorz
|
24a546acba |
+2
-1
@@ -3,7 +3,8 @@ name = "pipeline-360"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "360° equirect -> COLMAP cubemap rig pipeline (Rust/WebGPU rewrite)"
|
||||
license = "MIT"
|
||||
# The full GPL-3.0 text lives in ./LICENSE.
|
||||
license = "GPL-3.0-or-later"
|
||||
|
||||
[lib]
|
||||
name = "pipeline_360"
|
||||
|
||||
@@ -1,226 +1,81 @@
|
||||
# pipeline_360 — Rust/WebGPU rewrite
|
||||
# Warning
|
||||
**This project is fully vibecoded and behind no line in this code is human soul.**
|
||||
|
||||
An in-progress rewrite of the Python `pipeline_360.py` equirect→COLMAP-cubemap
|
||||
rig pipeline in **Rust**, with all heavy image math on the **GPU through
|
||||
WebGPU** (`wgpu`) for cross-vendor compatibility (Vulkan / Metal / DX12, and
|
||||
WebGPU/wasm).
|
||||
# pipeline_360
|
||||
|
||||
> **Status: complete — single binary, masks default, native GUI, GPU masks.**
|
||||
> One `pipeline_360` binary runs the full GPU pipeline and, with
|
||||
> `--features gui`, also launches a native egui+wgpu GUI when invoked with no
|
||||
> subcommand. The YOLO masks stage is on by default. Validated end-to-end on
|
||||
> real 8K equirect video. `cargo test` passes; 0 clippy warnings in pipeline
|
||||
> code. YOLO11x-seg via ONNX Runtime, and masks now run on the **GPU** (AMD
|
||||
> MIGraphX/ROCm) and on the **cube-face images** directly (no equirect
|
||||
> round-trip). All GPU stages are **batched** for throughput.
|
||||
> See [PLAN.md](PLAN.md) for the full design and milestone list.
|
||||
Convert equirectangular 360° videos into a **COLMAP-ready cubemap camera-rig
|
||||
dataset** — sharpest-frame extraction, movable-object masks, an
|
||||
equirect→cubemap projection, and a `rig_config.json` + nested file tree.
|
||||
|
||||
## Current state
|
||||
This is the **Rust + WebGPU rewrite** of the original Python pipeline (now in
|
||||
[python](https://git.dieterunterstrich.de/Ben_Kosytorz/3sixty2cubemap/src/branch/python_old), kept as legacy reference).
|
||||
|
||||
| Piece | Status |
|
||||
| --- | --- |
|
||||
| `math.rs` (face rotations, quaternions, intrinsics) | ✅ pure std, verified |
|
||||
| `stages/rig.rs` (rig_config.json + nested tree + linking) | ✅ pure std, verified |
|
||||
| `selftest.rs` (mirrors Python `rig --self-test`, 6 groups) | ✅ passes (`cargo test`) |
|
||||
| CLI / `main.rs` (derive-based, mirrors Python commands) | ✅ `extract` + `cubemap` + `all` + `rig` work |
|
||||
| `video/mod.rs` (ffmpeg/ffprobe CLI decode → rawvideo RGBA) | ✅ verified end-to-end |
|
||||
| `gpu/mod.rs` (adapter prefers **discrete** GPU; batched readback) | ✅ picks the dGPU (was defaulting to LowPower/iGPU) |
|
||||
| `gpu/shaders/equirect_to_cubemap.wgsl` | ✅ projection verified vs `py360convert.e2c` |
|
||||
| `gpu/shaders/sharpness.wgsl` (workgroup-reduced metrics) | ✅ verified end-to-end |
|
||||
| `gpu/shaders/c2e.wgsl` (cubemap→equirect nearest) | ✅ round-trip validated (legacy mask reprojection) |
|
||||
| `stages/extract.rs` (GPU sharpness + quality + selection, **batched**) | ✅ superset of Python; verified |
|
||||
| `stages/cubemap.rs` (GPU equirect→6 faces, skip/resume, **batched**) | ✅ verified vs `py360convert.e2c` |
|
||||
| `stages/masks.rs` (YOLO11x-seg on cube faces; **MIGraphX GPU**; cache) | ✅ IoU 0.96 vs ultralytics |
|
||||
| `gui.rs` (native **egui+wgpu** GUI; in the same binary via `--features gui`) | ✅ builds & runs |
|
||||
> ⚡ **Status: fully vibecoded, and the rewrite is done.** Everything described
|
||||
> here is implemented and running on real 8K test footage — GPU sharpness,
|
||||
> YOLO segmentation masks on the GPU, cubemap projection, and rig generation all
|
||||
> work end-to-end. It should just work; if something doesn't, the
|
||||
> [Wiki](wiki/Home.md) and issue tracker are the place to look.
|
||||
|
||||
## Build
|
||||
## What it does
|
||||
|
||||
There is **one binary**, `pipeline_360`. The YOLO masks stage is **on by
|
||||
default** (via ONNX Runtime). Use `--no-default-features` for a lighter build
|
||||
without masks.
|
||||
|
||||
```bash
|
||||
cargo build # default: full pipeline incl. YOLO masks
|
||||
./target/debug/pipeline_360 all --project-dir <proj> --video <v> --skip-masks
|
||||
cargo test # runs the rig self-test (6 groups)
|
||||
cargo run -- rig --self-test
|
||||
|
||||
# Lighter build without ONNX Runtime (no masks stage):
|
||||
cargo build --no-default-features
|
||||
|
||||
# Run the GPU extract stage (needs ffmpeg/ffprobe on PATH + a GPU):
|
||||
cargo run -- extract --video <video> --output-dir <out> \
|
||||
[--chunk-size 30] [--scale-width 1920] [--output-format png]
|
||||
[--min-sharpness 0] [--min-exposure 0 --max-exposure 255]
|
||||
[--min-contrast 0] [--min-motion 0 --max-motion 1e12]
|
||||
[--selection chunk|spread] [--target-count 0] [--min-gap 0]
|
||||
[--bad-frame-action keep|skip|warn] [--motion-mode frame_diff|off]
|
||||
|
||||
# Run the GPU cubemap stage (needs a Vulkan/Metal/DX12 device):
|
||||
cargo run -- cubemap --frames-dir <equirect_dir> --output-frames-dir <out> \
|
||||
[--face-size 1600] [--skip-faces "U,D"] [--resume]
|
||||
[--masks-dir <dir> --output-masks-dir <mask_out>] [--jpeg-quality 95]
|
||||
|
||||
# Native GUI (egui + wgpu; cross-vendor/cross-platform) — same binary, with the `gui` feature:
|
||||
cargo build --features gui
|
||||
./target/debug/pipeline_360 # no subcommand -> launches the GUI
|
||||
./target/debug/pipeline_360 --gui # explicit
|
||||
./target/debug/pipeline_360 all --project-dir <proj> --video <v> # subcommand -> CLI
|
||||
|
||||
# Full pipeline (extract → cubemap → masks → rig), as the GUI invokes it.
|
||||
# Note the order: cubemap runs *before* masks so masks run on the cube-face
|
||||
# images (no wasteful equirect round-trip).
|
||||
cargo run -- all --project-dir <proj> --video <v1> [<v2> …] \
|
||||
--extract-chunk-size 30 --cubemap-face-size 1600 --rig-ref-face F \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280
|
||||
|
||||
# YOLO masks stage (runs on the cube-face images the cubemap stage produced):
|
||||
# Export once: yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True
|
||||
cargo run -- masks --input-dir <03_cubemap/images> --output-dir <03_cubemap/masks> \
|
||||
[--mask-classes person,car,...] [--mask-model yolo11x-seg.onnx] \
|
||||
[--mask-device cpu|rocm] [--mask-imgsz 1280] [--mask-conf 0.35] \
|
||||
[--mask-dilate 6] [--mask-skip-faces ""] [--mask-empty-if-none] \
|
||||
[--mask-ep-cache <dir>] # MIGraphX compiled-model cache dir (see below)
|
||||
|
||||
# `--mask-projection` / `--mask-face-size` are accepted for CLI/GUI backwards
|
||||
# compatibility but are now no-ops: masks always run on cube-face images.
|
||||
```
|
||||
video(s) → 01_frames (sharpest, quality-filtered)
|
||||
→ 03_cubemap/images (equirect → 6 cube faces, on GPU)
|
||||
→ 03_cubemap/masks (YOLO11x-seg, on GPU)
|
||||
→ 04_colmap/{images,masks,rig_config.json} (nested tree for COLMAP)
|
||||
```
|
||||
|
||||
> Decode uses the **`ffmpeg` / `ffprobe` CLI** (piped `rawvideo`), so no native
|
||||
> FFmpeg bindings or `video` feature are required — just have `ffmpeg` on PATH.
|
||||
> Fetching crates and building `wgpu` requires network access. In this
|
||||
> environment that was granted, so `cargo build` / `cargo test` pass, and the
|
||||
> extract + cubemap stages were verified end-to-end.
|
||||
Everything heavy is on the **GPU via WebGPU** (`wgpu`) — Vulkan / Metal / DX12,
|
||||
so it's vendor-neutral. YOLO masks use the **system ONNX Runtime** with the AMD
|
||||
MIGraphX provider for GPU inference (falls back to CPU). A native **egui GUI**
|
||||
ships in the same binary.
|
||||
|
||||
## What the extract stage adds over the original Python
|
||||
|
||||
The original only picked the sharpest frame (variance-of-Laplacian) per fixed
|
||||
chunk. The Rust extract stage keeps that as the default and adds genuine
|
||||
quality control, useful for 360° SfM:
|
||||
|
||||
| Flag | Purpose |
|
||||
| --- | --- |
|
||||
| `--min-sharpness` | Drop blurry frames (variance-of-Laplacian floor) |
|
||||
| `--min-exposure` / `--max-exposure` | Drop over/under-exposed frames (mean luma 0..255) |
|
||||
| `--min-contrast` | Drop flat/featureless frames (luma std floor) |
|
||||
| `--min-motion` | **Parallax check** — require inter-frame motion (baseline) between kept frames |
|
||||
| `--max-motion` | Reject jittery/too-fast frames (likely motion blur) |
|
||||
| `--selection spread` | Pick well-distributed frames instead of fixed chunks |
|
||||
| `--target-count N` | Target exactly N frames (spread mode) |
|
||||
| `--min-gap` | Enforce a minimum frame spacing between selected frames |
|
||||
| `--bad-frame-action skip|keep|warn` | What to do when a chunk has no surviving frame |
|
||||
| `--motion-mode off|frame_diff` | Toggle the inter-frame motion proxy (cheap thumbnail diff) |
|
||||
|
||||
Metrics are written to `_sharpness_metadata[_<stem>].csv` as
|
||||
`frame,sharpness,exposure,contrast,motion` and reused on subsequent runs unless
|
||||
`--extract-regenerate-metadata` is given.
|
||||
|
||||
### Batching
|
||||
|
||||
`--extract-batch` (default 8) and `--cubemap-batch` (default 4) submit a group
|
||||
of frames' GPU work in one command encoder and read it all back with a single
|
||||
`device.poll` — amortising per-frame readback latency and letting the GPU
|
||||
pipeline across frames. Output is identical to the non-batched path (validated
|
||||
for both stages).
|
||||
|
||||
### Masks stage (GPU, on cube faces)
|
||||
|
||||
YOLO11x-seg segmentation via ONNX Runtime (`--features masks`). The masks
|
||||
stage runs **after** the cubemap stage and operates directly on the cube-face
|
||||
images it produced (`03_cubemap/images/{stem}_{FACE}.ext`). Rationale:
|
||||
|
||||
- Masking equirect frames directly is useless (poles distort objects).
|
||||
- The old design round-tripped `e2c → YOLO → c2e → equirect mask`, then the
|
||||
cubemap stage did `e2c` **again** — two projections + a stitch-back, wasted.
|
||||
- Now the cubemap stage projects once; masks run on the square, undistorted cube
|
||||
faces and stay as cube-face masks (`03_cubemap/masks/{stem}_{FACE}.ext.png`).
|
||||
- Because the inputs are square, the only resize is a uniform scale to `imgsz`
|
||||
with **no letterbox padding** — so there is no crop/offset to mis-scale on the
|
||||
way back. (The previous cubemap path used `r=imgsz/face_size` but letterboxed
|
||||
with `r=1.0` + a centre pad, producing **shifted masks**; that bug is gone by
|
||||
construction. Validated at IoU 0.96 vs ultralytics on a real cube face.)
|
||||
|
||||
Masks use the COLMAP convention: **black = excluded**. Movable objects
|
||||
(person, car, …) are masked out (black); the static scene is kept (white).
|
||||
|
||||
**GPU execution.** `ort` is built with `load-dynamic`, so it `dlopen`s the
|
||||
*system* `libonnxruntime` at runtime (override with `ORT_DYLIB_PATH`) instead
|
||||
of downloading a CPU-only build. On AMD/ROCm systems that lib is built with the
|
||||
**MIGraphX** execution provider, so `--mask-device rocm` runs YOLO on the
|
||||
Radeon GPU. Falls back to CPU automatically if the GPU EP can't be registered.
|
||||
## Build & run
|
||||
|
||||
```bash
|
||||
ORT_DYLIB_PATH=/usr/lib/libonnxruntime.so LD_LIBRARY_PATH=/opt/rocm/lib \
|
||||
cargo run -- masks --input-dir <cube_faces> --output-dir <masks> \
|
||||
cargo build # full pipeline (masks on by default)
|
||||
cargo run -- rig --self-test # quick sanity check
|
||||
|
||||
# single command, all four stages (cube faces → masks on the GPU):
|
||||
cargo run -- all --project-dir out --video vid1.mp4 [vid2.mp4 …] \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm
|
||||
|
||||
# native GUI (one binary, --features gui):
|
||||
cargo build --features gui && ./target/debug/pipeline_360
|
||||
```
|
||||
|
||||
**Compiled-model cache (auto).** MIGraphX compiles the graph to GPU kernels
|
||||
on the first run (slow — minutes for a large model on a new arch) and caches the
|
||||
compiled `.mxr`. The pipeline **auto-sets `ORT_MIGraphX_CACHE_PATH`** to
|
||||
`~/.cache/pipeline_360/migraphx` (override with `--mask-ep-cache <dir>`), so the
|
||||
first run compiles+saves and every later run loads in seconds. Without a cache
|
||||
dir MIGraphX would crash at inference on this build, so the cache is set up for
|
||||
you automatically.
|
||||
Prerequisites: a Rust toolchain, `ffmpeg`/`ffprobe` on `PATH`, a GPU with a
|
||||
WebGPU backend, and an **ONNX** export of the segmentation model
|
||||
(`yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True`).
|
||||
|
||||
```bash
|
||||
# first run: compiles + saves (slow)
|
||||
... masks --mask-device rocm ...
|
||||
# later runs: loads the cached model (fast)
|
||||
... masks --mask-device rocm ...
|
||||
# or use a custom cache dir:
|
||||
... masks --mask-device rocm --mask-ep-cache /some/dir ...
|
||||
## Features
|
||||
|
||||
- **Extract** — GPU sharpness/exposure/contrast scores + quality filtering,
|
||||
parallax (`--min-motion`) and bad-frame rejection, chunk or spread selection,
|
||||
multi-video, batched.
|
||||
- **Masks** — YOLO11x-seg on the cube-face images (no wasteful round-trip),
|
||||
COLMAP convention (black = excluded), GPU inference with auto cached
|
||||
compiled model (first run compiles, later runs are fast).
|
||||
- **Cubemap** — faithful port of `py360convert.e2c`, batched, skip/resume.
|
||||
- **Rig** — `rig_config.json` + nested `{FACE}/{stem}` tree, symlink/hardlink/copy.
|
||||
- **GUI** — native egui+wgpu, shells out to the CLI for the heavy work.
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
Cargo.toml, main.rs entry point, CLI, `all` orchestration
|
||||
src/gpu/ GpuContext + WGSL shaders
|
||||
src/stages/ extract, masks, cubemap, rig
|
||||
src/gui.rs egui+wgpu GUI (feature `gui`)
|
||||
wiki/ the documentation (setup, usage, GPU, troubleshooting)
|
||||
```
|
||||
|
||||
> The masks stage needs an **ONNX** export of the model (the `.pt` is not
|
||||
> loadable in Rust): `yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True`.
|
||||
## License
|
||||
|
||||
## Optional offline verification (no cargo)
|
||||
[GPL-3.0-or-later](LICENSE) — see [`LICENSE`](LICENSE).
|
||||
|
||||
The stage-4 core is pure `std`, so it can also be compiled and run with plain
|
||||
`rustc` as a fallback when crates.io is unreachable:
|
||||
## More
|
||||
|
||||
```bash
|
||||
rustc --edition 2021 --crate-name pipeline360 offline_check/main.rs \
|
||||
-o /tmp/pipeline360_selftest && /tmp/pipeline360_selftest
|
||||
# -> self-test: 6 check groups passed.
|
||||
```
|
||||
|
||||
`offline_check/main.rs` pulls the real source files via `#[path]` and is not
|
||||
part of the cargo build.
|
||||
|
||||
## Native GUI
|
||||
|
||||
`gui.rs` is a native **egui** app rendered through **wgpu** (via eframe) —
|
||||
the same WebGPU layer as the pipeline, so it is cross-vendor/cross-platform
|
||||
(Vulkan/Metal/DX12). It is part of the **same** `pipeline_360` binary, gated
|
||||
behind the `gui` feature — run the binary with no subcommand (or `--gui`) and it
|
||||
launches the GUI; give it a subcommand (`all`, `extract`, …) and it's the CLI:
|
||||
|
||||
```bash
|
||||
cargo build --features gui
|
||||
./target/debug/pipeline_360 # no subcommand -> GUI
|
||||
./target/debug/pipeline_360 all --project-dir <proj> --video <v> # subcommand -> CLI
|
||||
```
|
||||
|
||||
The GUI does **not** call the GPU pipeline library in-process. Instead it
|
||||
shells out to the `pipeline_360` CLI (itself, found beside the GUI exe, else on
|
||||
PATH) with `--gui-log-output`, streaming the `[progress] ` / log line protocol
|
||||
on a background thread into a live log panel — the same protocol the Python GUI
|
||||
uses. This keeps the heavy GPU work in the (optionally release-built) CLI and
|
||||
keeps the UI responsive, and means the GUI works anywhere the binary does.
|
||||
|
||||
The UI exposes every stage's options as forms with the CLI defaults: project
|
||||
dir + multi-video pickers, start stage, skip-masks, and collapsible sections for
|
||||
extract (incl. all the quality/selection/batch knobs), masks, cubemap, and rig,
|
||||
plus Run/Stop and a live log with progress.
|
||||
|
||||
> Building `--features gui` pulls a second wgpu major version (egui 0.36 uses
|
||||
> wgpu 30; the pipeline core uses wgpu 25). Both coexist fine because the GUI
|
||||
> process only renders with eframe's wgpu while the CLI subprocess does the
|
||||
> compute with its own. For a smaller binary build the GUI in release mode.
|
||||
|
||||
## Layout
|
||||
|
||||
See `PLAN.md` for the detailed design (architecture, stages, CLI parity,
|
||||
milestones, risks, verification strategy).
|
||||
- **Wiki** → [`wiki/Home.md`](wiki/Home.md) — setup, CLI reference, pipeline
|
||||
internals, GPU/MIGraphX setup, troubleshooting.
|
||||
- **Design notes** → [`PLAN.md`](PLAN.md) — the original rewrite plan.
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Architecture
|
||||
|
||||
## Source layout
|
||||
|
||||
```
|
||||
Cargo.toml binary + lib; features: default=["masks"], gui
|
||||
src/
|
||||
main.rs CLI entry, clap parsing, `all` orchestration, GUI launch
|
||||
lib.rs crate root, re-exports
|
||||
project.rs project layout (01_frames … 04_colmap) + file gathering
|
||||
logging.rs emit / emit_progress ([progress] line protocol)
|
||||
error.rs error + cancellation types
|
||||
math.rs face rotations, quaternions, intrinsics (pure std)
|
||||
json.rs minimal JSON writer (pure std)
|
||||
selftest.rs rig self-test (6 check groups)
|
||||
gpu/
|
||||
mod.rs GpuContext (device/queue), texture/readback helpers
|
||||
shaders/ equirect_to_cubemap.wgsl, sharpness.wgsl, c2e.wgsl
|
||||
stages/
|
||||
extract.rs stage 0: GPU sharpness + selection + frame writes
|
||||
masks.rs stage 1: YOLO segmentation (ONNX Runtime / MIGraphX)
|
||||
cubemap.rs stage 2: GPU equirect → cubemap
|
||||
rig.rs stage 3: rig config + nested tree
|
||||
video/
|
||||
mod.rs ffmpeg/ffprobe CLI decode → raw RGBA frames
|
||||
gui.rs egui+wgpu GUI (feature `gui`)
|
||||
offline_check/ pure-std fallback build of the rig core (no cargo/network)
|
||||
```
|
||||
|
||||
## How a run flows
|
||||
|
||||
1. **CLI** parses the command; `all` builds a `GpuContext` once and runs each
|
||||
stage in order, sharing the device/queue.
|
||||
2. **Extract** pipes decoded frames from ffmpeg to the GPU `sharpness` kernel,
|
||||
reads back tiny per-workgroup metrics, filters by quality, selects frames,
|
||||
and writes them out.
|
||||
3. **Cubemap** uploads each equirect frame and dispatches the
|
||||
`equirect_to_cubemap` kernel once per face, reading back square faces.
|
||||
4. **Masks** loads `yolo11x-seg.onnx` into ONNX Runtime (with the MIGraphX EP
|
||||
when requested), runs each cube face through the network, and synthesizes a
|
||||
binary mask (see [GPU](GPU.md)).
|
||||
5. **Rig** scans images + masks, writes `rig_config.json`, and links the nested
|
||||
COLMAP tree (symlink/hardlink/copy).
|
||||
|
||||
## Why Rust + WebGPU
|
||||
|
||||
- **Parallelism**: per-pixel/per-frame work maps naturally onto a compute
|
||||
shader; `rayon` covers CPU-side orchestration.
|
||||
- **Cross-vendor GPU**: one WGSL codebase → Vulkan / Metal / DX12 / WebGPU,
|
||||
instead of per-vendor libraries.
|
||||
- **Safety**: Rust's ownership model + wgpu's resource lifetimes avoid
|
||||
use-after-free and data races.
|
||||
- **Distribution**: a single binary; no Python environment.
|
||||
|
||||
Video **decode** stays on the CPU through ffmpeg because codecs aren't
|
||||
something you hand-roll on a bindless GPU layer; everything after decode is GPU.
|
||||
|
||||
## Feature flags
|
||||
|
||||
| Feature | Effect |
|
||||
| --- | --- |
|
||||
| `masks` (default) | enable ONNX Runtime YOLO masks (`ort` + `ndarray`, `load-dynamic`, `migraphx`) |
|
||||
| `gui` | build the native egui+wgpu GUI into the same binary |
|
||||
|
||||
`cargo build --no-default-features` gives a light build with no ONNX Runtime.
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
# GPU
|
||||
|
||||
This project uses **WebGPU** via the `wgpu` crate, so the same code runs on
|
||||
Vulkan, Metal, DX12, and WebGPU/wasm.
|
||||
|
||||
## GPU selection (all stages)
|
||||
|
||||
By default the pipeline asks wgpu for the **discrete** (high-performance)
|
||||
adapter and logs which one it picked:
|
||||
|
||||
```
|
||||
[gpu] using AMD Radeon RX 9070 XT (RADV GFX1201) (DiscreteGpu) via Vulkan
|
||||
```
|
||||
|
||||
If you see an integrated GPU or a software (llvmpipe) adapter, force the right
|
||||
backend/device. The `request_adapter` fallback enumerates adapters and prefers
|
||||
a discrete GPU. On dual-GPU laptops set `WGPU_POWER_PREF` / run under the
|
||||
discrete-GPU driver profile if wgpu still picks the iGPU.
|
||||
|
||||
## YOLO masks on the GPU (ONNX Runtime → MIGraphX)
|
||||
|
||||
The masks stage runs YOLO through **ONNX Runtime**, loaded **dynamically from
|
||||
the system libonnxruntime** (`ORT_DYLIB_PATH`) rather than a downloaded
|
||||
CPU-only build. On AMD/ROCm systems that system lib is built with the
|
||||
**MIGraphX** execution provider, so:
|
||||
|
||||
```bash
|
||||
# GPU inference on an AMD GPU
|
||||
pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280
|
||||
|
||||
# CPU (no ROCm/MIGraphX runtime needed)
|
||||
pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
|
||||
--mask-model yolo11x-seg.onnx --mask-device cpu
|
||||
```
|
||||
|
||||
- If the GPU EP can't be registered, the stage **falls back to CPU** and warns.
|
||||
- The binary is built with `--features masks` by default.
|
||||
- On your system the GPU-enabled `libonnxruntime` must be findable — the build
|
||||
uses `load-dynamic`, so it `dlopen`s `libonnxruntime.so` (or whatever
|
||||
`ORT_DYLIB_PATH` points at) at runtime, and ROCm must be installed.
|
||||
|
||||
### Compiled-model cache (important for first run)
|
||||
|
||||
MIGraphX compiles the ONNX graph to GPU kernels on the **first run**, which is
|
||||
slow (minutes for a large model like `yolo11x-seg` on a new GPU). It caches the
|
||||
compiled `.mxr` and loads it on later runs.
|
||||
|
||||
The pipeline **auto-sets `ORT_MIGRAPHX_CACHE_PATH`** to
|
||||
`~/.cache/pipeline_360/migraphx` (override with `--mask-ep-cache <dir>`), so:
|
||||
|
||||
- first run with `--mask-device rocm` → compiles + saves (slow),
|
||||
- every later run → loads the cached model in seconds.
|
||||
|
||||
### Env vars (useful)
|
||||
|
||||
| Variable | Meaning |
|
||||
| --- | --- |
|
||||
| `ORT_DYLIB_PATH` | path to the libonnxruntime to dlopen at runtime |
|
||||
| `ORT_MIGRAPHX_CACHE_PATH` | MIGraphX compiled-model cache dir (auto-set) |
|
||||
| `ORT_MIGRAPHX_MODEL_CACHE_PATH` | alias some MIGraphX builds read |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`[gpu] using llvmpipe / iGPU`** — set `WGPU_POWER_PREF=high` or switch to
|
||||
the discrete GPU.
|
||||
- **"not a segmentation model"** — you exported the detection model; export the
|
||||
**seg** one (`yolo11x-seg.pt`), not `yolo11x.onnx`.
|
||||
- **MIGraphX crash at inference with an empty `.mxr` path** — the cache env var
|
||||
was unset. The pipeline sets it automatically; if you run via an external
|
||||
launcher that strips env vars, set `ORT_MIGRAPHX_CACHE_PATH` yourself.
|
||||
- **First run "very slow"** — that's the one-time MIGraphX kernel compilation.
|
||||
Let it finish; later runs are fast.
|
||||
- **CPU-only ONNX Runtime** — if you overrode `ORT_DYLIB_PATH` to a CPU build,
|
||||
the MIGraphX EP won't be available and it falls back to CPU.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Getting Started
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A **Rust toolchain** (`cargo`, `rustc`).
|
||||
- **`ffmpeg` / `ffprobe`** on `PATH` — video decode shells out to the CLI and
|
||||
pipes raw frames; no native FFmpeg bindings or dev headers are needed.
|
||||
- A GPU with a **WebGPU backend** — Vulkan (Linux/Windows), Metal (macOS),
|
||||
DX12 (Windows), or WebGPU/wasm. On Linux, RADV (Mesa) or AMDVLK works.
|
||||
- For the **masks** stage: an **ONNX** export of a YOLO segmentation model
|
||||
(e.g. `yolo11x-seg.onnx`). The `.pt` is not loadable in Rust:
|
||||
|
||||
```bash
|
||||
yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
Masks are enabled by default, so a plain build is the full pipeline:
|
||||
|
||||
```bash
|
||||
cargo build
|
||||
```
|
||||
|
||||
Lighter build without ONNX Runtime (no masks stage):
|
||||
|
||||
```bash
|
||||
cargo build --no-default-features
|
||||
```
|
||||
|
||||
Build with the native GUI:
|
||||
|
||||
```bash
|
||||
cargo build --features gui
|
||||
```
|
||||
|
||||
## First run (sanity check)
|
||||
|
||||
```bash
|
||||
cargo run -- rig --self-test # 6 check groups; should pass
|
||||
```
|
||||
|
||||
## First real run
|
||||
|
||||
```bash
|
||||
cargo run -- all --project-dir out --video day1.mp4 \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280
|
||||
```
|
||||
|
||||
- `--mask-device rocm` runs YOLO on an AMD GPU (see [GPU](GPU.md)); use `cpu`
|
||||
if you don't have the ROCm/MIGraphX runtime.
|
||||
- The first GPU masks run compiles the model (slow); later runs are fast (see
|
||||
the cache note in [GPU](GPU.md)).
|
||||
@@ -0,0 +1,14 @@
|
||||
# pipeline_360 — Wiki
|
||||
|
||||
Everything beyond the short [README](../README.md), in one place.
|
||||
|
||||
## Contents
|
||||
|
||||
- **[Getting Started](Getting-Started.md)** — prerequisites, build, first run
|
||||
- **[Usage](Usage.md)** — CLI reference and the native GUI
|
||||
- **[Pipeline](Pipeline.md)** — stages, output layout, naming conventions
|
||||
- **[GPU](GPU.md)** — WebGPU adapter selection, YOLO-on-GPU (MIGraphX), cache,
|
||||
environment variables, troubleshooting
|
||||
- **[Architecture](Architecture.md)** — source layout and how a run flows
|
||||
|
||||
> The original rewrite plan is still in [`../PLAN.md`](../PLAN.md).
|
||||
@@ -0,0 +1,47 @@
|
||||
# Pipeline
|
||||
|
||||
The rewrite follows the original Python design but reorders the stages so
|
||||
**cubemap runs before masks**: masks run on the cube-face images directly,
|
||||
which avoids a wasteful equirect→cubemap→equirect round-trip.
|
||||
|
||||
## Output layout
|
||||
|
||||
```
|
||||
out/
|
||||
01_frames/ selected sharpest equirect frames (multi-video → <stem> prefix)
|
||||
02_masks/ (legacy; no longer written by `all`)
|
||||
03_cubemap/
|
||||
images/ {stem}_{FACE}.{ext} (equirect → 6 cube faces)
|
||||
masks/ {stem}_{FACE}.{ext}.png (per-face masks)
|
||||
04_colmap/
|
||||
images/ {FACE}/{stem}.{ext} (nested COLMAP tree)
|
||||
masks/ {FACE}/{stem}.{ext}.png
|
||||
rig_config.json rig cameras / intrinsics
|
||||
```
|
||||
|
||||
`FACE` ∈ `{F, R, B, L, U, D}`. The nested `{FACE}/…` tree is produced via
|
||||
symlink / hardlink / copy (`--rig-link-mode`).
|
||||
|
||||
## Stage details
|
||||
|
||||
1. **Extract** — decodes the video(s) with ffmpeg, computes per-frame GPU
|
||||
sharpness (variance-of-Laplacian), exposure, and contrast, optionally
|
||||
filters out blurry / badly exposed / flat frames, checks inter-frame motion
|
||||
(parallax), then selects the sharpest frame per chunk (or spreads them out).
|
||||
Writes `_sharpness_metadata_<stem>.csv` so the analysis is reused on
|
||||
subsequent runs.
|
||||
2. **Cubemap** — projects each equirect frame to six square cube faces on the
|
||||
GPU (a faithful port of `py360convert.e2c`), batched for throughput.
|
||||
3. **Masks** — runs YOLO11x-seg on each cube face, synthesizes a binary mask
|
||||
(COLMAP convention: **black = excluded**), and writes it next to the face
|
||||
image. See [GPU](GPU.md).
|
||||
4. **Rig** — scans the cubemap images + masks, writes `rig_config.json`
|
||||
(PINHOLE intrinsics + per-face `cam_from_rig_rotation`), and links the
|
||||
nested COLMAP tree.
|
||||
|
||||
## Naming / mask convention
|
||||
|
||||
- Masks are named `<image_filename>.png` (hence `frame_00001_F.png.png`).
|
||||
- In a mask, **black pixels are excluded** (movable objects); white pixels are
|
||||
kept (the static scene used for reconstruction). Masks mirror the image tree
|
||||
so COLMAP attaches them to the right cameras.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Usage
|
||||
|
||||
One binary, `pipeline_360`. Invoked with a **subcommand** it runs the CLI; with
|
||||
**no subcommand** (or `--gui`) it launches the native GUI (requires the `gui`
|
||||
feature).
|
||||
|
||||
## `all` — full pipeline
|
||||
|
||||
Runs extract → cubemap → masks → rig on one GPU device:
|
||||
|
||||
```bash
|
||||
pipeline_360 all --project-dir out --video v1.mp4 [v2.mp4 …] \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280 \
|
||||
--extract-chunk-size 30 --cubemap-face-size 1600 --rig-ref-face F
|
||||
```
|
||||
|
||||
- Multi-video: each source is prefixed with its filename stem in `01_frames`.
|
||||
- `--start-stage extract|cubemap|masks|rig` — resume from a later stage.
|
||||
- `--skip-masks` — skip the masks stage entirely.
|
||||
|
||||
## Individual stages
|
||||
|
||||
```bash
|
||||
# Extract sharpest frames (GPU sharpness + quality filtering)
|
||||
pipeline_360 extract --video v1.mp4 [v2.mp4 …] --output-dir out/01_frames \
|
||||
--min-sharpness 0 --min-exposure 0 --max-exposure 255 \
|
||||
--min-contrast 0 --min-motion 0 --max-motion 1e12 \
|
||||
--selection chunk|spread --target-count 0 --min-gap 0 \
|
||||
--bad-frame-action keep|skip|warn
|
||||
|
||||
# Cubemap projection (equirect → 6 cube faces, GPU)
|
||||
pipeline_360 cubemap --frames-dir 01_frames --output-frames-dir 03_cubemap/images \
|
||||
--face-size 1600 --skip-faces "U,D" --resume --jpeg-quality 95
|
||||
|
||||
# Masks on the cube-face images (GPU via --mask-device rocm)
|
||||
pipeline_360 masks --input-dir 03_cubemap/images --output-dir 03_cubemap/masks \
|
||||
--mask-model yolo11x-seg.onnx --mask-device rocm --mask-imgsz 1280 \
|
||||
--mask-classes person,car,bus,truck,bicycle,motorcycle \
|
||||
--mask-conf 0.35 --mask-dilate 6 --mask-skip-faces "" --mask-empty-if-none
|
||||
|
||||
# Rig config + nested COLMAP tree
|
||||
pipeline_360 rig --images-dir 03_cubemap/images --masks-dir 03_cubemap/masks \
|
||||
--output rig_config.json --link-tree 04_colmap/images \
|
||||
--masks-link-tree 04_colmap/masks --face-size 1600 --link-mode symlink
|
||||
```
|
||||
|
||||
## GUI
|
||||
|
||||
The native egui+wgpu GUI is in the same binary:
|
||||
|
||||
```bash
|
||||
cargo build --features gui
|
||||
./target/debug/pipeline_360 # no subcommand → GUI
|
||||
./target/debug/pipeline_360 --gui # explicit
|
||||
```
|
||||
|
||||
The GUI shells out to the CLI subprocess (the same binary) with
|
||||
`--gui-log-output` and streams progress into a live log panel. It exposes every
|
||||
stage's options as forms.
|
||||
|
||||
> `--mask-projection` and `--mask-face-size` are accepted for back-compat but
|
||||
> are no-ops: masks always run on cube-face images.
|
||||
Reference in New Issue
Block a user