SHA256
82 lines
3.2 KiB
Markdown
82 lines
3.2 KiB
Markdown
# Warning
|
|
**This project is fully vibecoded and behind no line in this code is human soul.**
|
|
|
|
# pipeline_360
|
|
|
|
Convert equirectangular 360° videos into a **COLMAP-ready cubemap camera-rig
|
|
dataset** — sharpest-frame extraction, movable-object masks, an
|
|
equirect→cubemap projection, and a `rig_config.json` + nested file tree.
|
|
|
|
This is the **Rust + WebGPU rewrite** of the original Python pipeline (now in
|
|
[`../python`](../python), kept as legacy reference).
|
|
|
|
> ⚡ **Status: fully vibecoded, and the rewrite is done.** Everything described
|
|
> here is implemented and running on real 8K test footage — GPU sharpness,
|
|
> YOLO segmentation masks on the GPU, cubemap projection, and rig generation all
|
|
> work end-to-end. It should just work; if something doesn't, the
|
|
> [Wiki](wiki/Home.md) and issue tracker are the place to look.
|
|
|
|
## What it does
|
|
|
|
```
|
|
video(s) → 01_frames (sharpest, quality-filtered)
|
|
→ 03_cubemap/images (equirect → 6 cube faces, on GPU)
|
|
→ 03_cubemap/masks (YOLO11x-seg, on GPU)
|
|
→ 04_colmap/{images,masks,rig_config.json} (nested tree for COLMAP)
|
|
```
|
|
|
|
Everything heavy is on the **GPU via WebGPU** (`wgpu`) — Vulkan / Metal / DX12,
|
|
so it's vendor-neutral. YOLO masks use the **system ONNX Runtime** with the AMD
|
|
MIGraphX provider for GPU inference (falls back to CPU). A native **egui GUI**
|
|
ships in the same binary.
|
|
|
|
## Build & run
|
|
|
|
```bash
|
|
cargo build # full pipeline (masks on by default)
|
|
cargo run -- rig --self-test # quick sanity check
|
|
|
|
# single command, all four stages (cube faces → masks on the GPU):
|
|
cargo run -- all --project-dir out --video vid1.mp4 [vid2.mp4 …] \
|
|
--mask-model yolo11x-seg.onnx --mask-device rocm
|
|
|
|
# native GUI (one binary, --features gui):
|
|
cargo build --features gui && ./target/debug/pipeline_360
|
|
```
|
|
|
|
Prerequisites: a Rust toolchain, `ffmpeg`/`ffprobe` on `PATH`, a GPU with a
|
|
WebGPU backend, and an **ONNX** export of the segmentation model
|
|
(`yolo export model=yolo11x-seg.pt format=onnx imgsz=1280 dynamic=True`).
|
|
|
|
## Features
|
|
|
|
- **Extract** — GPU sharpness/exposure/contrast scores + quality filtering,
|
|
parallax (`--min-motion`) and bad-frame rejection, chunk or spread selection,
|
|
multi-video, batched.
|
|
- **Masks** — YOLO11x-seg on the cube-face images (no wasteful round-trip),
|
|
COLMAP convention (black = excluded), GPU inference with auto cached
|
|
compiled model (first run compiles, later runs are fast).
|
|
- **Cubemap** — faithful port of `py360convert.e2c`, batched, skip/resume.
|
|
- **Rig** — `rig_config.json` + nested `{FACE}/{stem}` tree, symlink/hardlink/copy.
|
|
- **GUI** — native egui+wgpu, shells out to the CLI for the heavy work.
|
|
|
|
## Project structure
|
|
|
|
```
|
|
Cargo.toml, main.rs entry point, CLI, `all` orchestration
|
|
src/gpu/ GpuContext + WGSL shaders
|
|
src/stages/ extract, masks, cubemap, rig
|
|
src/gui.rs egui+wgpu GUI (feature `gui`)
|
|
wiki/ the documentation (setup, usage, GPU, troubleshooting)
|
|
```
|
|
|
|
## License
|
|
|
|
[GPL-3.0-or-later](LICENSE) — see [`LICENSE`](LICENSE).
|
|
|
|
## More
|
|
|
|
- **Wiki** → [`wiki/Home.md`](wiki/Home.md) — setup, CLI reference, pipeline
|
|
internals, GPU/MIGraphX setup, troubleshooting.
|
|
- **Design notes** → [`PLAN.md`](PLAN.md) — the original rewrite plan.
|