3sixty2cubemap/wiki/Pipeline.md

Pipeline

The rewrite follows the original Python design but reorders the stages so cubemap runs before masks: masks run on the cube-face images directly, which avoids a wasteful equirect→cubemap→equirect round-trip.

Output layout

out/
  01_frames/                    selected sharpest equirect frames (multi-video → <stem> prefix)
  02_masks/                     (legacy; no longer written by `all`)
  03_cubemap/
    images/                     {stem}_{FACE}.{ext}               (equirect → 6 cube faces)
    masks/                      {stem}_{FACE}.{ext}.png           (per-face masks)
  04_colmap/
    images/                     {FACE}/{stem}.{ext}               (nested COLMAP tree)
    masks/                      {FACE}/{stem}.{ext}.png
    rig_config.json             rig cameras / intrinsics

FACE ∈ {F, R, B, L, U, D}. The nested {FACE}/… tree is produced via symlink / hardlink / copy (--rig-link-mode).

Stage details

1. Extract — decodes the video(s) with ffmpeg, computes per-frame GPU sharpness (variance-of-Laplacian), exposure, and contrast, optionally filters out blurry / badly exposed / flat frames, checks inter-frame motion (parallax), then selects the sharpest frame per chunk (or spreads them out). Writes _sharpness_metadata_<stem>.csv so the analysis is reused on subsequent runs.
2. Cubemap — projects each equirect frame to six square cube faces on the GPU (a faithful port of py360convert.e2c), batched for throughput.
3. Masks — runs YOLO11x-seg on each cube face, synthesizes a binary mask (COLMAP convention: black = excluded), and writes it next to the face image. See GPU.
4. Rig — scans the cubemap images + masks, writes rig_config.json (PINHOLE intrinsics + per-face cam_from_rig_rotation), and links the nested COLMAP tree.

Naming / mask convention

• Masks are named <image_filename>.png (hence frame_00001_F.png.png).
• In a mask, black pixels are excluded (movable objects); white pixels are kept (the static scene used for reconstruction). Masks mirror the image tree so COLMAP attaches them to the right cameras.