Plan — kinect-view

Working plan for the project. The build order doubles as a rough roadmap: check items off as they land.

Goal

A Rust program that captures a fixed 3-camera rig, constructs a live Gaussian-splat scene each frame, and renders it from a virtual camera positioned anywhere between the physical cameras.

Output:

1. an egui UI for viewing and steering the virtual camera
2. a virtual webcam for Teams/Slack on Linux

Runs on Windows/ARM and Ubuntu 24/x86.

Core design

No 3DGS training. Per-frame construction of the Gaussian field (surfel splatting), handed to Brush's renderer as a library.

Smoothness comes from footprint-sized anisotropic splats:

• two tangential scales = depth × pixel_size / focal_length
• flat disk aligned to the depth-map normal
• few-mm thickness along the normal
• opacity ~0.9–0.99
• front-to-back alpha blend

Occluded far side of the face — the MVP uses an ellipsoid proxy: fit a head/shoulders ellipsoid per frame to the Kinect head point cluster; side-camera pixels are projected onto the proxy for position and recolored from the side camera.

Head mirroring is a future upgrade — kinect-one/libfreenect2 provide no head tracking (verified against the crate source: RGB-D only), so there's no head-orientation input to drive a symmetry plane. If mirroring is added later, it needs a head-pose estimator added on top (e.g., a lightweight head tracker or a face-landmark model), not a driver feature. Longer-term upgrade path beyond both: monocular depth (ort) on side cameras, metric-anchored to the Kinect head position.

Pipeline per frame

1. Capture — Kinect v2 RGB+depth via kinect-one; two webcams via nokhwa. Extrinsics known exactly from the existing offline calibration; camera overlap regions remain a runtime sanity check.
2. Depth preprocess — bilateral filter (3–5 px), flier removal, background culling → ~50–150k active splats.
3. Splat construction — footprint-sized surfels from Kinect depth; occluded-region splats from the ellipsoid proxy, recolored from side cameras, inflated thickness, lower confidence weight.
4. Render — Brush renderer (WebGPU/Burn), fresh Gaussian list per frame, 30–60 fps target; virtual camera pose from the UI.
5. Sinks — egui UI (virtual-camera control); Linux: v4l2loopback virtual webcam. Windows: no built-in sink; capture the UI window via OBS yourself.

Workspace layout

The workspace root of this repository is the kinect-view/ tree:

kinect-view/
├── rig/        # shared types: poses, intrinsics, splat batches, frame sources
├── capture/    # kinect-one, webcams, frame sync
├── replay/     # recorded/synthetic datasets — dev & CI without hardware
├── splatgen/   # depth filter, normals, surfels, ellipsoid proxy fill
├── render/     # Brush integration, virtual camera
└── sink/       # egui UI + v4l2loopback (Linux)

(rig/ was added during build-order step 1: the shared types need a home that none of the other crates depend on awkwardly.)

Testing

Test everything as it is built, headlessly — the container has no display and no Kinect:

• Unit tests live with the code they test (camera math, surfel construction, scene intersection, conversions).
• GPU integration tests render real frames through Brush and assert on image content (where the subject lands, what colour it is), tolerant enough for any conforming driver. Any wgpu-capable adapter works, including software Vulkan (lavapipe) in containers and CI.
• UI tests drive the real egui code path with egui_kittest: widget interactions (play/pause, the steering slider) and a full offscreen paint pass.
• The Kinect v2 integration cannot be tested in the container; replay sources stand in for it until hardware testing happens on a real machine.

Build order

• 1. replay + render + UI with synthetic data
• 2. splatgen on recorded Kinect frames
• 3. Live Kinect capture + ellipsoid proxy recoloring from side cameras
• 4. v4l2loopback sink
• 5. Polish: cross-camera color calibration, WB-lock webcams, seam tuning

Risks (ranked)

1. kinect-one on Windows/ARM64 → replay mode first, fork-and-fix.
2. Ellipsoid-proxy quality at nose/cheek contours → mirroring (with added head-pose estimation) or mono-depth as future upgrades.
3. Cross-camera color seams → calibration vs Kinect.
4. Frame sync across three async sensors → timestamp-align, tolerate 1-frame skew.
5. Brush/burn moving targets → brush-render is pinned to a verified revision, and the lockfile pins burn's branch = "main" resolution to the revision from Brush's own committed lock (see scripts/sync-tracel-lock.awk). burn's main drifting ahead of Brush broke the build once during step 1; re-run that script whenever the pins move, and never cargo update git-sourced packages blindly.

Out of scope

• Training/optimization
• 360° reconstruction
• Windows virtual-webcam driver
• Head mirroring in the MVP (future upgrade requiring a head-pose estimator)