Plan — kinect-view

Working plan for the project. The build order doubles as a rough roadmap: check items off as they land.

Goal

A Rust program that captures a fixed 3-camera rig, constructs a live Gaussian-splat scene each frame, and renders it from a virtual camera positioned anywhere between the physical cameras.

Output:

1. an egui UI for viewing and steering the virtual camera
2. a virtual webcam for Teams/Slack on Linux

Runs on Windows/ARM and Ubuntu 24/x86.

Core design

No 3DGS training. Per-frame construction of the Gaussian field (surfel splatting), handed to Brush's renderer as a library.

Smoothness comes from footprint-sized anisotropic splats:

• two tangential scales = depth × pixel_size / focal_length
• flat disk aligned to the depth-map normal
• few-mm thickness along the normal
• opacity ~0.9–0.99
• front-to-back alpha blend

The surfel constructor itself is shared (rig::Splat::surfel); splatgen assembles the batch from a depth frame around it (pipeline steps 2–3).

The rig converges on the subject, so at the convergence distance the subject stays framed as the virtual camera moves between the side cameras — zero parallax on the subject, parallax only on the background. Verified end-to-end in step 1; this is the desired virtual-camera look.

Occluded far side of the face — the MVP uses an ellipsoid proxy: fit a head/shoulders ellipsoid per frame to the Kinect head point cluster; side-camera pixels are projected onto the proxy for position and recolored from the side camera.

Head mirroring is a future upgrade — kinect-one/libfreenect2 provide no head tracking (verified against the crate source: RGB-D only), so there's no head-orientation input to drive a symmetry plane. If mirroring is added later, it needs a head-pose estimator added on top (e.g., a lightweight head tracker or a face-landmark model), not a driver feature. Longer-term upgrade path beyond both: monocular depth (ort) on side cameras, metric-anchored to the Kinect head position.

Pipeline per frame

1. Capture — Kinect v2 RGB+depth via kinect-one; two webcams via nokhwa. Extrinsics known exactly from the existing offline calibration; camera overlap regions remain a runtime sanity check.
2. Depth preprocess — bilateral filter (3–5 px), flier removal, background culling → ~50–150k active splats.
3. Splat construction — footprint-sized surfels from Kinect depth; occluded-region splats from the ellipsoid proxy, recolored from side cameras, inflated thickness, lower confidence weight.
4. Render — Brush renderer (WebGPU/Burn), fresh Gaussian list per frame, 30–60 fps target; virtual camera pose from the UI.
5. Sinks — egui UI (virtual-camera control); Linux: v4l2loopback virtual webcam. Windows: no built-in sink; capture the UI window via OBS yourself.

Workspace layout

The workspace root of this repository is the kinect-view/ tree:

kinect-view/
├── rig/        # shared types: poses, intrinsics, surfel construction, frame sources
├── capture/    # kinect-one, webcams, frame sync
├── replay/     # recorded / synthetic depth datasets — dev & CI without hardware
├── splatgen/   # depth preprocess (filter, fliers, culling), normals, batch assembly
├── render/     # Brush integration, virtual camera
└── sink/       # egui UI + v4l2loopback (Linux)

(rig/ was added during build-order step 1: the shared types need a home that none of the other crates depend on awkwardly.)

Testing

Test everything as it is built, headlessly — the container has no display and no Kinect:

• Unit tests live with the code they test (camera math, surfel construction, scene intersection, conversions).
• GPU integration tests render real frames through Brush and assert on image content (where the subject lands, what colour it is), tolerant enough for any conforming driver. Any wgpu-capable adapter works, including software Vulkan (lavapipe) in containers and CI.
• UI tests drive the real egui code path with egui_kittest: widget interactions (play/pause, the steering slider) and a full offscreen paint pass.
• The Kinect v2 integration cannot be tested in the container; replay sources stand in for it until hardware testing happens on a real machine.

Configuration

All settings (the rig — camera identifiers, positions, rotations, intrinsics — plus scene, playback, and view) are configurable via CLI arguments, KINECT_VIEW_* environment variables, and TOML config files, layered CLI > env > file > defaults by clap + clap_config (sink/src/config.rs). New settings for later steps (capture devices, loopback output) join the same arg surface.

Build order

• 1. replay + render + UI with synthetic data
• 2. splatgen on recorded Kinect frames — replay defines the recording format (synthetic recordings for CI); the synthetic source is refactored to emit raw depth frames like a Kinect, so the dev/CI path runs through the real depth preprocessing
• 3. Live Kinect capture + ellipsoid proxy recoloring from side cameras — the capture crate is code-complete but awaits hardware verification (the container has no USB; see Testing)
• 4. v4l2loopback sink
• 5. Polish: cross-camera color calibration, WB-lock webcams, seam tuning

Risks (ranked)

1. kinect-one on Windows/ARM64 → replay mode first, fork-and-fix. kinect-one's isochronous USB support is a Linux-only nusb fork, and nokhwa (webcams) unconditionally vendors mozjpeg (C), so even cross-compiling the capture crate for aarch64-pc-windows-msvc needs a real Windows toolchain. Live capture on Windows is fork-and-fix territory; replay and recordings are the Windows story until then.
2. Ellipsoid-proxy quality at nose/cheek contours → mirroring (with added head-pose estimation) or mono-depth as future upgrades.
3. Cross-camera color seams → calibration vs Kinect.
4. Frame sync across three async sensors → timestamp-align, tolerate 1-frame skew.
5. Brush/burn moving targets → brush-render is pinned to a verified revision, and the lockfile pins burn's branch = "main" resolution to the revision from Brush's own committed lock (see scripts/sync-tracel-lock.awk). burn's main drifting ahead of Brush broke the build once during step 1; re-run that script whenever the pins move, and never cargo update git-sourced packages blindly.

Out of scope

• Training/optimization
• 360° reconstruction
• Windows virtual-webcam driver
• Head mirroring in the MVP (future upgrade requiring a head-pose estimator)