Plan — kinect-view

Working plan for the project. The build order doubles as a rough roadmap: check items off as they land.

Goal

A Rust program that captures a fixed 3-camera rig, constructs a live Gaussian-splat scene each frame, and renders it from a virtual camera positioned anywhere between the physical cameras.

Output:

1. an egui UI for viewing and steering the virtual camera
2. a virtual webcam for Teams/Slack on Linux

Runs on Windows/ARM and Ubuntu 24/x86.

Core design

No 3DGS training. Per-frame construction of the Gaussian field (surfel splatting), handed to Brush's renderer as a library.

Smoothness comes from footprint-sized anisotropic splats:

• two tangential scales = depth × pixel_size / focal_length
• flat disk aligned to the depth-map normal
• few-mm thickness along the normal
• opacity ~0.9–0.99
• front-to-back alpha blend

The surfel constructor itself is shared (rig::Splat::surfel); splatgen assembles the batch from a depth frame around it (pipeline steps 2–3).

The rig converges on the subject, so at the convergence distance the subject stays framed as the virtual camera moves between the side cameras — zero parallax on the subject, parallax only on the background. Verified end-to-end in step 1; this is the desired virtual-camera look.

Occluded far side of the face — the MVP uses an ellipsoid proxy: fit a head/shoulders ellipsoid per frame to the Kinect head point cluster; side-camera pixels are projected onto the proxy for position and recolored from the side camera.

Head mirroring is a future upgrade — kinect-one/libfreenect2 provide no head tracking (verified against the crate source: RGB-D only), so there's no head-orientation input to drive a symmetry plane. If mirroring is added later, it needs a head-pose estimator added on top (e.g., a lightweight head tracker or a face-landmark model), not a driver feature. Longer-term upgrade path beyond both: monocular depth (ort) on side cameras, metric-anchored to the Kinect head position.

Pipeline per frame

1. Capture — Kinect v2 RGB+depth via kinect-one; two webcams via nokhwa. Extrinsics known exactly from the existing offline calibration; camera overlap regions remain a runtime sanity check.
2. Depth preprocess — bilateral filter (3–5 px), flier removal, background culling → ~50–150k active splats.
3. Splat construction — footprint-sized surfels from Kinect depth; occluded-region splats from the ellipsoid proxy, recolored from side cameras, inflated thickness, lower confidence weight.
4. Render — Brush renderer (WebGPU/Burn), fresh Gaussian list per frame, 30–60 fps target; virtual camera pose from the UI.
5. Sinks — egui UI (virtual-camera control); Linux: v4l2loopback virtual webcam. Windows: no built-in sink; capture the UI window via OBS yourself.

Workspace layout

The workspace root of this repository is the kinect-view/ tree:

kinect-view/
├── rig/        # shared types: poses, intrinsics, surfel construction, frame sources
├── capture/    # kinect-one, webcams, frame sync
├── replay/     # recorded / synthetic depth datasets — dev & CI without hardware
├── splatgen/   # depth preprocess (filter, fliers, culling), normals, batch assembly
├── render/     # Brush integration, virtual camera
└── sink/       # egui UI + v4l2loopback (Linux)

(rig/ was added during build-order step 1: the shared types need a home that none of the other crates depend on awkwardly.)

Testing

Test everything as it is built, headlessly — the container has no display and no Kinect:

• Unit tests live with the code they test (camera math, surfel construction, scene intersection, conversions).
• GPU integration tests render real frames through Brush and assert on image content (where the subject lands, what colour it is), tolerant enough for any conforming driver. Any wgpu-capable adapter works, including software Vulkan (lavapipe) in containers and CI.
• UI tests drive the real egui code path with egui_kittest: widget interactions (play/pause, the steering slider) and a full offscreen paint pass.
• The Kinect v2 integration cannot be tested in the container; replay sources stand in for it until hardware testing happens on a real machine. The same goes for the v4l2loopback device: creating /dev/video* needs the kernel module, so the loopback sink's device writer is exercised on a real machine (the frame-format writer, negotiation, and error handling are unit-tested here, against real ioctl failures where the container can produce them).

Configuration

All settings (the rig — camera identifiers, positions, rotations, intrinsics — plus scene, playback, and view) are configurable via CLI arguments, KINECT_VIEW_* environment variables, and TOML config files, layered CLI > env > file > defaults by clap + clap_config (sink/src/config.rs). New settings for later steps (capture devices, loopback output) join the same arg surface.

Build order

• 1. replay + render + UI with synthetic data
• 2. splatgen on recorded Kinect frames — replay defines the recording format (synthetic recordings for CI); the synthetic source is refactored to emit raw depth frames like a Kinect, so the dev/CI path runs through the real depth preprocessing
• 3. Live Kinect capture + ellipsoid proxy recoloring from side cameras — the capture crate is code-complete but awaits hardware verification (the container has no USB; see Testing)
• 4. v4l2loopback sink — the frame-format writer (RGBA→YUYV, negotiation, error handling, latest-frame handoff) is fully tested; attaching a real /dev/video* device needs the kernel module on a real machine (see Testing)
• 5. Polish: cross-camera color calibration, WB-lock webcams, seam tuning

Risks (ranked)

1. kinect-one on Windows/ARM64 → replay mode first, fork-and-fix. Confirmed on the real Windows machine: kinect-one does not compile off Linux at all — its isochronous USB support is a Linux-only nusb fork (IsoEndpoint is cfg(linux)), plus fork/upstream API drift (busnum). The capture crate's device stack is now Linux-gated (kinect-one and nokhwa; that also drops nokhwa's vendored C from Windows builds); off Linux --source kinect fails with a clear message and the rest of the workspace — viewer, replay, recordings — is verified to build for Windows by cross-checking from the dev container (x86_64-pc-windows-gnu fully green; pure-Rust crates also check for aarch64-pc-windows-msvc). Windows live capture remains fork-and-fix territory: a kinect-one fork would need isochronous-over-WinUSB or the official Kinect SDK as the backend.
2. Ellipsoid-proxy quality at nose/cheek contours → mirroring (with added head-pose estimation) or mono-depth as future upgrades.
3. Cross-camera color seams → calibration vs Kinect.
4. Frame sync across three async sensors → timestamp-align, tolerate 1-frame skew.
5. Brush/burn moving targets → brush-render is pinned to a verified revision, and the lockfile pins burn's branch = "main" resolution to the revision from Brush's own committed lock (see scripts/sync-tracel-lock.awk). burn's main drifting ahead of Brush broke the build once during step 1; re-run that script whenever the pins move, and never cargo update git-sourced packages blindly.

Out of scope

• Training/optimization
• 360° reconstruction
• Windows virtual-webcam driver
• Head mirroring in the MVP (future upgrade requiring a head-pose estimator)