kinect-view

A virtual camera synthesized live from a fixed three-camera rig.

kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two webcams, and every frame turns the depth measurements directly into a Gaussian-splat scene — flat, footprint-sized surface splats colored from the cameras, with no training or optimization pass. A WebGPU splat renderer then draws that scene from a virtual camera positioned anywhere between the physical cameras, producing a view that no single real camera could take.

Parts of the subject the cameras cannot see (such as the far side of the face) are covered by a head-and-shoulders ellipsoid fitted per frame and recolored from the side cameras.

The rendered output is delivered as:

• an egui window for viewing the virtual-camera image and steering the virtual camera interactively
• a virtual webcam on Linux (via v4l2loopback) that tools such as Teams or Slack pick up as an ordinary camera

Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.

Status

Build-order step 1 is complete: the full pipeline runs on synthetic data. A synthetic head-and-shoulders stand-in is raycast per depth pixel and turned into footprint-sized surfel splats — the same construction the real depth path will use — then rendered live by Brush's WebGPU renderer from a virtual camera steered between the rig's side cameras, and shown in the egui viewer. The whole chain, GPU rendering included, runs headlessly in tests (software Vulkan).

• cargo run -p sink — the viewer (needs a GPU, or lavapipe/Vulkan)
• cargo test --workspace — unit, GPU-integration, and headless UI tests

Workspace layout

Crate	Role
rig	Shared types: camera poses and intrinsics, the virtual camera, splat batches, frame sources
capture	Kinect v2 RGB + depth (kinect-one) and two webcams (nokhwa), timestamp frame sync
replay	Recorded / synthetic rig datasets, so development and CI run without hardware
splatgen	Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill
render	Brush renderer integration (WebGPU/Burn) and the virtual camera
sink	egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)

See PLAN.md for the full plan.