kinect-view

A virtual camera synthesized live from a fixed three-camera rig.

kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two webcams, and every frame turns the depth measurements directly into a Gaussian-splat scene — flat, footprint-sized surface splats colored from the cameras, with no training or optimization pass. A WebGPU splat renderer then draws that scene from a virtual camera positioned anywhere between the physical cameras, producing a view that no single real camera could take.

Parts of the subject the cameras cannot see (such as the far side of the face) are covered by a head-and-shoulders ellipsoid fitted per frame and recolored from the side cameras.

The rendered output is delivered as:

• an egui window for viewing the virtual-camera image and steering the virtual camera interactively
• a virtual webcam on Linux (via v4l2loopback) that tools such as Teams or Slack pick up as an ordinary camera

Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.

Status

Build-order steps 1–2 are complete. The full pipeline runs on synthetic data through the real depth path: a software Kinect emits raw millimetre depth maps plus RGB (with deterministic sensor noise), and splatgen culls the background, removes fliers, bilateral-filters the depth, estimates normals, and assembles footprint-sized surfel splats — which Brush renders live from a virtual camera steered between the rig's side cameras, shown in the egui viewer. The whole chain, GPU rendering included, runs headlessly in tests (software Vulkan), noise included.

Recorded datasets have an on-disk format (replay::recording): a directory with recording.toml (camera, fps, frame count) plus raw frame_NNNN.depth/frame_NNNN.rgb files. Real Kinect recordings can be converted into that shape; synthetic ones are generated for dev/CI.

• cargo run -p sink — the viewer (needs a GPU, or lavapipe/Vulkan)
• cargo test --workspace — unit, GPU-integration, and headless UI tests

Configuration

Everything is configurable three ways — CLI arguments, environment variables (KINECT_VIEW_*), and a TOML file — with later layers winning: CLI > environment > file > defaults. Every setting exists in all three.

kinect-view --rig-left-position=-0.6,0,0 --source-fps 24
KINECT_VIEW_RIG_RIGHT_POSITION=0.6,0,0 kinect-view
kinect-view --config-file rig.toml

# rig.toml — each camera has an identifier, position, rotation,
# resolution, and intrinsics.
rig_left_id = "left-cam"
rig_left_position = [-0.6, 0.0, 0.0]
rig_left_look_at = [0.0, 0.0, 2.0] # or rotation_deg / rotation_quat
rig_left_resolution = [640, 480]
rig_left_intrinsics = [554.256, 415.692, 320.0, 240.0]

scene_subject_position = [0.0, 0.0, 2.0]
source_fps = 30.0
background = [0.10, 0.10, 0.12]

kinect-view --help lists every setting with its environment variable.

Workspace layout

Crate	Role
rig	Shared types: camera poses and intrinsics, the virtual camera, splat batches, frame sources
capture	Kinect v2 RGB + depth (kinect-one) and two webcams (nokhwa), timestamp frame sync
replay	Recorded / synthetic rig datasets, so development and CI run without hardware
splatgen	Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill
render	Brush renderer integration (WebGPU/Burn) and the virtual camera
sink	egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)

See PLAN.md for the full plan.