kinect-view

A virtual camera synthesized live from a fixed three-camera rig.

kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two webcams, and every frame turns the depth measurements directly into a Gaussian-splat scene — flat, footprint-sized surface splats colored from the cameras, with no training or optimization pass. A WebGPU splat renderer then draws that scene from a virtual camera positioned anywhere between the physical cameras, producing a view that no single real camera could take.

Parts of the subject the cameras cannot see (such as the far side of the face) are covered by a head-and-shoulders ellipsoid fitted per frame and recolored from the side cameras.

The rendered output is delivered as:

• an egui window for viewing the virtual-camera image and steering the virtual camera interactively
• a virtual webcam on Linux (via v4l2loopback) that tools such as Teams or Slack pick up as an ordinary camera

Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.

Status

Build-order step 1 is complete: the full pipeline runs on synthetic data. A synthetic head-and-shoulders stand-in is raycast per depth pixel and turned into footprint-sized surfel splats — the same construction the real depth path will use — then rendered live by Brush's WebGPU renderer from a virtual camera steered between the rig's side cameras, and shown in the egui viewer. The whole chain, GPU rendering included, runs headlessly in tests (software Vulkan).

• cargo run -p sink — the viewer (needs a GPU, or lavapipe/Vulkan)
• cargo test --workspace — unit, GPU-integration, and headless UI tests

Configuration

Everything is configurable three ways — CLI arguments, environment variables (KINECT_VIEW_*), and a TOML file — with later layers winning: CLI > environment > file > defaults. Every setting exists in all three.

kinect-view --rig-left-position=-0.6,0,0 --source-fps 24
KINECT_VIEW_RIG_RIGHT_POSITION=0.6,0,0 kinect-view
kinect-view --config-file rig.toml

# rig.toml — each camera has an identifier, position, rotation,
# resolution, and intrinsics.
rig_left_id = "left-cam"
rig_left_position = [-0.6, 0.0, 0.0]
rig_left_look_at = [0.0, 0.0, 2.0] # or rotation_deg / rotation_quat
rig_left_resolution = [640, 480]
rig_left_intrinsics = [554.256, 415.692, 320.0, 240.0]

scene_subject_position = [0.0, 0.0, 2.0]
source_fps = 30.0
background = [0.10, 0.10, 0.12]

kinect-view --help lists every setting with its environment variable.

Workspace layout

Crate	Role
rig	Shared types: camera poses and intrinsics, the virtual camera, splat batches, frame sources
capture	Kinect v2 RGB + depth (kinect-one) and two webcams (nokhwa), timestamp frame sync
replay	Recorded / synthetic rig datasets, so development and CI run without hardware
splatgen	Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill
render	Brush renderer integration (WebGPU/Burn) and the virtual camera
sink	egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)

See PLAN.md for the full plan.