kinect-view

A virtual camera synthesized live from a fixed three-camera rig.

kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two webcams, and every frame turns the depth measurements directly into a Gaussian-splat scene — flat, footprint-sized surface splats colored from the cameras, with no training or optimization pass. A WebGPU splat renderer then draws that scene from a virtual camera positioned anywhere between the physical cameras, producing a view that no single real camera could take.

Parts of the subject the cameras cannot see (such as the far side of the face) are covered by a head-and-shoulders ellipsoid fitted per frame and recolored from the side cameras.

The rendered output is delivered as:

• an egui window for viewing the virtual-camera image and steering the virtual camera interactively
• a virtual webcam on Linux (via v4l2loopback) that tools such as Teams or Slack pick up as an ordinary camera

Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.

Status

Build-order steps 1–4 are complete. The full pipeline runs on synthetic data through the real depth path: a software Kinect emits raw millimetre depth maps plus RGB (with deterministic sensor noise) and side-camera images, and splatgen culls the background, removes fliers, bilateral-filters the depth, estimates normals, and assembles footprint-sized surfel splats. Occluded regions — the subject's flanks the Kinect never measured — are filled from a head/shoulders ellipsoid fitted to each frame's depth cloud, recolored from the side webcams (the plan's occlusion proxy). Brush renders it live from a virtual camera steered between the rig's side cameras, shown in the egui viewer. The whole chain, GPU rendering included, runs headlessly in tests (software Vulkan), noise included.

Sources are selectable: the synthetic dataset (default), a recorded dataset, or the live rig (--source synthetic|recording|kinect). The live rig capture (Kinect v2 via kinect-one plus two webcams via nokhwa) is implemented but not yet hardware-verified — this container has no USB.

Recorded datasets have an on-disk format (replay::recording): a directory with recording.toml (the rig's cameras, fps, frame count) plus raw frame_NNNN.depth/.rgb/.left/.right files. Real rig recordings can be converted into that shape; synthetic ones are generated for dev/CI.

Virtual webcam (Linux)

The rendered virtual-camera frames can be published as a webcam for Teams/Slack/Chrome via v4l2loopback:

sudo apt install v4l2loopback-dkms
sudo modprobe v4l2loopback video_nr=10 card_label="kinect-view" exclusive_caps=1
kinect-view --loopback-path /dev/video10

exclusive_caps=1 is what makes Chrome-based consumers see the device: it announces output capability until kinect-view attaches, then capture capability only. A failing or removed device disables the sink (with a logged error) rather than stopping the viewer. Off Linux the setting is rejected up front.

• cargo run -p sink — the viewer (needs a GPU, or lavapipe/Vulkan)
• cargo run -p render --release --example bench — pipeline/render frame times (vulkaninfo --summary shows which adapter is in use)
• cargo test --workspace — unit, GPU-integration, and headless UI tests

Configuration

Everything is configurable three ways — CLI arguments, environment variables (KINECT_VIEW_*), and a TOML file — with later layers winning: CLI > environment > file > defaults. Every setting exists in all three.

kinect-view --rig-left-position=-0.6,0,0 --source-fps 24
KINECT_VIEW_RIG_RIGHT_POSITION=0.6,0,0 kinect-view
kinect-view --config-file rig.toml

# rig.toml — each camera has an identifier, position, rotation,
# resolution, and intrinsics.
rig_left_id = "left-cam"
rig_left_position = [-0.6, 0.0, 0.0]
rig_left_look_at = [0.0, 0.0, 2.0] # or rotation_deg / rotation_quat
rig_left_resolution = [640, 480]
rig_left_intrinsics = [554.256, 415.692, 320.0, 240.0]

scene_subject_position = [0.0, 0.0, 2.0]
source_fps = 30.0
background = [0.10, 0.10, 0.12]

source = "recording"            # or "synthetic" / "kinect"
recording_path = "my-recording"  # for --source recording
proxy_enabled = true             # occlusion proxy from the side cameras

kinect-view --help lists every setting with its environment variable.

Workspace layout

Crate	Role
rig	Shared types: camera poses and intrinsics, the virtual camera, splat batches, frame sources
capture	Kinect v2 RGB + depth (kinect-one) and two webcams (nokhwa), timestamp frame sync
replay	Recorded / synthetic rig datasets, so development and CI run without hardware
splatgen	Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill
render	Brush renderer integration (WebGPU/Burn) and the virtual camera
sink	egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)

See PLAN.md for the full plan.