kinect-view

A virtual camera synthesized live from a fixed three-camera rig.

kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two webcams, and every frame turns the depth measurements directly into a Gaussian-splat scene — flat, footprint-sized surface splats colored from the cameras, with no training or optimization pass. A WebGPU splat renderer then draws that scene from a virtual camera positioned anywhere between the physical cameras, producing a view that no single real camera could take.

Parts of the subject the cameras cannot see (such as the far side of the face) are covered by a head-and-shoulders ellipsoid fitted per frame and recolored from the side cameras.

The rendered output is delivered as:

• an egui window for viewing the virtual-camera image and steering the virtual camera interactively
• a virtual webcam on Linux (via v4l2loopback) that tools such as Teams or Slack pick up as an ordinary camera

Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.

Status

Build-order steps 1–4 are complete. The full pipeline runs on synthetic data through the real depth path: a software Kinect emits raw millimetre depth maps plus RGB (with deterministic sensor noise) and side-camera images, and splatgen culls the background, removes fliers, bilateral-filters the depth, estimates normals, and assembles footprint-sized surfel splats. Occluded regions — the subject's flanks the Kinect never measured — are filled from a head/shoulders ellipsoid fitted to each frame's depth cloud, recolored from the side webcams (the plan's occlusion proxy). Brush renders it live from a virtual camera steered between the rig's side cameras, shown in the egui viewer. The whole chain, GPU rendering included, runs headlessly in tests (software Vulkan), noise included.

Sources are selectable: the synthetic dataset (default), a recorded dataset, or the live rig (--source synthetic|recording|kinect). The live rig capture — a Kinect v2 through libfreenect2 (via our kinect2-sys bindings) plus two webcams via nokhwa — works on Linux and Windows; the device paths await hardware verification.

Recorded datasets have an on-disk format (replay::recording): a directory with recording.toml (the rig's cameras, fps, frame count) plus raw frame_NNNN.depth/.rgb/.left/.right files. Real rig recordings can be converted into that shape; synthetic ones are generated for dev/CI.

Kinect v2 capture setup (--source kinect)

The Kinect driver is libfreenect2, linked through the kinect2-sys crate. cargo finds it via pkg-config (Linux), or via FREENECT2_DIR (Windows) pointing at a libfreenect2 install prefix.

Windows

1. USB driver (one of):
   • UsbDk — x64 installer, does not interfere with the official Kinect SDK; or
   • libusbK via Zadig: Options → "List All Devices", uncheck "Ignore Hubs or Composite Parents"; select "Xbox NUI Sensor (composite parent)" (VID 045E, PID 02C4/02D8); replace the driver with libusbK ≥ 3.0.7.0.
2. Build libfreenect2 with Visual Studio (the "Desktop development with C++" workload, which the workspace's C dependencies need anyway):

   git clone https://github.com/OpenKinect/libfreenect2 && cd libfreenect2
   # dependencies the CMake build looks for:
   #   libusb   → extract the .7z from https://github.com/libusb/libusb/releases
   #              to depends/libusb
   #   TurboJPEG → run the -vc64.exe installer from
   #              https://sourceforge.net/projects/libjpeg-turbo (default
   #              C:\libjpeg-turbo64, or depends/libjpeg-turbo64, or TURBOJPEG_ROOT)
   cmake -B build -G "Visual Studio 17 2022" -A x64 ^
     -DENABLE_OPENGL=OFF -DENABLE_OPENCL=OFF -DENABLE_CUDA=OFF ^
     -DCMAKE_INSTALL_PREFIX=C:/libfreenect2
   cmake --build build --config RelWithDebInfo --target install
   

   (or vcpkg install libfreenect2 and use the installed tree).
3. Point kinect-view at it and put the runtime DLLs on PATH (or next to kinect-view.exe): freenect2.dll, libusb-1.0.dll, turbojpeg.dll — from the libfreenect2 install's bin):

   setx FREENECT2_DIR C:\libfreenect2
   cargo run -p sink -- --source kinect
   

Linux (Ubuntu 24.04)

sudo apt install libusb-1.0-0-dev libturbojpeg0-dev cmake g++
git clone https://github.com/OpenKinect/libfreenect2 && cd libfreenect2
cmake -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DENABLE_OPENGL=OFF -DENABLE_OPENCL=OFF -DENABLE_CUDA=OFF -DENABLE_VAAPI=OFF \
  -DBUILD_EXAMPLES=OFF -DBUILD_OPENNI2=OFF -DCMAKE_INSTALL_PREFIX=/usr/local
# CMake ≥ 4 needs: -DCMAKE_POLICY_VERSION_MINIMUM=3.5
sudo cmake --install build && sudo ldconfig
# Device access without root:
sudo cp platform/linux/udev/90-kinect2.rules /etc/udev/rules.d/ && sudo udevadm control --reload-rules
cargo run -p sink -- --source kinect

The same steps are what the dev container runs, so the Linux path is known to configure, build, link, and fail cleanly without a device.

Virtual webcam (Linux)

The rendered virtual-camera frames can be published as a webcam for Teams/Slack/Chrome via v4l2loopback:

sudo apt install v4l2loopback-dkms
sudo modprobe v4l2loopback video_nr=10 card_label="kinect-view" exclusive_caps=1
kinect-view --loopback-path /dev/video10

exclusive_caps=1 is what makes Chrome-based consumers see the device: it announces output capability until kinect-view attaches, then capture capability only. A failing or removed device disables the sink (with a logged error) rather than stopping the viewer. Off Linux the setting is rejected up front.

• cargo run -p sink — the viewer (needs a GPU, or lavapipe/Vulkan)
• cargo run -p render --release --example bench — pipeline/render frame times (vulkaninfo --summary shows which adapter is in use)
• cargo test --workspace — unit, GPU-integration, and headless UI tests

Configuration

Everything is configurable three ways — CLI arguments, environment variables (KINECT_VIEW_*), and a TOML file — with later layers winning: CLI > environment > file > defaults. Every setting exists in all three.

kinect-view --rig-left-position=-0.6,0,0 --source-fps 24
KINECT_VIEW_RIG_RIGHT_POSITION=0.6,0,0 kinect-view
kinect-view --config-file rig.toml

# rig.toml — each camera has an identifier, position, rotation,
# resolution, and intrinsics.
rig_left_id = "left-cam"
rig_left_position = [-0.6, 0.0, 0.0]
rig_left_look_at = [0.0, 0.0, 2.0] # or rotation_deg / rotation_quat
rig_left_resolution = [640, 480]
rig_left_intrinsics = [554.256, 415.692, 320.0, 240.0]

scene_subject_position = [0.0, 0.0, 2.0]
source_fps = 30.0
background = [0.10, 0.10, 0.12]

source = "recording"            # or "synthetic" / "kinect"
recording_path = "my-recording"  # for --source recording
proxy_enabled = true             # occlusion proxy from the side cameras

kinect-view --help lists every setting with its environment variable.

Workspace layout

Crate	Role
rig	Shared types: camera poses and intrinsics, the virtual camera, splat batches, frame sources
kinect2-sys	libfreenect2 bindings (C shim, no bindgen) — the Kinect v2 driver
capture	Kinect v2 RGB + depth (libfreenect2) and two webcams (nokhwa), timestamp frame sync
replay	Recorded / synthetic rig datasets, so development and CI run without hardware
splatgen	Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill
render	Brush renderer integration (WebGPU/Burn) and the virtual camera
sink	egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)

See PLAN.md for the full plan.