Commit: 188a556

Scaffold kinect-view workspace

GLM Agent committed on 2026-10-25 at 06:47

Rust workspace with the five crates from the plan: capture, replay,
splatgen, render, sink. PLAN.md holds the full plan (pipeline, build
order, risks); README.md describes the project goals. Dependency
choices are centralized in workspace.dependencies: kinect-one and
brush-render as git deps, glam pinned to Brush's 0.30 for type
interop, nokhwa/egui/v4l2r for capture, UI, and the loopback sink.

Browse files at this commit

.gitignore +6 -0

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..f34095a
@@ -0,0 +1,6 @@
# Rust build artifacts
/target
# Editor / OS noise
*.swp
.DS_Store

Cargo.lock +23 -0

diff --git a/Cargo.lock b/Cargo.lock
new file mode 100644
index 0000000..8af2de2
@@ -0,0 +1,23 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "capture"
version = "0.1.0"
[[package]]
name = "render"
version = "0.1.0"
[[package]]
name = "replay"
version = "0.1.0"
[[package]]
name = "sink"
version = "0.1.0"
[[package]]
name = "splatgen"
version = "0.1.0"

Cargo.toml +59 -0

diff --git a/Cargo.toml b/Cargo.toml
new file mode 100644
index 0000000..369a752
@@ -0,0 +1,59 @@
[workspace]
resolver = "3"
members = [
    "capture",
    "replay",
    "splatgen",
    "render",
    "sink",
]
# The workspace root is the `kinect-view/` tree from PLAN.md.
[workspace.package]
version = "0.1.0"
edition = "2024"
[workspace.dependencies]
# Internal crates.
capture = { path = "capture" }
replay = { path = "replay" }
splatgen = { path = "splatgen" }
render = { path = "render" }
sink = { path = "sink" }
# Math. glam is pinned to match Brush's workspace: brush-render's public API
# speaks glam, and a different minor would compile a second copy of the
# crate and break type interop.
glam = { version = "0.30", features = ["serde"] }
bytemuck = "1.20"
# Capture.
kinect-one = { git = "https://github.com/Aytixel/kinect-one" }
nokhwa = "0.10"
# Rendering: Brush's renderer as a library (not published to crates.io; it
# lives under crates/brush-render in the Brush repo).
brush-render = { git = "https://github.com/ArthurBrussee/brush" }
# UI / sinks.
eframe = "0.37"
egui = "0.37"
pollster = "1"
# v4l2loopback output (Linux only). Fallback candidate if this proves
# awkward for output queues: edgefirst-v4l2.
v4l2r = "0.0.8"
# Serialization / data for replay datasets.
serde = { version = "1", features = ["derive"] }
image = "0.25"
# Errors / logging (Brush uses tracing + log).
anyhow = "1"
thiserror = "2"
tracing = "0.1"
tracing-subscriber = "0.3"
log = "0.4"
# Async plumbing (kinect-one's API is tokio-based).
tokio = { version = "1", features = ["macros", "rt", "rt-multi-thread", "sync", "time"] }

PLAN.md +96 -0

diff --git a/PLAN.md b/PLAN.md
new file mode 100644
index 0000000..7e6349b
@@ -0,0 +1,96 @@
# Plan — kinect-view
Working plan for the project. The build order doubles as a rough roadmap:
check items off as they land.
## Goal
A Rust program that captures a fixed 3-camera rig, constructs a live
Gaussian-splat scene each frame, and renders it from a virtual camera
positioned anywhere between the physical cameras.
Output:
1. an egui UI for viewing and steering the virtual camera
2. a virtual webcam for Teams/Slack on Linux
Runs on Windows/ARM and Ubuntu 24/x86.
## Core design
No 3DGS training. Per-frame construction of the Gaussian field (surfel
splatting), handed to Brush's renderer as a library.
Smoothness comes from footprint-sized anisotropic splats:
- two tangential scales = depth × pixel_size / focal_length
- flat disk aligned to the depth-map normal
- few-mm thickness along the normal
- opacity ~0.9–0.99
- front-to-back alpha blend
**Occluded far side of the face — the MVP uses an ellipsoid proxy:** fit a
head/shoulders ellipsoid per frame to the Kinect head point cluster;
side-camera pixels are projected onto the proxy for position and recolored
from the side camera.
Head mirroring is a future upgrade — kinect-one/libfreenect2 provide no head
tracking (verified against the crate source: RGB-D only), so there's no
head-orientation input to drive a symmetry plane. If mirroring is added
later, it needs a head-pose estimator added on top (e.g., a lightweight
head tracker or a face-landmark model), not a driver feature.
Longer-term upgrade path beyond both: monocular depth (`ort`) on side
cameras, metric-anchored to the Kinect head position.
## Pipeline per frame
1. **Capture** — Kinect v2 RGB+depth via kinect-one; two webcams via
   nokhwa. Extrinsics known exactly from the existing offline calibration;
   camera overlap regions remain a runtime sanity check.
2. **Depth preprocess** — bilateral filter (3–5 px), flier removal,
   background culling → ~50–150k active splats.
3. **Splat construction** — footprint-sized surfels from Kinect depth;
   occluded-region splats from the ellipsoid proxy, recolored from side
   cameras, inflated thickness, lower confidence weight.
4. **Render** — Brush renderer (WebGPU/Burn), fresh Gaussian list per
   frame, 30–60 fps target; virtual camera pose from the UI.
5. **Sinks** — egui UI (virtual-camera control); Linux: v4l2loopback
   virtual webcam. Windows: no built-in sink; capture the UI window via
   OBS yourself.
## Workspace layout
The workspace root of this repository is the `kinect-view/` tree:
```
kinect-view/
├── capture/    # kinect-one, webcams, frame sync
├── replay/     # recorded/synthetic datasets — dev & CI without hardware
├── splatgen/   # depth filter, normals, surfels, ellipsoid proxy fill
├── render/     # Brush integration, virtual camera
└── sink/       # egui UI + v4l2loopback (Linux)
```
## Build order
- [ ] 1. replay + render + UI with synthetic data
- [ ] 2. splatgen on recorded Kinect frames
- [ ] 3. Live Kinect capture + ellipsoid proxy recoloring from side cameras
- [ ] 4. v4l2loopback sink
- [ ] 5. Polish: cross-camera color calibration, WB-lock webcams, seam tuning
## Risks (ranked)
1. **kinect-one on Windows/ARM64** → replay mode first, fork-and-fix.
2. **Ellipsoid-proxy quality at nose/cheek contours** → mirroring (with
   added head-pose estimation) or mono-depth as future upgrades.
3. **Cross-camera color seams** → calibration vs Kinect.
4. **Frame sync across three async sensors** → timestamp-align, tolerate
   1-frame skew.
## Out of scope
- Training/optimization
- 360° reconstruction
- Windows virtual-webcam driver
- Head mirroring in the MVP (future upgrade requiring a head-pose estimator)

README.md +36 -0

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..3b7a5f8
@@ -0,0 +1,36 @@
# kinect-view
A virtual camera synthesized live from a fixed three-camera rig.
kinect-view captures a Microsoft Kinect v2 (RGB + depth) flanked by two
webcams, and every frame turns the depth measurements directly into a
Gaussian-splat scene — flat, footprint-sized surface splats colored from
the cameras, with no training or optimization pass. A WebGPU splat
renderer then draws that scene from a **virtual camera** positioned
anywhere between the physical cameras, producing a view that no single
real camera could take.
Parts of the subject the cameras cannot see (such as the far side of the
face) are covered by a head-and-shoulders ellipsoid fitted per frame and
recolored from the side cameras.
The rendered output is delivered as:
- an **egui window** for viewing the virtual-camera image and steering the
  virtual camera interactively
- a **virtual webcam** on Linux (via v4l2loopback) that tools such as
  Teams or Slack pick up as an ordinary camera
Targets Ubuntu 24.04 (x86-64) and Windows on ARM64.
## Workspace layout
| Crate      | Role |
|------------|------|
| `capture`  | Kinect v2 RGB + depth (`kinect-one`) and two webcams (`nokhwa`), timestamp frame sync |
| `replay`   | Recorded / synthetic rig datasets, so development and CI run without hardware |
| `splatgen` | Depth filtering, normals, footprint-sized surfel splats, ellipsoid-proxy fill |
| `render`   | Brush renderer integration (WebGPU/Burn) and the virtual camera |
| `sink`     | egui viewer/steering UI and the v4l2loopback virtual webcam (Linux) |
See [PLAN.md](PLAN.md) for the full plan.

capture/Cargo.toml +8 -0

diff --git a/capture/Cargo.toml b/capture/Cargo.toml
new file mode 100644
index 0000000..2bb2bd0
@@ -0,0 +1,8 @@
[package]
name = "capture"
description = "Live capture from the 3-camera rig: Kinect v2 RGB + depth, two webcams, frame sync"
version.workspace = true
edition.workspace = true
publish = false
[dependencies]

capture/src/lib.rs +15 -0

diff --git a/capture/src/lib.rs b/capture/src/lib.rs
new file mode 100644
index 0000000..2ddd10f
@@ -0,0 +1,15 @@
//! Live capture from the fixed three-camera rig.
//!
//! Pipeline role (PLAN.md, "Pipeline per frame" — capture):
//!
//! * Kinect v2 RGB + depth via `kinect-one`
//! * two side webcams via `nokhwa`
//! * timestamp alignment and frame sync across the three asynchronous
//!   sensors (tolerate ~1 frame of skew)
//!
//! Camera extrinsics are known exactly from the existing offline
//! calibration; the camera overlap regions are checked at runtime as a
//! sanity check.
//!
//! Live capture must present the same rig-frame interface as `replay`, so
//! that live and recorded/synthetic input are interchangeable downstream.

render/Cargo.toml +8 -0

diff --git a/render/Cargo.toml b/render/Cargo.toml
new file mode 100644
index 0000000..3494be6
@@ -0,0 +1,8 @@
[package]
name = "render"
description = "Brush renderer integration (WebGPU/Burn) and the virtual camera"
version.workspace = true
edition.workspace = true
publish = false
[dependencies]

render/src/lib.rs +7 -0

diff --git a/render/src/lib.rs b/render/src/lib.rs
new file mode 100644
index 0000000..ea3e521
@@ -0,0 +1,7 @@
//! Brush renderer integration and the virtual camera.
//!
//! Pipeline role (PLAN.md, "Pipeline per frame" — render): take the
//! freshly constructed Gaussian list from `splatgen` each frame and hand
//! it to Brush's renderer (WebGPU via Burn), then render it from the
//! virtual camera pose supplied by the UI. Fresh splat list per frame,
//! 30–60 fps target, front-to-back alpha blending.

replay/Cargo.toml +8 -0

diff --git a/replay/Cargo.toml b/replay/Cargo.toml
new file mode 100644
index 0000000..d25f382
@@ -0,0 +1,8 @@
[package]
name = "replay"
description = "Recorded / synthetic rig datasets — development and CI without hardware"
version.workspace = true
edition.workspace = true
publish = false
[dependencies]

replay/src/lib.rs +11 -0

diff --git a/replay/src/lib.rs b/replay/src/lib.rs
new file mode 100644
index 0000000..8a9ab55
@@ -0,0 +1,11 @@
//! Recorded and synthetic rig datasets.
//!
//! Pipeline role: a frame source for development and CI that runs without
//! any hardware. Presents the same rig-frame interface as `capture`, so
//! downstream code cannot tell live and replayed input apart.
//!
//! Sources:
//!
//! * recorded Kinect + webcam streams (for `splatgen` work on real data)
//! * synthetic scenes with known ground truth (for exercising `render`'s
//!   virtual camera and the `sink` UI)

sink/Cargo.toml +12 -0

diff --git a/sink/Cargo.toml b/sink/Cargo.toml
new file mode 100644
index 0000000..423653e
@@ -0,0 +1,12 @@
[package]
name = "sink"
description = "Output sinks: egui viewer/steering UI and the v4l2loopback virtual webcam (Linux)"
version.workspace = true
edition.workspace = true
publish = false
[[bin]]
name = "kinect-view"
path = "src/main.rs"
[dependencies]

sink/src/lib.rs +12 -0

diff --git a/sink/src/lib.rs b/sink/src/lib.rs
new file mode 100644
index 0000000..0c525f5
@@ -0,0 +1,12 @@
//! Output sinks for the rendered virtual-camera view.
//!
//! Pipeline role (PLAN.md, "Pipeline per frame" — sinks):
//!
//! * an egui UI for viewing the virtual-camera image and steering the
//!   virtual camera
//! * on Linux, a v4l2loopback virtual webcam so Teams/Slack can pick the
//!   output up as a regular camera
//! * on Windows there is no built-in sink; capture the UI window via OBS
//!   yourself
//!
//! The `kinect-view` binary in this crate is the application entry point.

sink/src/main.rs +9 -0

diff --git a/sink/src/main.rs b/sink/src/main.rs
new file mode 100644
index 0000000..30481e6
@@ -0,0 +1,9 @@
//! `kinect-view` application entry point (scaffolding).
//!
//! The egui application shell arrives with build-order step 1 in PLAN.md
//! ("replay + render + UI with synthetic data"); this placeholder keeps
//! the workspace runnable from the first commit.
fn main() {
    println!("kinect-view — application shell lands with PLAN.md build-order step 1.");
}

splatgen/Cargo.toml +8 -0

diff --git a/splatgen/Cargo.toml b/splatgen/Cargo.toml
new file mode 100644
index 0000000..71ee08e
@@ -0,0 +1,8 @@
[package]
name = "splatgen"
description = "Per-frame splat construction: depth filter, normals, surfels, ellipsoid proxy fill"
version.workspace = true
edition.workspace = true
publish = false
[dependencies]

splatgen/src/lib.rs +15 -0

diff --git a/splatgen/src/lib.rs b/splatgen/src/lib.rs
new file mode 100644
index 0000000..ed18686
@@ -0,0 +1,15 @@
//! Per-frame Gaussian-splat field construction (surfel splatting).
//!
//! Pipeline role (PLAN.md, "Pipeline per frame" — depth preprocess and
//! splat construction). No 3DGS training: the Gaussian list handed to
//! `render` is constructed fresh every frame.
//!
//! * depth preprocess — bilateral filter (3–5 px), flier removal,
//!   background culling; target ~50–150k active splats
//! * footprint-sized anisotropic surfels from the Kinect depth:
//!   two tangential scales = depth × pixel_size / focal_length, a flat
//!   disk aligned to the depth-map normal, few-mm thickness along the
//!   normal, opacity ~0.9–0.99
//! * occluded-region fill from the head/shoulders ellipsoid proxy,
//!   recolored from the side cameras, with inflated thickness and a
//!   lower confidence weight