A site that sees you: on-device depth and gesture control in the browser, zero build

A site that sees you: on-device depth and hand tracking in the browser, zero build

A quick word about the format, because it's unusual. I'm a product designer; the code was written by Claude Code. For each of the four decisions below I explain why the product needed it. How it works is explained by the one who built it: the "how" blocks are his words. That's more honest than me pretending to understand workers.

The project itself: a site (doubling as my portfolio) where the webcam is the primary input. A field of particles assembles into a volumetric reflection of the visitor, sections open with a bare hand in the air, photos are taken with a fist. No backend. No build step. Hosted on GitHub Pages.

Context for those outside the field: a webcam is a weak input source — it doesn't measure depth, loses hands, jitters. But today's libraries already stretch that weak signal into a working interface. This project is about how exactly.

• Demo: https://dimbo-design.github.io/dimbo_heavy-cv (desktop + webcam)
• Code: https://github.com/dimbo-design/dimbo_heavy-cv

1. Depth from a plain camera — in the browser

Why. The reflection has to be volumetric: a flat silhouette doesn't survive ten seconds of interest. And the promise "your video never leaves the device" has to be true rather than a line in a privacy policy. People allow the camera more willingly when it's verifiable: there is simply no server.

How — Claude:

Depth is pulled from a plain RGB frame by Depth Anything V2 Small (ONNX) via Transformers.js: WebGPU where available, WASM fallback in Safari/Firefox. Two decisions this wouldn't live without. Inference runs in a Web Worker: even on WebGPU the pre- and post-processing eat the main thread, and from a worker the UI never notices the model at all. And adaptive cadence: 20 fps while a person is in frame, 11 fps while they read a chapter, 6 fps in an empty room. The gaps between depth frames are hidden by a shader crossfade, so the low frequency is simply invisible.

The one thing I learned here the hard way: inference is the laptop's thermometer. Whenever something ran hot or slow, the knob to turn was the network's cadence, not the graphics.

2. Hands — on the main thread

Why. Gesture input can't afford perceptible latency: lag between a movement and the response kills trust faster than any bug. The cursor must sit on the hand as if stitched to it.

How — Claude:

MediaPipe Hand Landmarker (plus Pose for the skeleton and head position) lives on the main thread — that's MediaPipe's canonical path: ~5 ms per frame on GPU. No need to spread it across workers; the gesture layer wants the shortest distance to the DOM. From the 21 hand landmarks we derive a compact geometry (openness, pinch, fist, velocities), and the gesture vocabulary works on that, not on raw points. And don't build gestures on the "toward the camera" axis: a single camera measures hand approach only by the hand's size in frame, and that jitters ±35% from a simple palm rotation. Build on what is measured reliably: position, openness, pinch, fist, two hands.

From me: we did try the "toward the camera" axis first. The most intuitive gesture of all — pushing your palm at the screen — had to be cut entirely.

3. One shader for every state

Why. The site is conceived as a single substance: the same particle field that sleeps on the start screen becomes your reflection when you appear, and steps aside when you read a chapter. No swaps, no seams between states. That was a hard design requirement of continuity.

How — Claude:

Everything visible is one point geometry and one ShaderMaterial. The site's states are uniforms of a single shader, not different scene objects:

uMix        — crossfade between two depth maps (as a radial wave)
uCoherence  — drift ↔ form (assembly / dissolve)
uExhale     — the field's breathing
uPulse      — a directed radial accent

One draw call. Consequence: rendering is practically free, the lattice density could be doubled at no visible cost. It's the neural net that heats the laptop, so the thing to tune is inference cadence, not particle count.

4. Zero build

Why. This one I set as a requirement myself: the project must live for years without maintenance. No lock file may rot, no dependency may break the deploy two years from now.

How — Claude:

There is no package.json. ES modules via importmap, static files, any python3 -m http.server for local development. Every dependency — the neural nets included, ~120 MB of them — is vendored into the repo and served from the site's own origin, so no CDN outage, blocked host or model-hub migration can break the site years from now; the CDNs remain as a silent fallback. GitHub Pages plus .nojekyll is the whole pipeline.

Bonus: gesture regression when CI has no hands

Why. Every threshold tweak fixed one gesture and quietly broke a neighbor. After the third such loop it was clear the vocabulary wouldn't hold without tests.

How — Claude:

Two levels. Replay: the site has a raw hand-motion recorder (15 Hz, ⌥R in debug mode); the owner's real sessions, annotated with what he actually meant, became fixtures: "this trace must produce exactly one step back and zero pinches." And the browser level: puppeteer feeds the site a programmatically drawn depth map of a "person" plus synthetic hands, and runs 22 checks — from model load to opening a chapter by pointing. If your input is sensors rather than clicks, recording and replaying real input is the cheapest test infrastructure there is; build it on day one.

From me: I regret our recorder didn't exist from day one. Before it, validation ran on my retellings of "it twitched the wrong way".

Field notes after release

Three cases from other people's machines:

• An easter egg drew a symbol with fillText('♥'). Fixel has no such glyph, and Yandex Browser (same Chromium, same macOS) substituted a color emoji fallback that turned into mush in the luminance map. Cured with a vector path on the canvas. Font fallback is cross-browser work too.
• People outside tech couldn't find where to point. A cursor dot returned to the main screen, and a ghost-hand hint appeared for a couple of key gestures: when a visitor stalls, a recording of a real movement plays over the page — the same depth frames the built-in recorder writes, just in its own silvery palette. It dies after the first successful gesture and never bothers those who have learned.
• And an infrastructure one: for some visitors the neural nets never loaded at all — huggingface and the CDNs are unreachable from some networks. Everything is now vendored into the repo and served from the site's own origin, with the CDNs as a silent fallback.

Try it

https://dimbo-design.github.io/dimbo_heavy-cv/ — desktop, webcam, Chromium for WebGPU (Safari/Firefox run on WASM, they just "think" slower). The README has the full gesture vocabulary and a couple of URL params for experimenting with field density on your machine. There are easter-egg gestures too — neither in the README nor in this article.

The full field diary of the gesture vocabulary, pitfalls included, is in my Habr article (RU): https://habr.com/ru/articles/1058358/