Hand Tracking Lab

In the Quest: Put your controllers aside and make sure hand tracking is enabled. Hold both bare hands where the headset cameras can see them. Move and turn them slowly, then try covering a fingertip.

Lab 003: The purple line is the Quest's actual hand pointing ray (targetRaySpace), not the direction of your index finger. The sign shows its left/right heading and up/down elevation in the room's local coordinates, and its movement in degrees. With your fingers open (at least 70 mm apart), the lab keeps updating the original pointing direction. It holds that starting direction once your fingertips are within 50 mm. “At select” records how far the beam had moved when Quest recognized the pinch; “Peak before” remembers the largest change up to that moment. Values remain visible until a later recognized pinch. The gold line still measures thumb-to-index separation; green means the lab's 20 mm threshold, not Quest recognition. The pale finger-line shows the original index-finger direction.

Move and twist the readout: It starts to your left and stays in the room instead of following your face. Aim a hand's Quest pointing beam at the panel, pinch thumb and index finger to grab, then move your hand to reposition it and rotate your wrist to turn or tilt the sign. Release to leave it in place and at that angle. You can also pinch while touching the panel directly. It resets to its starting position and orientation each time you enter passthrough. Manipulating the panel does not change pinch calculations, although the grab pinch is counted.

Lab 004 · Pick up objects: Three objects appear in front of you. Reach your thumb and index finger near one, or aim the purple Quest beam at it. Use a Quest-recognized select pinch to pick it up. Move your hand to move it; wrist rotation turns it. Release the selection to leave it floating in its new position. The sign is still movable and rotatable, and the complete 25-joint hand skeleton remains visible. The Quest supplies joints, pointer poses and selection events—not a built-in object-pickup result. Our lab detects which object was near your fingers or ray. The readout labels these sources separately. A squeeze event or gripSpace may not be available for a hand; the sign reports what the browser actually exposes. No physical-world collisions or gravity are simulated.

Lab 005.2 · Finger depth masking: The three Lab 004 objects return. Reach your open hand close to an object and curl multiple fingers around it; no thumb-to-index pinch is required. A tracked hand-shaped invisible depth mask hides the parts of virtual objects behind your fingers, letting the real passthrough fingers show through. The object stays in the same palm-cradle placement as Lab 005.1; choose mask width and cradle spacing before entering VR. It then follows your hand and wrist, including rotation. Open your fingers to release it in place. Only nearby objects can be power-grabbed; the purple ray is shown for comparison but does not grab objects in 005. You can still use a regular pinch to move and twist the sign. The sign shows the curl percentage of the index, middle, ring and pinky fingers, whether our lab considers your hand open or gripping, the nearby target and the native Quest select/squeeze events (which may remain at zero). Curl percentages are our calculations from Quest joint poses—not extra Quest-provided gesture labels. Finger bend and size vary, so this first version is meant to help us tune the recognition. The object continues to follow the tracked palm and wrist from 005.1. In 005.2, our renderer draws an invisible 3D shape over the tracked hand before the object: its depth blocks only the pixels of the object that are farther away. Since the layer is transparent, the underlying real-hand passthrough image remains visible. The mask is estimated from tracked joint locations and joint radii; it cannot exactly match the real hand in the passthrough video, but should hide object pixels wherever the mask is closer to the viewer.

Change experiments: use the Quest's built-in left-palm menu gesture to exit immersive mode. Look at your left palm, pinch thumb and index finger until the menu icon fills, then release. The experiment menu returns here. This reserved system gesture is not part of our lab.