Intelligent cameras are an important consideration in the world of AR and VR tracking. Although AI cameras typically track people or object, the computer vision technology available today can also digitally assess the position of a person’s limbs.
The ability to see where a person’s hands are moving in the mixed reality environment is a powerful step forward for new interactions. Now, Ultraleap is using technology purchased from Leap Motion to develop a leading solution in hand-tracking, complete with enterprise-ready solutions that are available on standalone VR headsets and PCs.
The fifth-generation tracking solution, Gemini, aims to unlock a range of enterprise use cases for mixed reality, with accurate tracking of hands in real-time.
How Does Gemini Hand Tracking Work?
The Gemini hand tracking system uses computer vision to create digital hands with all the crucial components of knuckles, fingertips, and palms. Developers can therefore track actions from taps to twists. The wireframe skeletons provided by Ultraleap can easily be attached to realistic or surrealistic hands according to the needs of the app.
The technology effectively analyses hand motion data and converts it into intent – a process that has been difficult to perfect until now. For better precision, researchers have continued to work on wearable devices like gloves and rings.
The preview of Gemini is a significant step forward for the tech landscape, as it will allow for a more frictionless use of mixed reality headsets. Users simply need to place their goggles or glasses with the integrated software on their head, and they can start interacting with digital objects.
The Gemini solution builds on the fourth-gen service from Ultraleap, Orion, which was excellent at tracking single hands, but struggled with simultaneous two-hand tracking – particularly when the hands overlapped. Gemini delivers better tracking across both arms, which could support enterprise workers in a productive AR/VR environment.
Smoother Hand Tracking Experiences
Compared with the previous Orion technology, Gemini offers improved smoothness in hand movements, better pose fidelity, greater robustness, and better hand initialization. This collectively enables virtual arms and hands to appear more rapidly and move accurately without jitter issues.
The tracking system, now substantially rewritten to bolster two-hand interactions and tracking, doesn’t suffer from issues due the physical occlusion of one hand by the other. Developer preview is now only available for Windows 10 PCs with the Leap Motion control or Stereo IR 170 camera. Other platforms will receive support later.
To support PC developers, Ultraleap has introduced a new “screentop” mode which allows the tracking camera to be mounted above a screen. Mixed reality headsets using the Snapdragon XR2 chip from Snapdragon will also be able to leverage the new Gemini techniques, as Qualcomm has been working to bring Ultraleap’s software into the platform directly. This would make the technology more available to developers. Varjo will also be integrating Ultraleap’s tracking technology into VR-3 and XR-3 headsets.
