The special thing about XR is its spatial element. Thanks to fully immersive 3D spaces, users can interact with digital content from a new dimension. With RT3D interactable content, such as 3D design documentation, users can ditch cumbersome controller headsets today and use a built-in tracking camera or device that allows users to interact with virtual content using hand gestures or gaze control.

Eye and tracking bring several new use cases. With the technology, users no longer need familiarity with gaming-esc controllers, instead using more natural motions. Apple Vision uses hand tracking to introduce gesture controls such as swipes and pinches, similar to a phone screen.

However, the technology is still emerging, and hand and eye tracking has only just become an enticing feature to include in a major upcoming AR/VR/MR headset in 2024. However, hurdles in hardware, software, and adoption still exist to overcome for the sector’s leading vendors to provide ubiquitous workplace tracking services.

To explore the topic, XR Today gathered exclusive insight into hand and eye-tracking technology from industry experts representing Looking Glass, SpectreXR, and Finsa.

What Tech is Needed to Support Tracking?

Jose G. Pichel, Technical Department of Engineering, Finsa:

To fully harness the potential of hand and eye tracking technologies in enterprise environments, headsets need several key hardware advancements:

• High-Resolution and High-Speed Cameras:

Hand Tracking: High-resolution cameras with high frame rates (90-120 fps) for precise gesture detection and smooth tracking.

Eye Tracking: cameras or sensors with high resolution and frame rates for accurate eye movement capture.

• Depth Sensors and LIDAR:

Depth Sensors: 3D environment mapping for accurate hand tracking.

LIDAR: Precise distance measurements for enhanced AR and MR interactions.

• Powerful Processors and GPUs:

Processors: Multi-core processors for real-time data processing.

GPUs: High-capacity GPUs for rendering complex graphics and managing data streams.

• AI and Machine Learning Algorithms:

Gesture and Movement Recognition: Improve tracking accuracy and adapt to different users.

Prediction and Error Correction: Real-time error correction for smoother experiences.

• Advanced Connectivity:

Low Latency Wireless: Wi-Fi or Bluetooth for stable, fast connections.

Broadband Interface: Efficient data transmission.

• Ergonomic Design:

Lightweight and Comfortable: Reduces fatigue for prolonged use.

Custom Fit: Adjustable for different head shapes and sizes, wearable devices, and comfortables.

The best thing would be to have a new smart glasses design with 6DOF and tracking capacity.

• High-Resolution Displays and Wide FOV:

High Resolution: Sharp, detailed images.

Wide Field of View: Enhanced immersion.

• Biometric Sensors:

Heart Rate and Stress Sensors: Personalize experiences and improve user well-being.

New haptic devices and sensors are needed to make the experience more real.

• Smart Glasses:

Smart Glasses without passthrough cameras with tracking capacity and graphic capacity on the front. (AR) Augmented Reality vs (aR) Assisted Reality.

Interacting with Virtual Content

Looking Glass:

Interfacing with digital objects makes the interactive experience feel more natural.

Eye- and hand-tracking are leveraged to communicate and engage with 3D information, unlocking spatial reasoning and immersion.

In VR, the benefit of that tracking is primarily 1:1, which means that generally, only one person benefits from the experience. With group viewable 3D, the interaction is one.

One core advancement is using hand- and eye-tracking alongside technologies like Looking Glass displays to deliver many interactive experiences such as brand and retail activations.

Unlocking Workplace Potential

Ivan Rajković, SpectreXR, CEO:

To really bring hand and eye-tracking technologies to their full potential in enterprise environments, we’re going to need some hardware upgrades.

Imagine a surgeon using VR to practice a complex procedure. For that to work smoothly, the sensors in the headset need to capture eye and hand movements with incredible precision—down to the millimetre.

Right now, we’re not quite there. The accuracy and speed of these sensors are crucial. If they miss even the smallest detail, the whole experience can labeled as not good enough.

But it’s not just about accuracy. The real magic happens when everything feels seamless, with no noticeable lag. This is especially important in fields like healthcare or defense, where even a split-second delay can break the immersion and make the tech feel less reliable.

To get there, we need to beef up processing power and cut down on latency. The idea is to have the tech work so smoothly that you forget it’s even there—everything just flows naturally, like in the real world.

So, while the potential of hand-tracking and eye-tracking in business settings is huge, we’ve got to push the boundaries of both hardware and software to make it all work seamlessly.

The Role of Modularity

Ivan Rajković:

Modularity is key when integrating hand and eye-tracking tools into enterprise environments because it allows for unparalleled flexibility and scalability.

In a business landscape where needs can vary widely—from healthcare simulations requiring intricate hand-tracking to retail environments focused on simple gesture-based interactions—having a modular system means you can tailor the technology to fit the specific demands of each scenario. But it’s not just about customization.

Modularity also plays a crucial role in future-proofing your technology. As your company’s needs grow or change, a modular system makes it easy to add new components or adapt existing ones without overhauling the entire setup. This adaptability is essential for staying competitive in an industry that’s constantly evolving.

These are the reasons why we focused so heavily on modularity while developing OctoXR. We wanted to create a tool that not only meets your needs today but can grow and adapt alongside your business, ensuring long-term value and success.

Looking Glass:

Because hand-tracking technology is new and has a wide range of applications, being able to select from a large array of potential interaction schemas is critical to delivering use-case-specific interaction to end users.

Essential Workplace Factors

Jose G. Pichel:

Modularity is essential for integrating hand and eye-tracking tools in enterprise environments. Here’s why:

• Ease of Upgrades and Maintenance

Simplified Upgrades: Replace or upgrade individual components without overhauling the entire system.

Reduced Downtime: Maintain specific modules without affecting the whole system, minimizing disruptions.

• Scalability

Adaptive Growth: Start with a basic setup and expand by adding new modules as needed.

Flexible Integration: Integrate additional functionalities, such as new sensors or software, without major reconfiguration.

• Customization and Flexibility

Tailored Solutions: Customize configurations to meet specific requirements, optimizing performance.

Interoperability: Ensure compatibility with a wide range of systems and devices.

• Cost Efficiency

Lower Initial Investment: Invest in necessary modules initially and add more as required, reducing upfront costs.

Operational Efficiency: Efficiently manage and upgrade individual modules, leading to long-term savings.

• Innovation and Competitiveness

Rapid Adoption: Quickly integrate new technologies without a complete system redesign.

Testing and Adaptation: Pilot new modules in controlled settings before full-scale implementation.

• Practical Examples and Success Cases.

VR/AR Applications: Upgrade tracking capabilities without replacing entire headsets.

Healthcare: Incorporate new biometric sensors or advanced tracking algorithms.

Manufacturing: Tailor tracking systems for different processes, improving efficiency.

Modularity facilitates easy upgrades, scalability, customization, cost efficiency, and rapid technology adoption, enhancing performance and competitiveness in enterprise environments.

The Middleware Building Blocks

Jose G. Pichel:

The middleware for scaling hand and eye-tracking tools includes several key building blocks:

• APIs (Application Programming Interfaces)

Communication APIs: Facilitate data exchange between tracking devices and software applications.

Integration APIs: Enable seamless integration with enterprise systems like CRM and ERP.

• Device Management

Device Drivers: Ensure proper communication and control of tracking hardware.

Device Administration: Tools for configuring, monitoring, and updating devices.

• Real-Time Data Processing

Preprocessing: Clean and filter raw data to enhance accuracy.

Analytics: Algorithms for real-time gesture and gaze detection.

• Data Storage and Management

Databases: Store large volumes of tracking data for analysis and reporting.

Data Management Tools: Organize and manage data access and storage efficiently.

• Recognition Engines

Machine Learning Algorithms: Interpret hand gestures and eye movements.

Gesture Libraries: Predefined sets of recognizable gestures and movements.

• Security and Privacy

Authentication and Authorization: Secure access control to data and functionalities.

Data Encryption: Protect data during transmission and storage.

• User Interface (UI) and Experience (UX)

Dashboards: Real-time data visualization and device status monitoring.

Customization Tools: Configure and personalize tracking experiences.

• Scalability and Orchestration

Microservices Architecture: Independent, scalable components for each function.

Load Balancing: Distribute workloads across servers for optimal performance.

Containerization (e.g., Kubernetes): Manage and scale components efficiently.

With WebAR we can avoid the need to install special programs, no installation is required.

These building blocks work together to ensure a robust, scalable, and efficient hand and eye-tracking system in enterprise environments.

Scaling Hand and Eye Tracking

Ivan Rajković:

Scaling hand and eye-tracking tools in enterprise settings hinges on several key middleware components, with access to camera feed data being one of the most crucial—and often overlooked—elements.

This data is essential for accurate tracking, giving developers the raw material to create responsive, intuitive interactions within XR environments.

However, accessing this data raises significant privacy concerns, including questions about user consent, data security, and management. While these concerns are valid and must be addressed, they shouldn’t hinder innovation.

Reliable access to high-quality camera data is vital for advancing hand and eye-tracking, refining virtual interactions, and integrating machine learning. Properly managed, this access could be the catalyst that propels these technologies forward, enabling scalability across industries.

In addition to camera data access, software development kits (SDKs) are critical middleware components. Solutions like OctoXR offer modular SDKs that support the scalability and adaptability needed for these technologies to thrive in various enterprise settings.

Converting Essstaintial XR Data

Looking Glass:

From a software perspective, converting raw data into meaningful interactions. Improving middleware that can parse the data from hardware solutions will allow integrators to better access the value of eye and hand-tracking solutions.

From a physical perspective, hand-tracking UI interaction methods are less precise than physical controllers like a mouse or button.

The largest barrier to use, however, is the latency between user movement and tracking. We are seeing great leaps forward in alleviating such environmental restrictions, but they still exist today.

New Applications of Hand and Eye-Tracking

Jose G. Pichel:

It must be taken into account that, depending on the work environment, continuous eye movement can cause fatigue to the user.

In the enterprise space, hand and eye tracking offer numerous innovative applications:

• Training and Education

Immersive Simulations: Create realistic training environments where employees can practice complex skills safely.

Real-Time Feedback: Provide instant feedback based on eye and hand movements to enhance training effectiveness.

• Customer Service and Sales

Customer Interaction: Use AR tools for product demonstrations, tailoring presentations based on customer gaze.

Behaviour Analysis: Optimize product placement and marketing strategies by analyzing customer gaze patterns.

• Remote Collaboration.

Enhanced Virtual Meetings: Improve communication by allowing participants to point and direct attention through eye and hand tracking.

3D Collaborative Work: Teams can manipulate 3D models in real time, enhancing design and architecture collaboration.

• Productivity Enhancement

Contactless Interfaces: Control systems with eye and hand gestures, useful in hygiene-sensitive environments.

Task Automation: Automate repetitive tasks by learning user patterns through tracking.

• Security and Compliance

Biometric Authentication: Use eye and hand tracking for secure access to sensitive systems.

Fatigue Monitoring: Detect signs of fatigue in critical jobs to prevent accidents.

• Assistance and Support

Remote Technical Support: Technicians can provide precise guidance by seeing what users see via AR.

Process Guidance: Assist workers in complex tasks, ensuring accuracy and reducing errors.

Hand and eye tracking technologies can significantly enhance training, customer interaction, collaboration, productivity, security, and support in enterprise environments.

The Impact of Tracking in XR

Ivan Rajković:

I’m particularly excited about the impact that hand-tracking and eye-tracking technologies will have on training simulations in the enterprise space.

The integration of these technologies is poised to revolutionize how training is conducted across various industries, especially in fields like healthcare, aviation, and manufacturing. Imagine a surgical trainee using a VR simulation where eye-tracking can monitor their focus on critical areas while hand-tracking provides real-time feedback on their movements.

The system can analyze where the trainee’s attention is during complex procedures and assess their dexterity in performing delicate tasks. This level of insight allows for highly personalized training, where the program can adapt to the trainee’s needs, offering additional practice in areas where they may struggle.

These applications are just the beginning. As these technologies continue to develop, the potential for even more sophisticated and immersive training environments will grow, leading to better-prepared professionals and, ultimately, safer and more efficient workplaces. The ability to create such tailored and responsive training programs is a game-changer, and I’m looking forward to seeing how this will evolve in the coming years.

Bringing Tracking to the People

Looking Glass:

Bringing hand-tracking out of 1:1 applications and into rooms full of people, where everyone benefits from usage, whether they’re gesturing or watching someone else do so. The potential applications are endless.

Today, our customers combine hand-tracking technology with group-viewable 3D displays to better communicate and present 3D content.

Gesture interaction will deliver more value as the precision of the technology improves. When eye-tracking technologies resolve latency issues, there will, without doubt, be more integrations that deliver value to end-users.