A chief vertical of extended reality (XR) technologies, 5G is widely referred to as the lifeblood of the future virtual, augmented, and mixed reality (VR/AR/MR) industries and their developing products.
5G’s low latency and lightning-quick speeds have provided crucial results for those working in sectors such as sports, military, robotics, entertainment, teleoperations, medical, automotive, and many others.
Major telecoms such as Nokia, T-Mobile, Huawei, Ericsson, and others have begun trialling 5G, but upscaling the emerging technology will require immense feedback on its performance.
The Metaverse, an all-encompassing and pervasive interface between the digital and physical worlds, which has seen a meteoric rise following Facebook’s rebranding to Meta, will also rely on 5G connections to enable the rapid transfer of massive amounts of data for billions of users—all with minimal delay.
For our XR Today round table, we are pleased to welcome:
- Greg Jones, Director of Global Business Development and Product Management for XR at NVIDIA
- Florian Haspinger, Chief Executive for Holo-Light
- Urho Konttori, Founder and Chief Technology Officer for Varjo
- Mani Srinivasan, Chief Business Officer for TRIPP
- Joe Michaels, Chief Revenue Officer for HaptX
Panelists have explored the many benefits of using 5G connections for their solutions, critical network demands, and potential growth for global clients and individual users.
Can you discuss how your company aims to develop and grow XR solutions for 5G? Why is 5G important to the rollout of global XR technologies?
Greg Jones: Our NVIDIA CloudXR platform is a groundbreaking innovation built on NVIDIA RTX technology for delivering VR and AR across 5G networks.
CloudXR is a streaming system that, combined with the low latency, high bandwidth, and the low jitter of 5G networks, enables users to experience rich, immersive environments using mobile XR devices.
By combining CloudXR with NVIDIA RTX graphics processing units (GPUs) and virtualization technologies, the scalable capability of streaming high fidelity, photorealistic XR experiences is available to both dedicated and public 5G networks.
With our partners, we have shown several installations that demonstrate the value and deliverability of high-fidelity, photorealistic XR streamed over 5G networks in areas ranging from manufacturing to location-based entertainment.
Florian Haspinger: Augmented and virtual reality provide one of the most exciting applications of 5G. For real-time AR/VR and their associated use cases, ultra-low latency and high bandwidth are so crucial that 5G is more essential than in any other 5G-enabled use cases.
Conversely, 5G needs new immersive technologies like XR streaming to open the door for industry-changing apps and flagship use cases. Holo-Light is currently developing a 5G-ready immersive streaming platform, which is similar to a ‘Netflix’ for AR/VR apps.
By leveraging 5G and platform-based streaming technologies, we cut the boundaries–from the digital to the real world–to an absolute minimum, as content will be available anywhere at any time.
This will provide the industry with the potential for mass adoption of their XR use cases and mass rollouts in less time than ever before.
Urho Konttori: At Varjo, we believe in a wireless future and 5G is at the core of that. We work alongside experts in the domain, including our global partner Amazon Web Services (AWS) and Telia in the Nordics to develop better and faster solutions for the future of work and communication in XR.
To expand and secure coverage for XR solutions in 5G networks, we strongly believe that separate end-to-end encryption is crucial, not only in 5G, but in any wireless networks, to ensure that users feel confident and safe in the technology they are using.
Mani Srinivasan: TRIPP feels fortunate to be the category leader in Wellness XR [technologies] today, having begun with a compelling VR offering and expanding recently into AR.
We plan to expand coverage through collaboration with top device original equipment manufacturers (OEMs) such as Meta’s Oculus, Sony Playstation VR, HTC, HP, Pico-ByteDance, Nreal, JioTesseract, and Apple, as well as telcos such T-Mobile, Deutsche Telecom, Verizon, Qualcomm, KDDI, and others.
By participating in the T-Mobile Wellness Accelerator programme, we are in the process of leveraging 5G edge technologies to make “reality layering” a reality.
5G allows us to shift [computer] processing to the cloud rather than being limited by what is available on the device, enabling us to deliver rich experiences to lightweight devices like Nreal’s MR glasses and HTC’s VR glasses. Once 5G is broadly deployed, there will be a tremendous amount of innovation driven by the introduction of edge computing.
Joe Michaels: Many of our customers use HaptX Gloves to operate dexterous robot hands at the end of robotic arms. These human-operated robot hands must perform delicate, complex, mission-critical tasks ranging from repair of dangerous IT equipment to medical procedures. This remote control often takes place over wide distances, sometimes [over] thousands of miles and across multiple networks.
Low-enough network latency can mean the difference between success and failure in these teleoperation tasks. HaptX and our customers will increasingly depend on the low latency of 5G networks to make realistic haptic feedback possible during robotic teleoperation procedures.
How can 5G networks facilitate XR technologies at the telecom, industrial, or consumer level? What are the benefits of 5G streaming for standalone hardware?
Greg Jones: For industrial and consumer XR use cases, 5G is a complete change. We are working with partners in providing tools, such as NVIDIA Omniverse, to simulate and optimize networks for deployment in both dedicated and public networks.
After design, we provide an array of capabilities for building the network from our NVIDIA Aerial Application framework to build high performance, software-defined, cloud-native 5G applications to address increasing consumer demand.
XR technologies require networks with low latency, high bandwidth, and low jitter. This is a difficult combination to provide for any network.
The handshake between network and XR applications that run on that network requires a dynamic optimization of both server and client functions such as encode compression and frame buffering.
We address this need with our CloudXR platform. The benefit to users is a lightweight and easy to use end device with access to nearly unlimited computing power, including NVIDIA RTX graphics.
Florian Haspinger: Traditionally, companies would need to invest in their own on-premises edge sites to gain the benefits of 5G. For many industrial enterprises, this investment is necessary to ensure that applications receive the capabilities required to carry out mission-critical processes.
By utilizing, for example, a telco edge, firms get the advantages of edge computing while not having to invest and maintain their own costly infrastructure, to provide the best of both worlds: low latency with a 5G mobile network and the flexibility and scalability of a cloud data center.
The benefits of 5G XR streaming are immense. Outsourcing the rendering process to a powerful external server and streaming via 5G networks will be the key to limitless usage of mobile applications in AR and VR.
Streaming allows [users] to break performance barriers of mobile devices while creating availability on a global scale. This will ultimately help to further optimize the form factor and pricing of standalone hardware as local processing power is no longer essential, bringing immersive technologies to the mainstream.
Urho Konttori: 5G provides incredible end-to-end latency, but for it to really succeed, the carrier backbone needs to be revamped across the world to better support 5G bandwidths. A low latency link for low bandwidth data is also going to be crucial for 5G networks to facilitate XR technologies.
For example, with the Varjo XR-3, you can have a realistic telepresence in other parts of the world. In the future, with the support of 5G and Varjo Reality Cloud, you’ll be able to virtually teleport your colleagues to the physical space you’re in by recreating a virtual version of it.
Mani Srinivasan: From a tech infrastructure standpoint, the benefits of 5G-powered XR content is that there will be very low latency connections from the headset to the edge, which opens up the door to creativity and will allow for greater personalization and higher quality of service from TRIPP.
As such, there is a virtuous circle whereby the value of hardware and content experiences, in conjunction with 5G, become greater and the end user benefits tremendously.
Meanwhile, smaller form factors are rapidly evolving in the progression from hand to head. Nreal in AR and HTC Vive Flow in VR have terrific early new devices that are setting a standard. Moving forward, there are going to be fewer distinctions between VR, AR, and mobile with convergence and integration setting in quickly. Again, the primary beneficiary will be the consumer in all of this and that is exciting!
Joe Michaels: We have shared our observations about the requirements of robotic teleoperation with network providers like Verizon which, we should disclose, is an investor in HaptX.
We explained that haptic data must move at a maximum of 200 milliseconds from end to end in order for human operators to interact naturally with remote objects.
Any slower and the robot hands may knock over an object before the operator feels it. The Tactile Telerobot, which we developed in partnership with Shadow Robot Company and Tangible Research, works best within this range.
With absolutely no network latencies, the typical elapsed time from when a Tactile Telerobot operator moves their fingers for the data to reach the robot, and for the haptic feedback to get back to the operator, is about 40 milliseconds. Verizon’s 5G Ultra Wideband network has reported latency of less than 30 milliseconds. This speed will enable extremely reliable and safe robotic teleoperation.
Why is 5G essential to developing your XR solutions? What specifically does it afford the many people, firms, and systems that will depend on it?
Greg Jones: XR is a demanding service for wireless delivery, and XR applications are inherently intolerant of high latencies, limited bandwidth and high jitter. Poor network performance not only results in a less-than-perfect XR experience, but even nausea for the user.
XR experiences are effective in wireless networks, which generally means a high demand on bandwidth and increasing interference—both network parameters where 5G shines.
Streaming 5G-enabled immersive experiences fundamentally changes the distribution of XR applications. 5G will enable the high-fidelity, photorealistic XR user, previously defined as an individual with access to a high-powered workstation, to be anyone with a mobile device including handhelds and All-in-One head-mounted displays (HMDs).
For the people, firms, and systems that will depend on 5G to deliver XR solutions, this means a simple to access, rich XR experience for work, play, and learning available, anywhere and at any time.
Florian Haspinger: The low latency and high bandwidth that 5G brings is crucial in making higher quality experiences available to a broader audience. Whenever real-time data processing is a critical factor, 5G edge infrastructure is a good choice. The move to the edge means that image frames can be rendered much closer to the end-user (compared to the cloud) on a powerful data center grade GPU.
Companies like German car manufacturer BMW already use AR software with integrated streaming capabilities to speed up vehicle design processes by as much as twelve months.
They apply streaming to visualize and work on 3D content in very high quality and complexity, which would not be possible with the regular computing power of mobile devices.
5G will further enhance these kind of use cases as it will be key to realize the ‘content everywhere, at any time, and in high-quality’ mantra.
Urho Konttori: The use of 5G enables us to have a guaranteed quality of service in all environments
where our headsets are being used.
While this can also be achieved in many WLAN environments, reliability and robustness just isn’t the same. Simulating things at the same acuity you see in real life is a game changer compared to standard VR approaches.
The workspace of the future needs to be rendered in the cloud–with millions of pixels of extremely high-resolution, uncompressed content with single-digit millisecond latencies delivered to our devices–whether on premises at carmakers or in remote sites through 5G.
This advancement will help us further realize our vision for creating a reality-based metaverse, where we will be able to more effectively design products of the future, meet and collaborate with our colleagues far away, and experience any remote place in real-time.
Mani Srinivasan: TRIPP’s recent launch on Nreal’s mixed reality glasses has highlighted how we are at the forefront of making digital health and wellness accessible and adaptable to the needs of our customers.
Through 5G, we have been able to create experiences that change the way people perceive and interact with the world around them, ultimately transforming the fascination that can be explored in the mundane.
Through our partnerships with various XR platforms, we have found that five-minute sessions allowed users to find peace and calm “on-the-go,” wherever and whenever our members are, throughout the day.
The Edge is a game changer because there will be no dependency on WiFi and the speed at which data will be exchanged and incorporated into greater, more personalized experiences will be fantastic.
Joe Michaels: Teleoperated robots will allow humans to perform dangerous, dirty, or dull jobs from a distance. For example, engineers can safely do work on radioactive nuclear sites. Technicians can remotely repair high-voltage electrical equipment, and doctors can offer life-saving medical care even when they can’t be next to the patient.
In many cases, successful robotic teleoperation won’t be possible without the reliability, availability, and low latency of 5G networks. Operators will wear [VR] head-mounted displays to see what the robot sees and also HaptX Gloves to feel what the robot feels.
It is critical to ensure the operator can receive and process all of this data in proper sync. Think about how Zoom and Microsoft Teams sometimes struggle to keep the audio and video of a virtual meeting in proper sync. Now add another sensory input – touch feedback – and you can understand the importance of 5G network performance.
