Passthrough AR with Stereo Cameras

Augmented reality (AR) enables you to enhance the real world by layering virtual content on top of it. Most AR experiences are phone based, but there are loads of great headsets available that can help create immersive AR experiences. These headsets can be grouped into two main categories, the most popular of which is found in Meta2, HoloLens, and Magic Leap. It uses a transparent screen to layer digital content on top of the real world, and it's perfect for applications that require virtual information to be added to the users' view. We used this technique back at CES 2018 - check out the video below for more information.

While this is great in some situations, it's difficult for us to use on client projects because we can't accurately recreate the look of dark materials and surfaces via transparent AR. This is where the second category of AR headsets comes in. This alternative uses stereo cameras to pass the visuals through from the real world onto a display that is very similar to a standard VR headset. This solution relies on extremely low latency between the cameras and the display to prevent the user from getting disorientated by the lag. This camera data actually means we can create a perfect blend of virtual and real, so we're able to replicate the same high-quality materials and objects that we show in VR.

The technology is still in its early stages, but many hardware manufactures are teasing what the future will hold. The Oculus Quest, for example, gives a glimpse of this when you set up your VR bounds, and the VIVE Pro (and Pro Eye) can use its front-facing cameras to give the user a stereo view of the real world around them. These cameras have a fisheye lens that delivers a 96? horizontal and 80? vertical field of view with a 480p resolution per eye. They can also capture up to 90fps with an average latency of 200ms, so they're good enough to avoid any lag-related nausea.

The video below shows a tech demo we created to show the current state of passthrough AR technology. This experience renders a highly detailed 7-million polygon Pagani Huayra on the HTC VIVE Pro. Using HTC's SRWorks SDK (Software Development Kit), we can enable the stereo cameras and composite our virtual objects over the stream. This creates the illusion of the vehicle being present in the user's real surroundings.

An interesting advantage of this kind of AR is that the user can easily switch between AR and VR with the click of a button. When perfected, this will lead to a much more practical and safe virtual experience. In fact, as the hardware improves across the board, we'll start to see higher resolution and lower latency cameras, which will slowly bring the real-world camera view in line with the high-resolution virtual content.

With the launch of Valve's Index VR headset and the upcoming mixed reality addon, the XR-1 for the Varjo VR-1, we'll see the use of passthrough AR become more popular as an extension to VR solutions.

Original article can be found here: https://zerolight.com/news/tech/passthrough-ar-with-stereo-cameras

Follow our ZeroLight Tech twitter page for all our latest tech news and developments.