XR is the new name of Head Mounted Computing and AI does not use a display when it plays video games: XRDC and xRS 2019 Conferences held in SF

By Ted Pollak (Editing and contributions by Jon Peddie)

Due to the incorporation of cameras on the front of what have been traditionally known as “VR Headsets” the VR and AR industries were faced with a conundrum and an overlap. AR can be experienced through cameras in a VR head-mounted display (HMD). In fact the vast majority of headset AR is experienced through cameras in the market today. The solution was to put an umbrella over the VR and AR computing “states”. The term that has been agreed upon by the vast majority of industry participants is “XR”. To add a bit of confusion XR to some also includes non-headset based display systems in some cases like a “holodeck”. I think the key defining factor should be complete freedom of visual perspective using natural movement. This is the R part of XR; Reality!

Augmented Reality is one part of the metaverse. Paul Milgram (1938 - ) and Fumio Kishino, defined augmented reality in 1994 as part of the continuum middle ground between virtual reality (completely synthetic) and telepresence (completely real).

Simplified representation of the metaverse continuum (Milgram, 1994).

The term MR has been superseded today by XR, the term introduced and popularized by Qualcomm in 2017.

XRDC 2019 at the Fort Mason Pavilion (Source: Ted Pollak)

XRDC 2019, owned by Informa Tech (who also own GDC) was held this year at the Fort Mason Pavilion, in San Francisco. This is a stunning venue right on the waterfront in what used to be Navy supply depots. The conference was unique because of its configuration. The building is one large room so the organizers utilized a newer format called a “silent conference”. This is where talks, panels and seminars are given almost side by side and the audience use color coded headphones for the talk of interest. The presenters and Q&A sessions of other talks cannot be heard. It works quite well. I did hear one complaint of an attendee who said his ears started to hurt from the headphones. An easy solution would be to offer an alternative to the “on ear” phones with “in ear” phones. Using disposable foam inserts this would give the ear flap a break if needed. It’s also a handy way to deal with translations should they be needed.

XRDC “Silent Conference” Format (Source: Ted Pollak)

One of the talks I attended was by Omer Shapira a Senior VR Engineer at Nvidia. Omer discussed how game engines are being used to train AI. The most difficult thing for an AI to deal with are anomalies and training AI for anomalies in real world settings is almost impossible. Using a game engine to create potentially infinite scenarios makes this process more manageable.

Omer Shapira Senior Engineer Nvidia (Source: Ted Pollak)

When it comes to training AI that relies on visual and environmental sensors, Nvidia does not render the synthetic game engine data on a display for the sensors to “see”. They directly inject the visual code into the sensors, they call it “rendering” and they use GPU’s to process it. They even ray trace lighting.

But there is no need for a display or an image because of the direct code processing. This is obviously superior in many ways. For example the sensor “model” itself could be injected with synthetic data such as a single frozen snowflake (from the environmental model), or three snowflakes, or a splotch of oil. And then the game engine code is learned though the “handicapped” sensor as best it can with other sensors, or the AI learns it’s best to ignore that sensor and produce a warning or apply a remedy if it does not add any usable data. 

Nvidia Drive Constellation Architecture (Source: Nvidia)

There are infinite scenarios, permutations, and combinations. So another thing that is going on is “accelerated learning”. Since the AI can learn at faster than real time, why hold back the scenario analysis processing to real time? Years of learning can be fit into days. It’s all a matter of computing horsepower how fast it can learn. Essentially the AI is playing “serious games” to learn, nothing needs to be rendered on a display, and it is doing it faster than real time. In some cases MUCH faster.

We also spoke with Unity’s Tim McDonough, Unity’s VP & GM of Industrial and Dan Miller, XR Evangelist. Unity is taking “serious games” seriously. Whether its industrial use of XR or education and training. Unity also spoke about synthetic data used to train AI. Their engine knows the ones and zeros of 3D objects. Thus an engine is necessary to inject the scenario data into sensors and AI whether the images are visually rendered on a display system or not.

Moving to GreenLight Research’s xRS conference at the swanky Kabuki Hotel in Japantown, I attended a medical XR talk and had a chance to speak with a Professor of Surgery from Stanford University School of Medicine, Carla Pugh. Game tech is moving toward surgery with VR training, pre-procedural AR planning, and eventually real time HMD surgery (e.g. HMD on and scalpel or joystick in hand). In order for XR to be effective in real time real world surgery the HMD must equal or exceed the fidelity of traditional displays and/or provide extremely precise CGI overlays or renders. An HMD would also allow for a more natural posture for the surgeon as currently they look at a display that’s not in line of sight to their hands (which appears awkward). However some solution must be made so that the user sees their hands AND the laparoscopic display. This is true for both see-through (e.g. Hololens) and digital view (e.g. Oculus or Varjo) scenarios. Perhaps there could be a back/forth switch or a side by side display in the HMD. 

Awkward Posture during Laparoscopic Surgery (Source: Heidelberg University Hospital)

xRS moved locations the second day to Microsoft Reactor Studios; just down the street from my office. Greenlight did a good job organizing the event, keeping it moving at a comfortable pace, and providing a whitepaper to attendees. Microsoft is in a nice sweet spot with HoloLens on display. We have covered the HMD extensively in other articles. We will instead focus on something new.

Greenlight’s xRS at Microsoft Reactor Studios (Source: Ted Pollak)

One of the more interesting talks was entitled “True to Light: Exploring Light Field Displays for xR”. Ryan Damn of Visby and Jon Karafin of Light Field Labs discussed the science behind this display technology (which exceeds the scope of this article). I encourage people interested to visit Greenlight Research’s website and read the white paper. The paper discusses the challenges and opportunities across all XR display tech. However far off, light field holographic displays could represent a paradigm shift for not only XR but how we view images, 3D objects, and video in general. 

Light Field Display (Source: Light Field Labs)

In conclusion, The San Francisco Bay Area is shaping up to be an epicenter for more than just earthquakes. XR and AI are measuring very high on the Richter scale. In the course of just a few months three entire XR conferences are being held by different organizers.