How Many Realities Do We Want Anyway?

I consider myself an expert in 3D graphics and augmented reality (AR), having done research, development, papers and productization in both domains.  However, with recent literature about AR and MR (Mixed Reality), there is general confusion about what makes something AR versus what makes something MR.  I thought I would do a post as to what I think, and would love to hear from you as to what you understand.

What I know about AR

The basic definition from Wikipedia: a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data.   I would like to elaborate on this definition with some specific examples below.

Visual display of graphics data

AR is often associated with visualization, so the common augmentation can be with graphics such as:

• 2D graphics data: text, 2D images, etc. Think Google Glasses.  Note that this example also has sound and GPS data incorporated as well, as per the definition above.
• 3D graphics data: means 3D geometry as part of the real-world environment. Think IKEA furniture browsing and placement.  This is a marker-based solution, which means it requires fiducial markers to orient and place the 3D graphics objects.  A markerless solution evaluates the scene to orient and place the 3D graphics objects accordingly – see the Volvo repair concept.

Live view

The live view portion of the above AR definition may be a little misleading.  As early as 1990, a bunch of us were working on merging 3D graphics with live-action film (think Jurassic Park, with digital dinosaurs).  We worked on camera matching, light matching, and how to render shadows properly, so the 3D graphics would look integrated with the live action.  Not until 1992, when I was made aware of a paper on computer augmented reality, that I realized we have been working on AR.  Film work does not represent a live view in the pure sense – it’s a recorded view of the live action, but this merging is AR.

Furthermore, imagine we have a real-time AR app that displays different parts information as your camera pans through the equipment.  If I were to record that camera panning as a video, then ask the same app to process it so we can see the parts information show up during the replay of the video, would the app be considered AR even though the view is not live?  I would.

Immersion

Not explicitly stated in the definition, but AR has no implication on whether it is immersive – it can be either.  I believe the confusion arises because the expectation of VR (virtual reality) does include immersion.  In fact, many of the AR apps are not immersive (the IKEA example above).  Some examples of immersive AR include the Hololens and Magic Leap.

What I know about MR

What I understood about MR is that it is part of the virtuality continuum as described in 1994 (see image above), where the two ends of continuum represents the real world, and a world that is entirely virtual.  The items in the continuum are AR and augmented virtuality (AV).  The distinction is:

• AR: virtual items are inserted/merged into the real-world.
• AV: the reverse – real things are inserted/merged into a virtual world. Think Tron.

I get the above definition of MR, and AR is part of the continuum.  But in recent years, the MR definition has changed in multiple conflicting ways (including the use of an equivalent MxR acronym sometimes), such as:

• A separation from AR entirely, but never clarified as to what exactly is different. In fact, some literature goes to the extent of saying “(something) is MR, not AR.” 
• MR = AR + VR. Since AR already has virtual elements augmenting reality, what exactly is the VR in MR that is not already in AR?
• MR contains 3D geometry integrated into the environment, and AR does not. However, evidence of 3D integrated into the environment dates back to the ‘90s and has been classified as AR, so what’s changed now?
• AR’s virtual objects are semi-transparent overlays, and MR’s virtual objects are opaque. If this were the case, I am not sure how to classify what we did at NGRAIN, which is to do 3D-outline-rendering to highlight objects (to minimize obscuring the real object).

What about this?

In looking over what has recently been classified as MR (such as Hololens and Magic Leap), I have drawn a few conclusions, such as they have characteristics of a live view, 3D and immersive AR.  Is immersion (immersion implying 3D and a live view) the main ingredient that separates MR and AR?  If so, why not refer to them as Immersive AR instead of MR?  Then we can avoid the MR versus AR confusions entirely.

Agree, disagree, even more confused, don’t really care?