Skinned Reality: Part 1 - The Virtuality Experience (VE) Continuum

Over time, the Milgard and Kishino RV Continuum has become a de facto reference point for attempts to understand and categorize the various types of virtualized experiences that the relevant technologies make possible. Yet, as the Speicher, et al. 2019 paper "What is Mixed Reality?" demonstrates, there is still much confusion around the definitions of AR, VR, and MR among industry professionals. In addition, despite several attempts, no practical matrix to analyze and define such experiences has been broadly adopted.?Others have expanded or modified the original Milgard and Kishino dimensions, sometimes (as in the case of Speicher, et al.) so broadly that the result is a matrix of classifications of little practical value and other times so orthogonally that you wonder if the authors have ever actually experienced the technology.

Whereas Milgram and Kishino and others have focused their analyses around various display technologies and viewing methods, direct and indirect, my exploration will be in the context of virtuality experienced via Head Mounted Display (HMD) devices, including their various peripherals and likely future capabilities. Current examples of this class of device include the Microsoft HoloLens and Lenovo A3 and the tethered and untethered VR headsets from Meta (formerly Oculus), HTC, and others. HMDs enable the broadest range of MR Experiences for business and consumer uses and I believe these devices will eventually become ubiquitous.

Real Environment and Virtual Environment

As mentioned above, the Milgram et al. RV Continuum (Figure 1) has become the de facto reference for general explanation of Mixed Reality for those casually interested in the technology and also for developers and marketers of MR Experiences. Due to the popularization of AR and VR over the last several years, and the existence of readily available consumer and commercial MR applications, the general concept of the Continuum is now grasped by most people. However, the fact that the popular term "Virtual Reality" does not appear on the Continuum points to a fundamental inconsistency in its design.

Milgram and Kishino's original focus on display technologies resulted in a taxonomy that specifically describes various mixes of "Environments." In their words, "The concept of a "virtuality continuum" relates to the mixture of classes of objects presented in any particular display situation." They go on to say that the left side of the Continuum (labeled Real Environment) "defines environments consisting solely of real objects" whereas the right side (labeled Virtual Environment) "defines environments consisting solely of virtual objects." Real objects are those "that have an actual objective existence," regardless of whether they are viewed directly or are sampled and resynthesized through a display. For example, a real object viewed via video is still considered a real object. Virtual objects are those "that exist in essence or effect, but not formally or actually." Basically, if the object has been simulated, even if the simulation is based on a model of the real object, it's a virtual object. This gets a little murky when thinking of real objects that may be resynthesized with additional simulated elements, but you could still break the individual components of the resulting object down as real or virtual. Also, although Milgram and Kishino's original paper is primarily concerned with visual display of objects, these same concepts apply to non-visual "objects" like sound or scent.

Within the bounds of Milgram and Kishino's Real Environment and Virtual Environment, they describe two distinct MR Environment proportions of real and virtual objects which they label "Augmented Reality" and "Augmented Virtuality." The first refers "to any case in which an otherwise real environment is "augmented" by means of virtual (computer graphic) objects," and the second being the converse scenario where the environment is mostly virtual and is augmented by real objects. Here it is notable that the label "Virtual Reality" is absent from the RV Continuum. This is because they consider VR to be represented by the Virtual Environment, consisting of only virtual objects. They purposefully exclude VR from the MR scale, stating "In this paper we focus on a particular subclass of VR related technologies that involve the merging of real and virtual worlds, which we refer to generically as Mixed Reality (MR)."

This makes some sense within the display technology context of their discussion. Certain displays are built only to relay the Real Environment and others are designed only show a Virtual Environment. However, for our purposes a more useful continuum should include VR, not only because the absence of such a popular and important category seems odd, but also because VR (and the whole range of Virtuality within MR for that matter) should be defined as an "Experience," separate and distinct from Milgram and Kishino's Environment.

Reality Versus Real Environment

Environment is commonly viewed as the physical world around you. The Oxford English Dictionary defines its primary meaning as "the surroundings or conditions in which a person, animal, or plant lives or operates." Milgard and Kishino utilize this definition similarly when they talk about the Real Environment and the Virtual Environment, which consist of either real or virtual objects. They go on to state "…the most straightforward way to view a Mixed Reality environment, therefore, is one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the virtuality continuum." They think of Augmented Reality and Augmented Virtuality also in terms of Environments.

This labeling is confusing, and I wish they had instead used the more accurate labels "Augmented Real Environment" and "Augmented Virtual Environment." I understand that at the time of their writing the term AR was already being used commonly, but this was a research paper, where more precise, descriptive labels are preferable. The core problem is that the term "reality" has a different implication than the word "environment." The Oxford English Dictionary defines reality as "The state of things as they actually exist as opposed to an idealistic or notional idea of them." This sounds similar to the above definition of environment. However, the two words are not synonymous. Reality infers perception through experience.

Reality (with a big R) is our perception of the real environment through our human interaction with it rather than the real environment in itself. That is to say, our Reality, which consists of a plethora of mental constructs and relationships, is solely based on our experience of interaction with the physical world as human beings. Without this interaction through our various senses, we have no way of constructing our human concept of Reality. So, Reality is a construct of experience through interaction.

To illustrate, a person totally blind from birth cannot understand color because they've never experienced it. Although studies show that blind people can form concepts of color through object associations, that is only the case because they are aware of the existence of color through communication with non-blind people. In isolation, no blind person would have a concept of color because it simply wouldn't exist based on their lack of ability to experience it. It would not be part of their Reality, yet the physical properties that enable color to be experienced by non-blind persons still exist in the Real Environment.

For those who are non-blind, let's do a quick experiment. Close your eyes after reading this paragraph and think of the color Orange. Visualize it. Then think of the color Magenta. How does it differ from Orange? Now think of the color Obutane. How would you describe it? Turns out, you can't describe Obutane because it is not a color that we, as non-blind humans, have defined in our collective Reality. Similar to all colors for blind persons, the color Obutane currently does not exist for non-blind persons.

The point of this is to emphasize that Reality and Real Environment are distinct. Reality is a result of our experience of human interaction with, and independent from, the Real Environment. Everything stems from physical experience, even our most abstract concepts. Reality is a fabric woven of Experience.

The Virtuality Experience (VE) Continuum

If we reframe the terms "Reality" and "Virtuality" as expressions of "Experience" rather than of "Environment," we can create a more homogeneous Continuum of Experiences. Within this reframing, the Real Environment and Virtual Environment are no longer necessary because we are only concerned with Experiences and not Environments. The far left of the Experience scale starts with Reality as that is the closest Experience we have of the Real Environment. To the right of that is Augmented Reality (AR), then Augmented Virtuality (AV).

While AR has been adopted popularly and its meaning is somewhat understood, AV is not a term most people recognize. Still, it does precisely describe the Experience where real objects are placed in a mostly virtual scene. You might think the term Virtual Reality (VR), a more popularly recognized label, could be used in place of AV.?But the two are different. Pure VR does not have any real entities and is the closest Experience to the Virtual Environment. Therefore, VR is the right-most point on the Continuum.

Figure 2 shows the new Continuum which is now re-framed to focus on Experiences. Notice that the range of MR Experiences does not include Reality but partially includes VR. Here we could again have a philosophical discussion about whether our human Reality could be considered MR given all interaction with the Real Environment is through our senses, which are sometimes augmented by assistive devices such as glasses. hearing aids, and mirrors.* (see footnote for more on mirrors) You could also say that Reality as experienced through a VR HMD is necessarily augmented since the reproduction of the Real Environment through sensors and renderers is never exactly the same as the actual Real Environment. Still, I don't believe that adds practical value to our distinctions. Perhaps someone can come up with better reasons to include it, but I will leave it outside of MR for now.

A more difficult decision is whether to extend the MR umbrella to include part or all of VR. As shown in the VE Continuum, VR is partially included in the range of MR Experiences. First, the division between these categories is not at all sharp. Second, most VR experiences include a representation of the participant's tracked controllers or hands, which are real objects within a scene. Those experiences that are purely virtual with no real entities present are typically less interactive, for example 360° video. For those exceptions, I leave a portion representing pure VR outside of the MR Experience range.

In their March 2021 paper, "Revisiting Milgram and Kishino's Reality-Virtuality Continuum," Skarbez, Smith, and Whitton claim that sensory conflicts are inherent in conventional Virtual Reality systems and so the theoretical Virtual Environment does not exist outside of the MR spectrum. Instead, they label this the "External Virtual Environment" and break the Continuum to place a "Matrix-like" Virtual Environment outside of the MR spectrum in place of Milgram and Kishino's Virtual Environment as shown in Figure 3.

When Skarbez, Smith, and Whitton talk about their "External Virtual Environment" and the "Matrix-Like" Virtual Environment, I believe this is an example of the aforementioned confusion between Environment and Experience caused by the inconsistency in labels used by Milgram and Kishino. What Skarbez, et al. really mean by External Virtual Environment (and they affirm this in the note to their diagram above) is an Experience called Virtual Reality. Some level of sense conflict will always exist in even the most advanced VR Experience by definition. This is what separates Virtual Reality from Reality.

Using the movie, The Matrix to illustrate (since Skarbez, et al. brought it up), Neo believes at first that he is experiencing Reality but begins to notice some inconsistencies (sensory conflicts) in this Reality. Once he becomes conscious that his Reality is actually a simulation, only then does he understand he is experiencing Virtual Reality.

(As an aside - I read an article last year by Andy Gstoll on Mixed-Reality.io in which he bends the end of the RV Continuum back to its starting position. He also mentions The Matrix and says that 100% virtuality is the same as subjective reality, or as I call it, Reality.)

Environment does not include any notion of the perception of the person experiencing it. So the Skarbez, et al. "Matrix-like" Virtual Environment is no different than the Virtual Environment of Milgard and Kishino. In the clarified VE Continuum, there is no need for relabeling and breaking?the Continuum. Virtual Reality (the equivalent of the Skarbez, et al. "External Virtual Environment" reframed as an Experience) can exist inside of the MR range as they prefer. Virtual Environment (or the Skarbez, et al. "Matrix-like" Virtual Environment) drops off our VE Continuum since, as an Environment, it is outside the bounds of our Experience scale.

The VE Continuum and Mixed Reality Experiences

The resulting VE Continuum (Figure 2), reframed to represent a range of possible Experiences rather than Environments allows us now to clearly consider all possible levels of Virtuality from none (Reality) to all (Virtual Reality) when attempting to categorize an MR Experience.?Using the Continuum, you can place different MR applications sensibly along the timeline. Below (Figure 4) is our VE Continuum populated with some examples. In general, the typical AR experiences built for Hololens would fall along the left region and VR games and experiences you can find for VR HMDs would fall to the right.

• At far left I've placed the Hololens Homescreen, which is basically a pass-through experience with a user menu that can be navigated. This menu is not by default anchored to the Real Environment. You could imagine this level of virtuality also applying to experiences where IoT data related to a piece of nearby machinery is displayed or a settings menu pops up.
• To the right of that I've placed one of the first demo experiences Microsoft built for the Hololens, RoboRaid, (which is still very entertaining by the way) in which robotic scorpion-like insects seemingly break through the walls of the Real Environment to attack. To achieve this experience, the system needs to accurately scan the Real Environment and anchor the virtual action to real objects.
• Next, I place Meta's Horizon Workrooms with desktop passthrough activated. This experience allows users in a largely Virtual Environment where they are sitting at a virtual conference table to cut a spatial region in vr that shows their Real Environment desktop. The practical result is that you can see where your computer keyboard is and where you've set your coffee so you don't knock it over.
• Under the AV label is the warehouse-scale roaming experience, Star Wars: Secrets of the Empire, developed by The Void, which blends Real Environment spatial mapping with VR. Users can walk around a warehouse space with real objects (hallways, benches, ramps) that are mapped to the Virtual Environment to enhance the level of Reality felt by the user. They even throw in some haptic, olfactory, and tactile elements.
• The popular VR game Beat Saber is next. This experience entails cutting through 3D blocks with hand-held light sabers to the rhythm of a song track as they fly towards you. The otherwise virtual experience is anchored on the movement of your real hands represented by the virtual light sabers.
• Further right is the sweet and short VR movie "Allumette" from Penrose Studios. Although not anchored to the Real Environment, this 3D story presentation allows the viewer to move about the scene freely, changing their viewing perspective on the story. Users can pop their head inside a floating ship to see story action they would otherwise miss from outside the ship.
• Lastly, I have put 360 Video (when viewed via HMD) as the most virtual of our examples. This experience is not anchored or spatially aware and is typically passive without any interaction other than initiating playback via a menu.

I've tried to pick some distinct examples to show the different levels of virtuality along the continuum. There are many others I could have used, but I doubt their inclusion would have added much. Just determining the level of virtual and real objects within an experience doesn't help us to fully understand what that experience is like unfortunately. That's the problem with the VE Continuum - the level of virtuality (the number of real objects versus virtual objects present) is only one aspect that makes up an MR experience.

The Mixed Reality (MR) Experience Graph

Because the Continuum alone is not sufficient to fully categorize an MR Experience, like Milgard and Kishino and many others, in Part 2 I will describe a taxonomy that enables analysis and description of any Experience on the VE Continuum through multiple dimensions. These are comprised of “Mix of Virtuality,” “Level of Interaction,” and (adopted from Milgard and Kishino) “Extent of World Knowledge.” In addition to these 3 axes, I've included an important 4th qualitative aspect called “Agency,” which indicates whether user presence or interaction influences the course of the MR experience.

I call this the MR Experience Graph.

COMING SOON - PART 2: THE MIXED REALITY (MR) EXPERIENCE GRAPH

?Footnote:

* Mirrors are perhaps the first human-made MR device. They reverse the image. Since what you see is not what would naturally be seen by a person gazing directly at an object, you could say that the image has been artificially altered and is virtual. Imagine a mirror so tall and wide that it encompasses your entire field of view. Everything you would see reflected would be altered from its natural appearance.