The slow rise of Mixed Reality

Author : Dominique Hudon

Enabling organic interactions between humans, computers, and the environment

If you’re into tech in the slightest, Virtual Reality (VR) and Augmented Reality (AR) are probably no news to you.

VR systems were first popularized by the gaming industry, but are now used for a plethora of use cases throughout many industries. AR systems, on the other hand, were a bit slower to emerge, but are now becoming more numerous in marketing, gaming, social media, and more. These two immersive technologies have been explored extensively in the last couple of years; so much so, that they practically became mainstream.

Today, however, we’re seeing these two technologies merge and give rise to the best of both worlds: Mixed Reality. ??

Mixed Reality (MR) is a relatively recent innovation. Not so long ago, it was still being refined, and wasn’t yet available or inexpensive enough for most consumers to enjoy; but now, it’s slowly rising and starting to get more widely accessible. Major tech powerhouses, such as Meta and Apple, are investing in this emerging technology, helping it get out of its niche and unlocking a huge number of potential applications. Should this pattern continue, MR could soon become commonplace in a multitude of professions, and mark inevitable change across several business facets.

What it is

Now, you may be wondering… What is MR exactly? ??

Mixed reality, also known as MR, MX, or hybrid reality, stands halfway between Augmented and Virtual Reality. Digital content is overlayed over the user’s real environment and can be interacted with in real time, which means elements of both the real and the digital world can be manipulated and interacted with simultaneously. For example, you could take a virtual box from your real table, open it, and see what’s inside.

There remains a collective ambiguity between AR and MR. Both overlay virtual content on top of a real-world view: so, what’s the difference? Well, the thing is: their terminologies mostly overlap, so they are largely synonymous. In both, digital additions are interwoven with the physical world such that they are perceived as being part of it, and can be interacted with as if they were real. The main difference is that MR headsets, as opposed to AR headsets, allow the user to transition between full immersion and partial immersion within the same experience. Yeah… Confusing, we know! ??

The immersive experiences spectrum, also known as “continuum of virtuality”, offers an insightful perspective in imaging these concepts.

The continuum is basically a linear representation of all the realities; its scale ranges from completely physical, to completely virtual. On the far left, nothing is digital; and on the far right, everything is. Near the digital end of the spectrum, the concept of Augmented Virtuality is introduced, where the virtual world is augmented with physical objects. Augmented Reality stands near the physical end of the spectrum since the vast majority of the information you view and interact with is genuine, with little digital enhancement. Now, Mixed Reality lies in the center of it all where the digital and physical worlds collide to create an all-rounded experience. It englobes both AR, and AV as an umbrella term.

For a more comprehensive overview of all immersive technologies and what differentiates them, check out our article?A whole range of realities . ??

How it works

At its core, Mixed Reality works with the help of Cloud Computing and Artificial Intelligence to create computer-generated content, and link it to the real world. Simple as that.

But, MR can also incorporate several other technologies such as cameras and sensors, spatial mapping and anchoring, infrared detectors, hand and gaze tracking, and vocal input, in addition to displays. Input devices and virtual lighting, for example, can heighten the sense of immersion; gaze tracking can personalize the experience; position tracking can register more complex interactions and relationships; 5G networks can allow faster performance and minimal latency; and advanced computer vision can allow for more advanced environment recognition and more efficient augments. Simply put: the more a device’s capabilities, the greater the Mixed Reality experience. ??

AR/MR devices such as?Microsoft HoloLens ?and?Magic Leap ?have been on the market for a while now, already leveraging overlays, scene understanding, and spatial anchors; but they also come with a pretty high price point. As of today, plenty of other devices are becoming available within all kinds of price ranges, trying to bridge the gap between top-of-the-line AR wearables and mass-marketed VR devices. The?Meta Quest 2 , for example, is a less pricey and more accessible device for general use; and the upcoming?Meta Quest 3 ?will allow higher quality MR experiences than its predecessor through higher resolution color passthrough. The?Meta Quest Pro , on the other hand, is a slightly pricier version, with a better display, higher resolution, color passthrough, eye tracking, facial expression recognition, and improved weight distribution for extended use. Then, there’s high-end hardware like?Varjo XR-3 ?and the upcoming?Apple Vision Pro , which offer a much higher performance, sky high resolution, and more. These headsets aim to answer pain points associated with everyday use, which prevented MR technology to be generally adopted on a daily basis, and to recreate virtually everything AR can do (from reading cell phone texts, to performing basic tasks seamlessly while in complete immersion).

All in all, what ties these new devices together is that it’s basically VR devices offering MR capabilities, whereas the previous ones are AR devices offering MR capabilities. Headsets such as Hololens or Magic Leap have clear lenses that let reality simply pass through, which is then augmented via projection; headsets such as Meta or Varjo, on the other hand, put the user in full immersion while built-in cameras capture their surroundings and send the video feed to their field of view, for it to then be augmented. Both achieve similar results, but through different filters; as such, they each have their advantages and drawbacks, and can be used in different contexts. VR/MR headsets, for example, can augment reality in a more immersive and realistic way, whereas AR/MR headsets have the advantage of not blocking the user’s field of view with an opaque screen, which is a big asset for on-site activities.

How it helps

This ability to interact with both physical and virtual content gives Mixed Reality technologies a huge number of potential applications. As it spreads and its costs come down, the chances are high that it will be widely adopted in different sectors. In fact, there are already many examples of MR-driven use cases; sometimes, also leveraging VR at the same time.

Now, we know what you must think: sure, it sounds cool and all, but it also sounds… insanely complicated. Granted, the process of creating MR content may have been time-consuming, hardware-intensive, and have required knowledgeable experts. Well… not so much anymore. ??

Our content-creation software?StellarX , by?OVA , is a sandbox tool that allows anyone to create immersive content — without code. Originally VR-only, we’ve now introduced a set of Mixed Reality features within StellarX in the last June and July releases, which allow you to merge the real and virtual worlds seamlessly. Our vision for StellarX has always been to democratize immersive technologies; we want to eliminate the middleman by allowing anyone, regardless of tech expertise or coding ability, to be able to create virtual worlds intuitively. We can safely say we’ve achieved that with VR — and now, are on our merry way to do the same with MR!

StellarX even offers co-location capabilities via shared spatial anchors, enabling multiple mixed-reality participants to collaborate in the same physical space.

So, yeah: now that we’ve established that customized Mixed Reality content is far from out of reach… Let’s look into what it’s possible to do with it exactly! ??

Teaching students

Mixed Reality can be used for interactive learning within the education industry, to teach any kind of subject. It allows students to interact with and manipulate virtual objects, all while having access to real-life environments or tools. They can gain a deeper sense of understanding as to what it is they’re studying, have a closer look at things that would be otherwise hard or plain impossible to access, and practice maneuvers with expensive or fragile equipment without any consequences. Plus, the gamification of the material makes it far more fun and engaging, boosting interest and retention rates. In a?research study carried out by Microsoft , MR immersive experiences in universities led to a 22% improvement in test scores, and a 35% increase in student engagement and retention. ??

For example, in an immersive simulation offered by Fundamental VR, MR is used to simulate the experience of doing surgical operations on the spinal column?(1). Students can manipulate real tools without being tied to controllers like in VR, but the subject is fake, managing any possible fallouts. Just being able to produce three-dimensional models of the anatomy, complete with information accessible by just a simple gesture, could change the way healthcare and medicine is taught?(2).

Designing systems

There are various ways in which the engineering sector can take advantage of Mixed Reality. Realistic holograms, 3D sculpting and real-time simulation can help with planning and design stages, and significantly improve quality assurance and control processes. Design teams can generate?digital twins ?of items, allowing them to quickly prototype, experiment with many materials, and test innovative concepts with minimal waste, and minimal cost. Project monitoring apps can even project 3D or 4D design models over structures right as they are being built. This gives engineers the best chance for visualizing progress and spotting errors, while also allowing real-time manipulation of their designs. ??

Ford, for example, has been using Mixed Reality technology to visualize vehicle designs, resulting in reduced development times and costs. The digital holograms allow them to explore more proposals while shaving time off design and engineering processes, which previously took weeks and even months using traditional clay models?(3).

Building stuff

Mixed Reality can teleport manufacturing plants directly into your office space. It can let you test production processes, oversee assembly lines, and monitor inventory, without actually having to be in close proximity to it. On-site, it can also greatly facilitate assembly, maintenance, and repair processes; algorithms can provide contextual information to operators in real time, saving lots of time (and money). ??

For example, AR/MR was leveraged by hardware manufacturer Dagri for the assembly of a gas turbine power plant. A step that used to take 480 minutes ended up being done in less than 45 minutes, without the need for any previous training?(4). Another example comes from the French manufacturer Renault Trucks. They have successfully adopted the HoloLens at one of their facilities to improve quality control for engine assembly operations. Now, instead of reading paper instructions, operators can get relevant information right there, while they’re doing the job?(5).

Training workforce

Mixed Reality provides tremendous training possibilities. Employees can train virtually, connect and collaborate with mentors in real-time, receive the help of AI for valuable insight, and more. Holograms can effectively substitute expensive, fragile, or simply inaccessible equipment with minimum disruption to usual work processes; they also allow for increased safety, which can be crucial in dangerous industries such as oil and gas, safety and security, or defence. Workers can be trained on fatal maneuvers, equipment, or sites, without any risk of injury. Muscle memory is developed with virtually no investment, allowing companies to save valuable time and money. ??

Japan Airlines (JAL), for example, actively uses the HoloLens to train engineers with virtual assistance from their location, without them having to visit a hangar at all?(6).

Mixed Reality could even be used jointly with Virtual Reality, for an all-rounded training experience.?Some of StellarX’s new features , leveraging the passthrough and bounding boxes, allow you to import a real-life scene to a virtual environment, gaining touch with reality while still being in total immersion. So, one could, for example, learn about a manipulation in an entirely computer-generated VR simulation; manipulating virtual equipment all while being guided by an expert, shown through a seamlessly integrated window and manipulating the real-life instance of said equipment in real time.

Breaking distance

Mixed Reality enables the development of far more engaging shared experiences. MR video conferencing can allow us to interact in new ways; maybe even to overcome language barriers by translating language in real time. Cameras can feed material to remote coworkers, allowing them to comment on what they observe and to offer adapted instructions. They can even draw on their screens to convey what they are trying to say, which places holograms over physical objects in the immersed user’s view. Even more so, remote manipulations performed on a virtual system can be applied to its real-life twin, facilitating maintenance, and cutting travel costs. ??

NASA, for example, turned to Mixed Reality to upgrade the Cold Atom Lab, a quantum laboratory aboard the International Space Station, since it was virtually impossible to move an entire team of engineers to perform maintenance?in Space. The same concept is also used to control desalination plants, water purification plants, and wind farms remotely?(7).

Creating experiences

The recent growth of Mixed Reality has encouraged businesses to use it to expand their marketing campaigns and scale their ROIs, as it gives a hands-free product experience to the customers. Using MR headsets, consumers can experiment and interact virtually with a company’s products or services, long before paying. It can also be a powerful tool of product presentation for businesses at trade shows, allowing specialists to perform demonstrations while leveraging the surrounding environment, without isolating the customer from it like VR would. Finally, it can also be used to boost productivity by creating work environments and workflows that would be impossible to achieve in real life. ??

For example, Meta introduced Infinite Office, a collection of new Mixed Reality capabilities designed to create a customized virtual office space that allows you to have infinite screens at once, if you’d like?(8).

Not so long ago, we were still in the early phases of MR discovery. But now, it’s outgrowing its introductory phase, and has moved into developing the stability that’s needed for widespread use.

As we’ve seen, MR applications are slowly emerging into various sectors; but this is just the tip of the iceberg. As development continues on software and hardware, it seems logical to expect even further integration of this tech into our lives, helping us figure out a better way of doing things. Distance, language barriers, financial constraints, safety concerns — these previously challenging barriers vanish with the help of Mixed Reality. The glimpse we’ve had so far has had a drastic impact on many sectors; so, think about what might be possible in the not-too-distant future! ??

Through StellarX, we’re aiming to democratize MR, as we so successfully did with VR. We believe that we are poised to give businesses and individuals alike the tools they need to better themselves, streamline their operations, and exercise endless creativity. Immersive technologies shouldn’t be put up on a pedestal; or rather, if they have to be, everyone should be able to jump on it with them!

In the coming years, we expect we’ll get to see how everyone uses Mixed Reality in unison to work, play, and communicate their ideas. Limitless prospects, limitless results! And, in the meantime, if you wish to jump on the bandwagon — don’t hesitate to?check StellarX out , or to?hit us up ?for a bit of help. We can even?schedule a demo ?if you’d like us to show you how it works. We are glad to deliver technology, expertise, creativity, and services to help you unleash the full potential of immersive technologies! ?

Nous sommes une entreprise fièrement québécoise. Pour lire cet article en fran?ais, c’est?par ici !

_____________

1, 5: Girnyak, M (2021).?Mixed Reality Examples in Business, Gaming, and E-learning, Post Industria. URL:?https://postindustria.com/mixed-reality-examples-in-business-gaming-and-e-learning-ar-commerce/

2, 4:?Top Mixed Reality Applications Today, Lanner. URL:?https://www.lanner-america.com/blog/top-mixed-reality-applications-today/

3: Dearborn, Mich (2017).?Make Way for Holograms: New Mixed Reality Technology Meets Car Design as Ford Tests Microsoft Hololens Globally, Ford. URL:?https://media.ford.com/content/fordmedia/fna/us/en/news/2017/09/21/ford-tests-microsoft-hololens-globally.html

6: Mattoo, S (2022).?What Is Mixed Reality? The Better Side of Technology, G2. URL:?https://learn.g2.com/mixed-reality

7:?An Eye to the Future: 5 Innovative Applications of Mixed Reality, I’m Novation Hub. URL:?https://www.imnovation-hub.com/science-and-technology/innovative-applications-mixed-reality/

8:?Infinite Office?(video), Meta Quest. URL:?https://www.youtube.com/watch?v=5_bVkbG1ZCo