MR & AI together for a better future

Today I want to explore concepts at the intersection of Artificial Intelligence & Mixed Reality and how capabilities offered by these two innovative technologies can be combined to deliver new and unique experiences in a near future.

Capabilities enabled by Mixed Reality

Mixed Reality (MR) technology enables the blending of physical and digital worlds to create entirely new environments and experiences.

Microsoft views mixed reality as any point on a spectrum between Augmented Reality (AR) & Virtual Reality (VR). On this spectrum, VR offers an entirely immersive digital experience without any representation of the real world, whereas AR overlays digital objects onto a real-world canvas enabling a blended experience. In AR, users require a device, such as a mobile phone (with a camera), or a specialized mixed reality device, such as HoloLens, which blends both digital and real-world experiences to provide an illusion of the two worlds interacting.

The following are some of the key capabilities unlocked by Mixed Reality technologies:

• Ability to blend digital and physical worlds by augmenting real-world experiences with digital artifacts. [Augmented Reality]
• Ability to fully immerse a user in a digital experience that resembles a synthetic version of the real-world. [Virtual Reality]
• Ability to synthetically capture physical world using depth sensing capabilities embedded in a mixed reality device.
• Ability to engage with digital artifacts in Mixed-reality experiences using hand gestures and voice commands.
• Ability to simulate moving of objects or avatars in augmented or virtual experiences based on external data inputs and/or built-in logic or machine learning model-based results.
• Starting with a physical world, an object/avatar can be placed in the digital world. For example, adding an avatar to create an annotation system for asynchronous collaboration, which could guide the user through a complex series of steps for equipment operation.
• Starting with a digital world, physical boundaries, such as walls and furniture, are added digitally to prevent users from colliding into these objects.

Capabilities enabled by Artificial Intelligence (AI)

Artificial Intelligence (AI) is a broader term used for systems that exhibit human like behavior. Machine Learning (ML), which is a sub-field of AI, deals with the science of identifying patterns in data, making predictions, or creating models using data. Deep Learning which further categorizes ML refers to a special type of learning technique that makes use of artificial neural networks and its use is employed in computer vision, natural language processing (NLP), and machine translation. The science (statistics & algorithms) behind machine learning technology provides systems an ability to learn from data without being explicitly programmed. 

The following are some of the key capabilities unlocked by AI technologies:

• Ability to learn from data without being explicitly programmed.
• Ability to use regression or other statistical learning techniques to predict and forecast attributes given enough historic data.
• Ability to learn through trial and error by way of interacting with environment and learning from the experiences (Reinforcement Learning).
• Ability to infuse Intelligence and progressive learning through incoming stream of new data.

AI infusion in Mixed Reality

In industries such as Manufacturing, Energy, or Oil & Gas, technologies like VR or MR may drive higher levels of interaction and the ability to upskill first-line workers. An MR-based application to be fully adopted and functional should infuse Artificial Intelligence (AI) to become an Intelligent Mixed Reality Solution by learning from the available data and then making classifications, recommending, predicting, grouping, segmenting, or sending alerts when an anomaly is detected. 

A MR-based application to be fully adopted and functional, should infuse Artificial Intelligence (AI) to become an Intelligent Mixed Reality Solution

AI and VR/MR be combined in a variety of ways and can be used as building block technologies in solutions to address multiple domain applications. Just think about the inputs and outputs of each technology, each of them are the fueling required by the other: the perfect Yin Yang (i.e., Taijitu Symbol see below) balancing each other.

As an example, here are four scenarios that require a combination of capabilities offered through VR/MR and AI to amplify human ingenuity, competencies, and therefore, our efficiency.

1. A radiologist with VR/MR may obtain a better visual of the organs and the surrounding area in an MRI or CT Scan. A better visualization experience can help the specialist to increase accuracy in the identification of the location of an abnormality. Using deep learning models, AI could help the specialist to segment various parts of the image and identify abnormalities based on existing patterns and experiences.  
2. An aerospace engineer may use AR during an airplane's inspection. AR helps aircraft components to be digitally displayed over the real-world canvas and informing the engineer about the last replacements, inspections, and maintenance history. AI could be used to determine which component is likely to fail next or allow her/him to update maintenance records through hand gestures and speech synthesis.
3. An engineer from a mining or O&G company may use VR/MR to visualize the existing data in a digital replica of the mine - including different vehicles and miners. Adding AI will allow engineers to archive more. As an example, the use of machine learning can help engineers to train models to determine potential route optimizations, run maintenance simulations, identify hazardous situations based on location data, or determine patterns of evacuation in case of emergency. 
4. An AR-based legal research assistant application that allows a paralegal or a document reviewer to view physical documents but still be able to automatically highlight and identify texts or labels that are of significance. This can help expedite the physical document review process without having the documents scanned and pre-processed in digital form.