Extended Reality in Life Sciences and Healthcare

Solutions, market moves, and opportunities

Maximilian Bucher, Jonathan Müller, Kjell Meershoek, Sebastian Wurst

The initial hype around Virtual Reality seems over, and for the most part, products have been disappointing. However, Extended Reality (XR) solutions are silently becoming more mature, and innovative use cases are already being explored especially in the healthcare sector. The article at hand outlines the following. The first section gives a general market perspective on extended reality products; the second does a deep dive of Life Science, Pharmaceutical, and Healthcare industry use cases in XR; the third analyzes the use cases based on their estimated time-to-market and market impact; the fourth contains a discussion around the possible quick wins and current questions is conducted; and fifth is a summary and outlook.

Introduction

Extended Reality (XR) comprises both Virtual Reality (VR) and Augmented Reality (AR). In Virtual Reality, the user entirely submerges into virtual surroundings, typically by wearing a VR-headset that projects images to the human eye. Think of the PlayStation VR. Augmented Reality adds virtual elements to real-world imagery. Usually, a camera takes a live picture that is then being analyzed in real-time and enriched with additional information. A prominent example of Augmented Reality is the smartphone game Pokémon Go. XR is not limited to VR headsets and smartphones though. Heads-up displays, bendable displays, or holographic projectors are technologies that can all enable concepts of XR.

One of the key questions for businesses has been for quite a while: Is XR finally mature enough to get tangible value that justify necessary investments? A variety of industry players believe it is. BMW recently announced large heads-up displays in their cars starting to possibly change the concept of car dashboards as we know them. DHL employees are using smart glasses to make their warehouse operations faster and less error prone. And surgeons use XR preparing for complex procedures that allow to mimic real conditions as best as possible.

More and more solutions are becoming available as more and more mature XR products, platforms, and services become available.

Products

• VR Headsets — e.g. Facebook’s Oculus has the Quest and Rift S VR headsets.
• Smart Glasses — e.g. Microsoft has the HoloLens 2 in its shelves, magic leap offers the Magic Leap 1.
• Heads-up Displays (HUDs) — e.g. WayRay creates AR displays that are typically sold as a component in a vehicle.

Platforms

• XR Platforms — e.g. Apple is using their App Store to market a variety of XR apps
• XR Apps — e.g. with Ikea Place you can virtually place furniture in your home and see how it fits

Services

• Boutique Coding & Content Services — e.g. Visualize is a group of film-makers, producers, technologists, and developers and they focus on creating immersive content for XR.
• Consultancy Services — e.g. Accenture offers industry specific end-to-end XR solutions, for example, for immersive learning, the AR connected worker, virtual merchandise, and the XR consumer journey.
• Extended Reality Training — e.g. Osso VR is offering a VR based training for surgeons, sales teams, and hospital staff.

The overall XR market is expected to grow from about 19 to over 260 billion USD over the next 5 years with a CAGR of 77%. The XR hardware products segment has reached 6 million units sold at a CAGR of 17%. Although these numbers have been estimates without the current COVID19 market situation taken into account, the mid to long-term trend clearly indicated that this market has a bullish perspective. Also, some recent startup transactions show an increasing willingness to bet on the growth of this market. So, it’s definitely time for a closer look at what XR has to offer!

XR in Life Sciences and Healthcare

Exploring what people are doing in the Life Sciences, Pharmaceutical, and Healthcare industry, we found a lot of examples from established companies, primarily clinical researchers and pharmaceutical companies, from digital health startups, and from big tech companies venturing into healthcare. The numbers in brackets indicate a specific use case that has been used for further evaluation in the next section.

Established industry players

• Stroke Rehabilitation — Stroke patients have an impaired motion and are thus prone to falls, etc. Rehabilitation with XR provides a motivational, task orientated, and controlled virtual environment that supports the patient’s recovery. (1)
• Physical Therapy — Stanford is currently running a clinical trial to show that XR is supporting the acute rehabilitation due to deconditioning and surgery. The trial focuses on measuring the mobility aspects of patients, but it is also hoped that XR helps with pain management. (2)
• Surgeon Training — A UCLA study found that surgeons trained with VR goggles are 20 % faster and complete 38 % more steps correctly. The surgeons used the Osso VR service. Ultimately, such improvements can save thousands of lives. (3)
• Parkinson’s disease awareness — Abbvie created a VR environment to simulate the effects of Parkinson’s disease. This can be used for disease awareness, as for example family members of Parkinson’s patients can use this to better understand and cope with the disease. (4)
• Migraine awareness and marketing — GSK worked on a migraine simulator to increase the awareness and understanding of migraine patients. They created this as part of a marketing campaign for one of their drugs. (5)
• Medical Meetings — Bayer used XR at one of their medical meeting booths and attendance soar from under 2 minutes to up to 10 minutes on average. Similar approaches may be applicable in the fields of product demonstrations and eDetailing. (6) (7) (8)
• VR for pain management — Cedars-Sinai has conducted a study to investigate the effect of VR-based treatments for relieving pain. The patient group which was treated with regular relaxing and meditative VR experiences has shown statistically significant improvements in pain levels — with some patients now using the therapy regularly at home. (9)
• VR-assisted lazy eye treatment — Researchers of the Jessenius Faculty of Medicine (Slovakia) have used a VR-based computer game on a modified Oculus Rift headset to treat Amblyopia, commonly known as “lazy eye”. (10)
• XR-controlled process adherence — Medical animation studio Random42 is proposing options how AR and VR can be applied in Pharma manufacturing to visualize manufacturing processes and enable remote monitoring of process adherence, e.g. of bioreactors. (11)
• VR-supported protein assessment — Novartis researchers are using VR headsets to immersive themselves into protein structures in so called “molecule walks”. These visual assessments support them in gaining a more natural understanding of the molecular structure and interactions and hence facilitate the design of new drugs. (12)
• VR-assisted clinical trials — In a recent study published in the Interactive Journal of Medical Research (IJMR) researches investigated the use of VR in clinical trials. In their study, they were able to demonstrate how supplementary VR experiences had a significant effect on patient’s likelihood to follow their treatment regimens. With patient compliance remaining an important influencing factor for the outcomes of clinical trials, these findings certainly indicate further potential. (13)

Startups

• VR-supported ocular-motor impairment diagnostics — Syncthink is providing an FDA approved, VR-based solution to conduct assessments of patient’s ocular-motor coordination abilities and support related diagnoses. (14)
• VR-controlled surgical robots — Bill Gates invested into a MIT-originated robotic startup that uses VR for surgeons to control tiny surgical robots. This novel idea to potentially lowers cost of existing surgical robots and could thus make these widely available. (15)
• Alzheimer’s disease detection — Altoida basically provides a VR-based hide-and-seek game that can detect Alzheimer’s disease early. If Alzheimer’s is detected early, current available drugs can be used effectively to stop the progression of the disease. (16)
• Workforce training — There are multiple startups focusing using XR for training. The ability to control and augment the visual environments provide powerful training scenarios. STRIVR provides trainings that already have been widely used at Walmart. (17)
• Training for healthcare professionals — Surgeon, medical students, and many more require extensive training for their work. Osso VR and GIGXR are two startups that have a focus on the healthcare market. (18)
• XR-supported lab operations — Apprentice.io is offering augmented reality solutions for laboratory environments that include voice guided instructions, technique demonstrations, or hands-free data capture. (19)

Big Tech

• Microsoft — Not only is the HoloLens hardware production a backbone of Microsoft’s XR strategy. They also invest into startups and focus specifically the medical market.
• Facebook — Continues to develop their Oculus VR headsets from a a hardware and software perspective. Additionally they have an eye on education and training in healthcare.
• Google — The Smart Lens projects has been discontinued recently — despite a partnership with Novartis. What will be next for them?
• Apple — Clearly their AR platform is one of the strongest and is not only focusing on games. Tim Cook recently announced that AR can help to transform healthcare.
• SAP — SAP has integrated AR into its warehouse management systems to provide an AR-assisted warehouse picking solution. (20)

Use case evaluation

The examples illustrate the prominence of extended reality applications within the life sciences industry. Being mindful of the limited resources, both financially and organizationally, life science and healthcare players face, we need to make sure the investments into XR applications are properly steered and prioritized. Assessing what quick wins could be and which applications might be more relevant for a long-term strategy requires a thorough analysis of the opportunity space. We choose to tackle this analysis by mapping the use cases listed above into a matrix consisting of the stages of the pharmaceutical and healthcare value chain on one hand, and the expected time to market on the other hand.

Use cases for XR in the Pharma and the Healthcare industry along a typical industry value chain

When looking at the set of use cases, 5 clusters emerge:

1. Intelligent Operations (XR use cases around manufacturing and supply chain operations): Several use cases have already been widely adopted across different industries, such as AR-assisted warehouse operations, e.g., pick & place, or XR-assisted workforce training, e.g., specific manufacturing procedures, safety measures in labs. Further potential is in XR-controlled process adherence and deviation detection, e.g., complementing traditional sensor suites in monitoring production parameters batch quality in biopharmaceuticals manufacturing. Use cases in this area are highly mature compared to others, driven from success stories in other industries.
2. Medical Marketing (XR to demonstrate medical products and facilitate communication with healthcare professionals): Effective communication between pharma companies and HCPs is a key lever for sales especially for prescription drugs. XR could be used by both sales representatives and medical-scientific liaisons to facilitate their HCP communications, e.g., via product demonstrations or XR-enhanced e-detailing.
3. Medical Training (XR-based training as a product for medical professionals): The aim is to provide professionals with a better understanding of diseases and conditions, the medical devices they rely on, and options to practice and enhance their skills in real-world-like simulations. These applications have already arrived in the practice and are being increasingly used with very promising results.
4. Patient Treatment (XR used to treat patients). Research and small-scale practical applications have shown promising results in using XR-assisted treatments for certain diseases like Alzheimer’s or stroke rehabilitation stimulating specific areas of the patient’s brain. Similar effects have also been applied in video game therapies such as Akili Interactive’s EndeavorRx which has already gained an FDA approval as therapy regimen for children with ADHD. However, while these developments have been perceived well, the field of XR-supported patient treatment is still in a very early stage of its development.
5. XR-Supported R&D (XR technology in pharmaceutical research and clinical trials): During the pre-clinical phase XR could be used to visualize the molecular structures in of target proteins during in silico studies with valuable insights for the lead generation process. In the clinical phase AR devices have already shown positive impact on therapy adherence and could potentially enable at-home patient observations, reducing the on-premise testing and bringing down costs. Quick adoption in this area seems unrealistic though. While regulatory agencies like the FDA are slowly opening for new technologies like the “Complex Innovative Trial Design” initiative shows, patient safety requires maximum diligence.

Success Stories & Challenges

It has been repeatedly shown that humans take in the XR contents very well, so they work and train more efficient or are more likely to complete a sale. And the medical industry can now build on these successes to establish a foundation for more advanced medical product type use cases, i.e., medical diagnostics or treatment.

However, XR hardware has room to improve. The headsets are often costly, heavy, bulky, and simply feel unnatural. Current state of (technical) development, clearly limits mass adoption of XR headsets and hence market growth is affected. Although, truth be told, it does make sense to differentiate between B2B and B2C use cases — the B2B ones clearly being less affected by this.

It is also interesting is to draw a parallel between the current XR platforms and the smart phone market a decade ago. There is no open platform development but rather a small group of tech giants providing these platforms and it is unclear which platforms will prevail. Building XR now may mean to remain flexible on the underlying hardware and use agile structures to make strategic moves. Additionally, some sort of a middle ground between “build or buy”, including partnering with technology players and startups, can be seen.

Summary and Outlook

There are signs that the Extended Reality (XR) market is maturing. The overall market size has been growing over the past few years and experts have promising forecasts for the future. Big tech companies, such as Google, Microsoft, and Facebook are developing the hardware and software of XR platforms and they have put an eye on the medical industry. Here, XR offers a variety value adding use cases for the Pharma, Life Sciences, and Healthcare industries. And most importantly, there is already research indicating that some XR use cases can improve current school medicine procedures, thus saving lives!

The above use case analysis investigated areas along the medical value chain, where XR provides value and an analysis of the time a company typically needs to reach the market with XR. The areas that offer quick results due to efficiency gains are Intelligent Operations, Medical Marketing, Medical Training. The use case areas with slightly higher (regulatory) barriers are XR supported R&D and Patient Treatment. Promising research results and startups at least indicate that the end of the trough of disillusionment is near. And all this is happening despite that current XR hardware is often not user friendly, bulky, and expensive.