Health Metaverse

A decade ago, Internet penetration was already significant, while mobile and cloud computing were just getting started and artificial intelligence was still more science fiction than reality. Having a medical consultation without leaving home didn’t sound preferable, effective or even possible. In the last couple of years, however, we were almost forced to adopt telemedicine, and a lot faster than anyone would have guessed. And yet we were able to do it.

Innovation starts with disruption and change comes with fear. So, I would like to begin this essay by asking –and inviting you to ask– where is the Internet going to, and how will it look in the 10 years?

Milestones of the Internet evolution

The digital age comes with its own lexicon, words and acronyms. Many of these new terms have found their way into our everyday vocabulary, although the meanings are often confused or blurred.?

For example, the Web (previously called the World Wide Web. -hence the "www"-), refers to the pages/sites that we see when we get online, while the Internet consists of a series of interconnected computer systems in which the Web works, forming the means by which files and emails are transferred.

The revolutions of the Internet over the years gave rise to the different milestones in the versions of the Web:

• Web 1.0 was a way to access information based on text
• Web 2.0 was a way of bringing people together based on interaction (social networks)
• Web 3.0 is a way of connecting people with places and things through an immersive experience. They can be completely virtual (VR) or part of the real world, with an augmented reality (AR) level. Web 3 is characterized as the decentralized Internet, built on distributed technologies such as blockchain and decentralized autonomous organizations (DAOs), rather than being centralized on private servers owned by individuals or corporations.

The concept of a metaverse first appeared in the 1992 science fiction novel Snow Crash by Neal Stephenson. The Snow Crash characters become avatars and work in a 3D virtual reality called the metaverse.

Specifically, the metaverse today refers to any online space that emphasizes creating immersive environments, a virtual reality that exists beyond reality. It is not a simple combination of the world and virtual reality, but an interaction.

The Acceleration Studies Foundation (ASF), a representative metaverse research organization, announced the metaverse roadmap in 2006. It presented the concept and the 4 types of the metaverse.

Four types of metaverse

They presented 2 axes to explain metaverse types. One is 'augmentation versus simulation', and the other is 'internal versus external'.

Augmentation refers to the technology that allows the addition of functionality to a real existing system. For example, in the metaverse, it overlaps the physical environment that we see with new information. Simulation, meanwhile, refers to technology that provides an unique environment by modeling reality. In short, augmentation and simulation vary depending on whether the information is going to be implemented in physical or virtual reality.

On the other hand, the metaverse is divided into an inner world and an outer world. The inner world focuses on the identity and behavior of an individual or object, while the external world is usually focused on aspects of external reality centered on the user. Therefore, this axis classifies applications based on whether they display information about the user or their environment.

Hence, when we cross these two axes, we get the four types of metaverse: augmented reality, lifelogging, mirror world and virtual reality.

Augmented reality

Augmented reality is a type of augmentation of the external world. It can use GPS or WiFi to provide information about the user’s location to augment already existent information that enhances the real physical world.

In the medical field, several examples of augmented reality technology are emerging. The Virtuali-Tee is an Augmented Reality T-Shirt that enables the exploration of the circulatory, respiratory and digestive systems in fully immersive 360 videos.

Lifelogging

Lifelogging is a type of augmentation of the inner world. People record details of their daily life by using smart devices. There are also services that utilize biometric information stored through wearable devices. This can be used in the medical field, for example to record the amount of exercise or location.

Mirror world

Mirror world is a type of external world simulation where the appearance, information and structure of the real world are transferred to virtual reality as if reflected in a mirror. All activities in the real world can be done through the internet or mobile applications, and a mirror world metaverse is a place that makes life in the real world convenient and efficient.

Virtual reality

Virtual reality is a type of metaverse that simulates the inner world. It is a world where users feel that they are completely in a virtual reality, characterized by simultaneous access and participation of several users. The space, the cultural background, the characters and the institutions are designed differently from reality.

Integration in the metaverse

The metaverse, then, becomes a space where different aspects from virtual reality and the real world combine.?

From a functional standpoint, the metaverse integrates information retrieval, social networking, and gaming elements. From a technical point of view, the metaverse is a complex of virtual reality technologies. Socially, it is a space where members of the digital native generation, as early adopters, leave traces in their daily and economic life.

Initially, the 4 types of the metaverse developed independently. However, it has been evolving towards a new kind of convergence by breaking down boundaries with interaction.

Metaverse and health

In the health field, the metaverse opens up novel applications in telemedicine, virtual care, remote patient monitoring, personal health management, surgical assistance, analysis of patient clinical data, to name a few. In this way, it raises virtual attention from a two-dimensional experience to a three-dimensional one.

Traditionally, the effectiveness of the interaction between health professionals and patients comes from the support of health institutions and centers, while the online platform only acts as an intermediary. With the metaverse, this changes. The key is to build efficient immersive health services in a three-dimensional virtual space applying the knowledge of the medical domain.

Therefore, the health metaverse consists of a virtual health community parallel to the real world but independent from it. It uses multimodal medical information, which aims to promote intelligent medical informatics applications to virtualize and visualize all kinds of medical knowledge and services. So, building a health metaverse requires not only using existing technology, but also crucial knowledge and technologies from the medical field as a foundation for the proper development of virtual professional roles.

Standardization in the metaverse

Virtual reality medical applications require highly accurate multimodal medical information standards in order to ensure the integrity and authority of medical knowledge. For example, disease coding will no longer use text coding like ICD codes, but instead use three-dimensional virtual entities to accurately describe each type of information.

However, a major problem in the construction of these virtual entities is that multimodal medical information is difficult to process and standardize. Important consideration should be given to the creation of multimodal health information standards based on mature and reliable existing ones, so they can gradually become multimodal to carefully verify and maintain the information and ensure the flow of knowledge that meets medical technical specifications.

Social and medical data

Social networks digitalize the roles of participants and integrate all users and their data. The healthcare metaverse connects professionals with patients, using virtualization technology to create an efficient multimodal world. This way, socializing in the metaverse elevates the interaction between medical staff and patients to a new level.

In this scenario, telemedicine and online health management presents an opportunity. It can establish a virtual environment for professionals and patients to communicate in a 3D VR and AR environment, to display patients' personal health conditions in an intuitive display, such as, for example, showing the effects of the virtual body before and after treatment.

Personal health entertainment and socialization is a very promising development direction. Multimodal information from the patient's body is collected and analyzed in their daily routines to provide users with a more comprehensive assessment of personal health conditions.

Furthermore, haptic devices are in an early stage of development and would allow physical interaction with objects in the metaverse. Mixed reality systems allow surfaces to provide tactile feedback. These interactions create a more compelling immersive experience and make selection more precise, in the same way that drawing in air is not as precise as drawing on a physical surface.

Artificial Intelligence in the metaverse

Building large-scale distributed medical AI effectively promotes innovation in clinical decision support today. Patient data such as the user's voice, actions, clinical data, and physiological signals captured by different devices can provide a rich data source for medical AI.

Many types of research today are primarily focused on single-source, single-modality AI-based medical decision-making models. These single-mode AI models are decentralized and cannot be dynamically updated. In the Metaverse, all medical decision support models built will be dynamically updated to form a collaborative decision-making mechanism, providing a virtual world of self-awareness, learning, and collaboration for all types of entities, ultimately promoting the intelligentization of health.

Robotics

The use of robotics in the metaverse can provide an immersive scene, allowing them to enter the virtual body of the patient during the operation and perform high-precision surgeries on subtle parts of the body's organs. It may even become the first training testbed for the next generation of surgical robots.

Challenges in the metaverse

Based on past experience, the healthcare industry moves forward very slowly in accepting, promoting, and implementing emerging technology. Every time a new technology is introduced, the impact of the technology on patients is carefully evaluated. The challenges can involve multiple aspects, such as the technology itself (interoperability, portability, stakeholder customization), human factors (skills, resistance, mistrust, cyber attacks), legislation and regulations.

Working collaboratively with colleagues around the world is a key feature of the metaverse concept and will require high-level communications infrastructure. With people and devices connected massively, there will inevitably be significant security gaps. Protection of the user's privacy and physical and psychological security raises several questions and concerns that involve personal, public, and social aspects. Such risks threaten the personalized nature of the doctor-patient relationship.

In a metaverse that values freedom, the countless amounts of data created and shared by users around the world cannot be censored one by one. Therefore, there is a possibility that it will become a lawless zone. In this case, caution should be exercised, as well as, ethical education will be required to cultivate citizenship in the virtual world.

A well-designed virtual reality environment is essential for a comfortable user experience, as data analysis and visualization can be more intense. The 3D space around the user can be filled with data boards. Displaying such data on a portable and underpowered device is a huge challenge. Likewise, non-spatial data, such as descriptive text, tables, 2D images, and media sources, should also be accessible in the metaverse. AR and VR systems allow the outside world to be viewed and potentially magnified.

Opportunities

Promote research and development of essential technology of the metaverse in health.

More research is required on automated machine learning-based design methods that extract features from boards combined with human processes.

Drive the transformation of medical and health-related service providers.

In the framework of web3, the decentralized Internet is built on distributed technologies and decentralized autonomous organizations (DAO).

Establishment of the digital economic value system.

The existing technology companies are focused on games, entertainment and e-commerce, which are inseparable from the profitable aspects of busines. However, the value system should not be limited to virtual currency, but there should also be an effort to build an evaluation system that reflects the value within the metaverse in health, such as maintaining honor, professionalism, and social contribution in order to establish and maintain a system that is sustainable, effective and transparent for virtual healthcare.

New ways to interact with data

Speech-to-text is the obvious way to enter data instead of typing, but it could also be used to activate features instead of using a menu in the VR environment. The increasing need to analyze complex multimodal biological datasets with multidisciplinary teams from different institutes has become the norm and the metaverse should surely take advantage of that.

On the other hand, there is concern about the generation of content by users, which lacks a factual medical basis, which makes it difficult to satisfy the needs of users in a safe virtual world.

Next steps

The metaverse has been heralded as the next computing platform and the future of the Internet, achieving the integration of the digital and physical worlds, digital and real economies, digital and social life, digital and real identities, and the integration of the digital with physical assets.

While the current Internet is transitioning into the metaverse, eight fundamental technologies provide the means to do so. These include extended reality, user interaction (human-computer interaction), AI, blockchain, computer vision, Internet of things and robotics, cloud and edge computing, and future mobile networks. However, there is still a huge gap to achieve the transformation of the Metaverse in the medical and health field. Existing platforms are still far from an ideal health metaverse.

As data sets grow in size, there's a need for better infrastructure that will enable the collection of patient's data from their daily lives, outside the realm of medical practice. Thus, information is collected and processed in real-time, which doesn't necesarily become apparent in real-world interactions.

Last thoughts

The metaverse is predicted to change our lives and economies far beyond the realm of gaming and entertainment. In fact, we are already seeing how social, cultural and economic activities are moving to this new platform. It has made "social connection" possible for users by providing a place where people with hobbies and interests can meet and communicate even under real-world restrictions like "social distancing" in response to the COVID-19 pandemic.

The metaverse has infinite potential as a new space for social communication. It provides a high degree of freedom to create and share, while providing a unique and immersive experience.

A substantial amount of work is needed to explore how we can transition from 2D to 3D workflows so that 3D interaction reaches its full potential. With access to large, multidimensional visualizations, there is a gap we need to fill to assess cognitive load and human-computer interaction, so that we can ensure that humans are at the center of the systems we build. It will require us to think beyond the current 2D paradigms that have dominated the computer industry thus far, and embrace the advantages that the metaverse can bring.

?

I believe that, for all of the above, this is a key moment. One that offers a great opportunity for health professionals and health informatics, to start thinking on how we want that interaction to be and not wait for it to be all resolved to begin to get involved.

References

Kye B, Han N, Kim E, Park Y, Jo S. Educational applications of metaverse: possibilities and limitations. J Educ Eval Health Prof. 2021;18:32. doi: 10.3352/jeehp.2021.18.32. Epub 2021 Dec 13. PMID: 34897242; PMCID: PMC8737403.

Chen, Donghua and Zhang, Runtong, Exploring Research Trends of Emerging Technologies in Health Metaverse: A Bibliometric Analysis (January 5, 2022). Available at SSRN: https://ssrn.com/abstract=3998068 or https://dx.doi.org/10.2139/ssrn.3998068

Smart J, Cascio J, Paffendorf J. Ann Arbor (MI): Acceleration Studies Foundation; 2007. Metaverse roadmap: pathway to the 3D web [Internet] [cited 2021 Nov 29]. Available from: https://metaverseroadmap.org/MetaverseRoadmapOverview.pdf .

Go SY, Jeong HG, Kim JI, Sin YT. Concept and developmental direction of metaverse. Korea Inf Process Soc Rev [Internet]

Sébastien, D., Conruyt, N., Courdier, R., & Tanzi, T. (2009, May). Generating Virtual Worlds from Biodiversity Information Systems: requirements, general process and typology of the metaverse's models. In 2009 Fourth International Conference on Internet and Web Applications and Services (pp. 549-554). IEEE. https://doi.org/10.1109/ICIW.2009.89

Lee S. Seongnam: Software Policy & Research Institute; 2021. Log in Metaverse: revolution of human×space×time (IS-115) [Internet]. Available from: https://spri.kr/posts/view/23165?code=issue_reports .