Could AI enable you to speak with the dead in the Metaverse?

All technological pillars for building a solution that can enable communication with a virtual clone of a deceased person already exist. The right question is not whether but when this would become a reality.

Why it matters

The ability to communicate with deceased people in the metaverse through AI can bring closure, comfort, and advice to individuals, as well as wisdom, excellence, and enlightened problem-solving to society. However, significant technological, social, ethical and consciousness-related questions and challenges remain.

Introduction

Have you ever wondered why anyone would seek to communicate with the dead? First, the psychology of loss will be discussed to answer this question before presenting different means of seeking contact with the deceased. Next, the technological underpinnings of the potential for communicating with the dead in the metaverse through AI will be presented. Afterwards, the personal and collective use cases will be outlined. Finally, the challenges to realising this vision and potential future directions will be presented.

Psychology of loss

Losing someone we love can trigger an intense emotional experience called grief [1]. The word derives from the Latin ‘gravis’, which translates roughly to ‘a heavy burden’. It expresses itself in a variety of symptoms [1]:

• Physical – loss of appetite, insomnia, fatigue, aches and pains, suppressed immunity
• Cognitive – dissociation, disbelief, confusion, poor memory, lack of focus
• Emotional – isolation, depression, guilt, anger, hostility, anxiety, despair, hopelessness
• Interpersonal – withdrawal, distance, resentment
• Lifestyle – failure to perform, self-care neglect, resignation

Naturally, these symptoms can have far-reaching negative consequences for the body and brain of the grieving person. Consequently, both personal and professional relationships may suffer. This, in turn, could bring financial difficulties, triggering further anxiety. Ultimately, the death of a loved one can have devastating effects on people in their immediate circle triggering negative downward spirals.

Speaking with the dead

Some people want to limit the damaging effects of loss or to find closure by seeking contact with the deceased.

Human mediums

An old and widely used method of contacting deceased people involves readings from mediums. In fact, the growing public interest in the field, especially among bereaved persons, led to a scientific study examining the matter [2]. This preliminary 2011 study identified one particularly gifted medium whose performance in the scientifically controlled study consistently met the criteria for statistical significance that cannot be explained by randomness.

More recently, a 2021 meta-analysis of the scientific evidence demonstrated that some mediums can retrieve information about people who’ve passed away through yet unknown methods [3]. 14 papers with a total of 18 experiences were included. Despite controlling for publication bias and after rigorous statistical testing, the results yielded a value of .18 (95% CI = .12 - .25) above the chance level, which is significant.

However, many still reject the notion that such communications are possible. Moreover, even if they were, they lack immediate contact with the deceased person as they occur through an intermediary, the medium. Therefore, some people choose alternative options.

Technological means

The idea of being able to communicate with digital versions of people who passed away is not new. Existing solutions include:

• California-based HereAfter.ai interviews people about their life and lets loved ones hear their stories virtually after they pass away [4].

• You, Only Virtual (YOV), based in Los Angeles, creates a versona, a “virtual persona” – an authentic virtual representation of one’s personality. YOV uses artificial intelligence (AI) and machine learning (ML) to recreate the communication dynamics between you and your loved one, who’s no longer alive through chat messages [5].
• Another U.S.-based company, StoryFile, also uses AI to help people communicate with their departed relatives [6]. However, it only plays back pre-recorded responses and doesn’t generate new ones.?

• Israelian company, HourOne.ai, enables you to create a digital clone of yourself that can speak on camera without input from live humans [7]. However, the use cases here are primarily commercial and communicating with a deceased one is not mentioned.
• Amazon’s Alexa can now recreate the voice of a relative who passed away and read you a book, for example [8].?

• And in 2020, Microsoft patented a system using written, audio, and visual inputs to create a chatbot that can replicate the communication style of a real person [9].

Although some of these, like Microsoft’s solution, date back to 2020, they only started receiving press attention in 2022, as evidenced by the dates of the videos.

Putting ethics aside, for now, all these solutions have inherent limitations:

• They are only text-based (e.g. YOV, Microsoft’s solution), which is a very limited experience of someone’s presence.
• They are pre-recorded (e.g. StoryFile, HereAfter.ai), which makes them very static and limited.
• They mimic a person’s voice or appearance (e.g. Alexa, HourOne.ai) but capture little of their personality.
• They all lack the deep immersiveness that 3D, metaverse-like experience can offer.

Consequently, two dimensions emerge that need improvement:

• Behavioural resemblance – Thinking and feeling like the real person.
• Experiential immersiveness – Looking and sounding like the real person.

Technological enablers

In this section, I’ll explore how several technologies could converge to provide hyperrealistic virtual experiences with digital replicas of people we knew.

Artificial Intelligence (AI)

Advances in AI and deep neural networks support the growth of synthetic media. This artificially and automatically generated photo, video or audio content is highly believable and “true to life” [10]. These technologies gave rise to the so-called “deep fakes” that could impersonate anyone. This short video summarises the process and demonstrates the believability of deep fakes.

However, the example above demonstrates relatively static, pre-recorded deep fakes. More recent advancements in AI have made it possible to create a digital clone of a real person. You can then use this virtual human to generate endless video content from text input [7]. This example demonstrates a higher degree of AI autonomy where random prompts can be recreated in video format, as demonstrated by YouTube influencer Dom Esposito’s video below.

However advanced, these technologies still lack the critical element of interactiveness required to create a hyper-realistic experience of communicating with a real person. So the search leads us to human digital twins (HDTs).

Human digital twins (HDTs)

Digital twin (DT) is the process of digitally defining and modelling characteristics, composition, performance, and functions of physical entities (e.g. humans, machines, processes) using information technology [11]. HDT refers to a replica of an actual human in the digital world. It is essentially a model or database that records current and historical data of humans through smart sensors (e.g. wearables, smartphones, GPS). It then transfers the data to the digital world, updates the database according to recorded information, and then analyses it to extract meaningful insights [12].

HDTs have many use cases, including medicine development, fitness management, and industrial production [11]. Moreover, selecting the most appropriate data is essential to design a useful model. Human-related data can be divided into five categories: external (gender, age, size), internal (e.g. heart rate, blood pressure, temperature), behaviour (e.g. lifestyle, diet, sleep habits), social interaction (social media time, speech patterns, mannerisms), environment (e.g. temperature, humidity, cleanliness) [11].

The most important categories for creating a hyper-realistic and interactive digital human clone seem to be:

• External data – to create an accurate visual representation of the person you want to communicate with.
• Behaviour data – to recreate the thinking and acting patterns of the real person.
• Social interaction data – to enable the virtual clone to interact with you similar to the real person.

All these data types then feed into human behaviour modelling in the virtual world that approximates human behaviour across all aspects as closely as possible. Specifically, activity modelling focuses on physical activities, and social interaction modelling aims to simulate classic human mode physical communication in the virtual world [11].

But the critical question is – how accurately can we predict human behaviour based on models?

Behaviour prediction

Over a decade ago, scientists sequenced a set of dynamic models to achieve 95% accuracy in predicting human behaviour and subsequent actions [13]. Admittedly, this study explored automobile drivers’ actions. More recently, however, scientists used deep learning models to predict consumer behaviour from social media activities with 98.78% accuracy [14]. As a matter of fact, the predictive value of such insights is immense. For example, the recently launched startup Caden.io allows users to monetise their personal data from various apps and services by sharing it with advertisers and receiving a cut of the generated revenue [15].

Additionally, social media data such as content posting and liking behaviour can be used for a deep learning-based method to predict individuals’ personality traits [16]. Specifically, researchers were able to align such predictive models with the Big Five personality profiling framework. Importantly, a quantitative synthesis of 50+ meta-analyses demonstrated that the Big five personality traits have a robust association with human performance and behaviour [17]. Therefore, let’s theorise that such an accurate behavioural prediction model of a human being can be built and subsequently applied to an HDT. Then, the virtual clone’s behaviour would be an exact approximation of the real person’s.?

Virtual Reality (VR) and the Metaverse

Finally, we need a meeting place for the real human and the highly accurate replica of a deceased relative in the virtual world.

First, a human-computer interface acting as the bridge between the physical and the digital world is critical in enabling real-time interaction between a human and the HDT. Of course, this already exists as interactive technologies such as VR headsets. Second, the metaverse is a post-reality universe, a perpetual and persistent environment merging physical reality and digital virtuality, supporting multiple users simultaneously. It is enabled by the convergence of technologies enabling multisensory interactions with virtual environments, digital objects and real people [18]. As this definition demonstrates, it is the perfect candidate for delivering experiential immersiveness from the perspective of the real person seeking contact with a loved one who’s no longer alive.

Therefore, the technological advancements I’ve already presented make it evident that we have the prerequisites for enabling hyper-realistic interactions between humans and virtual clones of deceased people.

Use cases

Personal

On a personal level, people can choose to communicate with a deceased loved one for a variety of reasons.

• Closure – the discomfort of the grieving process can sometimes be alleviated by saying something to the deceased person you couldn’t communicate while they were alive.
• Comfort – like with the case of Alexa reading in your passed grandmother’s voice, we may want to reexperience the comfort associated with the virtual presence of someone we loved.
• Advice – alternatively, we may want to tap into the wisdom of someone we respected and trusted when faced with important decisions.
• Company – finally, we may just want to spend some time with a loved one who passed away and share some experiences in the virtual world afforded by the metaverse.

Collective

On a group level, interacting with virtual clones of particularly influential people can be desirable for some.

• Geniuses – creating a virtual clone of a renowned artist or scientist about whom we have enough data from their writings or recordings can help humanity by accelerating development in the respective fields.
• Spiritual leaders – imagine speaking with Gandhi or your favourite spiritual figure. For example, someone like Dr David R Hawkins was a prolific writer and was featured in many videos throughout his career [19]. This provides sufficient data for the recreation of an accurate virtual clone that can be a source of inspiration and wisdom for the masses.
• Political figures – complex geo-political situations can be resolved with the help of experienced leaders bringing experience from the past.

Challenges

Of course, the presented vision of the possibility of communicating with a digital twin of a deceased person in the metaverse comes with challenges.

Technology

For a start, data heterogeneity will be a challenge as the information needed to build an accurate model and realistic replica must come in different formats from various sources. Data volume can also be a challenge as a hyper-realistic virtual clone would require loads of information. Moreover, the data processing required to analyse all the information and organise it into reliable models would require a lot of computation power. Data quality and reliability could also be an issue, which fortunately can be addressed with the help of blockchain technology [11]. Finally, data collection can be problematic as well. For example, optimal results could require longitudinal data collection, ideally spanning years, if not decades, of a person’s life to build an accurate posthumous virtual clone. Moreover, the person may be reluctant to engage in such exercise due to the discomfort of continually acknowledging the inevitability of death. ?

Enabling real-time interaction would need to rely on 5G and 6G networks, which enable precise real-time data collection and access. Therefore, both are critical for realising remote real-time interaction. For example, holographic telepresence allows people to communicate with the authentic 3D model of objects to deliver an immersive virtual experience is another essential application of 6G [20]. However, beyond bandwidth, these also require substantial computational power for real-time high-quality 3D rendering, which may prove challenging. ?

Social and ethical

The application of HDT in communicating with virtual clones of people who passed away can be a polarising topic embedded with questions about morality, ethics, safety and privacy. Therefore it is essential to develop robust governance mechanisms and policies to oversee and manage the development and application of such use cases.

For instance, HDT is sometimes seen as digital cloning and, according to some, challenges human ethics and social regulation as outlined in the United Nations Declaration on Human Cloning [21]. Unsurprisingly, the European Court of Justice confirmed the need to protect fundamental rights such as human dignity and integrity [22].

Data privacy and ethics should always be prioritised when designing HDT systems and solutions [23]. However, the sensitive nature of the data used to construct the models and systems makes it a prime target for attackers. Fortunately, federated learning enables local model training and minimises the need for data transfer, thus reducing risks of loss or breach [24].

Additionally, from a moral and ethical viewpoint, some assert that digital resurrection violates the dignity of the dignity and may cause ethical issues. Therefore, researchers argue that it is necessary to build a metaverse embodying moral consciousness where various HDTs can live [25].

Attitudes toward trust in AI vary as well. The European Commission’s High-Level Expert Group suggests that humans should aim to establish trust-based relationships with AI and develop trustworthy AI [26]. Conversely, others assert that AI cannot be trusted because it lacks the emotional and normative account requirements of trust [27]. From these opposing views, researchers have synthesised three pillars of HDT trust [11]:

1. Protecting users’ ethics and privacy rights is a must for establishing trust in HDTs.
2. Improving computing power increases HDT modelling accuracy and raises trust.
3. A simple, operable, and intelligent human-computer interface is essential.

These three can alleviate concerns, reduce risk, and improve control over AI, facilitating greater trust in HDT. Ultimately, in my view, informed consent is likely to remain the most critical consideration. The relatives and the deceased need both to be aware of the risks and benefits and consent if they choose to proceed. However, consent cannot be obtained posthumously, limiting the possibility of digitally resurrecting influential figures that could benefit humanity.

Human consciousness

People’s thinking and decision-making processes often rely on vast stores of subconscious information, which is difficult to abstract, capture, and analyse [11]. Turing and colleagues were the first to propose imitating human thinking through computer programs [28]. Subsequent studies identified the need to consider cognitive stimulation when designing HDT systems that can analyse, interpret and respond to human behaviour [29]. In recent years, scientists have analysed human information processing (e.g. perception and thinking) and applied its principles to designing an intelligent HDT [30]. Ultimately, although science and technology are advancing, there are still many unknowns about human thought and consciousness. This makes its digital cloning an exceedingly challenging task. ?

Moreover, even if an abstracted digital clone could think and behave exactly as the real human prototype, they are likely to be static. For example, even the most emotionally stable person would exhibit behavioural variability based on mood, physical comfort, and other external stressors. Moreover, most humans evolve continually throughout life as each new impression and experience modifies their personality.?

Future directions

Based on everything presented, most of the technological pillars necessary to enable you to speak with a deceased person in the Metaverse via AI already exist. So, the short-term direction may focus on improving the capabilities of data generation, capture, and processing to enable the seamless integration of multiple sources. This would provide the rough output for generating hyper-realistic virtual clones. Additionally, it is essential to enhance significantly on-demand computational capacity. Distributed and/or edge cloud computing seem like strong candidates here [31].

Mid-term, the focus should be on addressing the previously discussed social and ethical issues. Ideally, large-scale studies would first study perceptions on a population level and then design the appropriate regulatory frameworks to preserve all users’ safety, privacy, and dignity.

Long-term, of course, the focus needs to be on an ever-deeper understanding of human consciousness, which will enable its more accurate abstraction and potential extraction into digital mediums. Aspirations for immortality are not new among humans and are detailed well in Yuval Noah Harari’s Homo Deus [32]. Digital resurrection could, in theory, achieve partial immortality but not one that the virtually recreated human can experience. Still, advances in AI and the metaverse could be a checkpoint on the way to actual immortality.

Conclusion

The psychology of loss and its associated grief can have devastating consequences. So, some people seek contact with the dead via human mediums or technological means, such as emerging applications in efforts to alleviate the pain. Recent advancements in AI, HDTs, behaviour prediction modelling, and the metaverse could make it possible to communicate in real time with a virtual clone of a deceased person. Importantly, the use cases for such solutions can bring comfort, closure, and support on an individual level, and wisdom and solution to complex problems on a societal level. However, many challenges must be overcome before this vision becomes a reality. Beyond evolving and, importantly, integrating all the discussed technological fields, profound social and ethical questions need addressing. Finally, the complexity of consciousness makes it exceedingly difficult to abstract and recreate outside an actual human being.

Ultimately, let me clarify that, in my view, this is not about playing god. Instead, it is a radical act of honesty acknowledging our humanness and the pain we feel when we lose a loved one. At the same time, it is a celebration of our courage and ingenuity in using technology to solve previously unsolvable problems.

Sources:

[1]??????W. Smith, ‘The Psychology of Grief: The 4 Stages Explained’, PositivePsychology.com, Jun. 20, 2021. https://positivepsychology.com/grief-stages/ (accessed Dec. 21, 2022).

[2]??????E. W. Kelly and D. Arcangel, ‘An investigation of mediums who claim to give information about deceased persons’, The Journal of nervous and mental disease, vol. 199, no. 1, pp. 11–17, 2011.

[3]??????M. Sarraf, M. A. Woodley of Menie, and P. Tressoldi, ‘Anomalous information reception by mediums: A meta-analysis of the scientific evidence’, EXPLORE, vol. 17, no. 5, pp. 396–402, Sep. 2021, doi: 10.1016/j.explore.2020.04.002.

[4]??????HereAfter, ‘HereAfter AI — Interactive Memory App — Try Free’, 2022. https://hereafter.ai/ (accessed Dec. 22, 2022).

[5]??????YOV, ‘Never say goodbye’, May 19, 2022. https://www.myyov.com/ (accessed Dec. 22, 2022).

[6]??????StoryFile, ‘StoryFile - Video SaaS Technology for Education and Business Solutions’, StoryFile, 2022. https://storyfile.com/ (accessed Dec. 22, 2022).

[7]??????Hour One, ‘Make AI Videos To Train Anyone or Explain Anything | Hour One’, 2022. https://hourone.ai/ (accessed Dec. 22, 2022).

[8]??????Amazon’s Alexa will soon be able to mimic family member’s voice, (Jun. 24, 2022). Accessed: Dec. 22, 2022. [Online Video]. Available: https://www.youtube.com/watch?v=JAAYmPmGgA8

[9]??????S. Quaglia, ‘Amazon’s new invention could let you “speak” to the dead — but should it?’, Inverse, Jul. 29, 2022. https://www.inverse.com/innovation/amazon-alex-ai-dead-relatives-chat-bot (accessed Dec. 22, 2022).

[10]????L. Whittaker, T. C. Kietzmann, J. Kietzmann, and A. Dabirian, ‘“All Around Me Are Synthetic Faces”: The Mad World of AI-Generated Media’, IT Prof., vol. 22, no. 5, pp. 90–99, Sep. 2020, doi: 10.1109/MITP.2020.2985492.

[11]????Y. Lin et al., ‘Human Digital Twin: A Survey’, 2022, doi: 10.48550/ARXIV.2212.05937.

[12]????W. Shengli, ‘Is Human Digital Twin possible?’, Computer Methods and Programs in Biomedicine Update, vol. 1, p. 100014, 2021, doi: 10.1016/j.cmpbup.2021.100014.

[13]????A. Pentland and A. Liu, ‘Modeling and Prediction of Human Behavior’, Neural Computation, vol. 11, no. 1, pp. 229–242, Jan. 1999, doi: 10.1162/089976699300016890.

[14]????N. Ali Hakami and H. A. Hosni Mahmoud, ‘The Prediction of Consumer Behavior from Social Media Activities’, Behavioral Sciences, vol. 12, no. 8, p. 284, Aug. 2022, doi: 10.3390/bs12080284.

[15]????M. Graham, ‘This New App Wants to Pay You to Share Your Data For Advertising’, Wall Street Journal, Dec. 20, 2022. Accessed: Dec. 21, 2022. [Online]. Available: https://www.wsj.com/articles/this-new-app-wants-to-pay-you-to-share-your-data-for-advertising-11671490542

[16]????J. Sun, Z. Tian, Y. Fu, J. Geng, and C. Liu, ‘Digital twins in human understanding: a deep learning-based method to recognize personality traits’, International Journal of Computer Integrated Manufacturing, vol. 34, no. 7–8, pp. 860–873, Aug. 2021, doi: 10.1080/0951192X.2020.1757155.

[17]????E. Zell and T. L. Lesick, ‘Big five personality traits and performance: A quantitative synthesis of 50+ meta‐analyses’, Journal of Personality, vol. 90, no. 4, pp. 559–573, Aug. 2022, doi: 10.1111/jopy.12683.

[18]????S. Mystakidis, ‘Metaverse’, Encyclopedia, vol. 2, no. 1, pp. 486–497, 2022.

[19]????Veritas Publishing, ‘David R. Hawkins – Live your life like a prayer’, 2022. https://veritaspub.com/ (accessed Dec. 23, 2022).

[20]????A. El Mettiti and M. Oumsis, ‘A Survey on 6G Networks: Vision, Requirements, Architecture, Technologies and Challenges’, networks, vol. 3, p. 4, 2022.

[21]????UN, ‘United Nations Declaration on Human Cloning ’:, Feb. 2005, Accessed: Dec. 23, 2022. [Online]. Available: https://digitallibrary.un.org/record/541409

[22]????J. Truby and R. Brown, ‘Human digital thought clones: the Holy Grail of artificial intelligence for big data’, Information & Communications Technology Law, vol. 30, no. 2, pp. 140–168, May 2021, doi: 10.1080/13600834.2020.1850174.

[23]????M. W. Lauer-Schmaltz, P. Cash, J. P. Hansen, and A. Maier, ‘Designing Human Digital Twins for Behaviour-Changing Therapy and Rehabilitation: A Systematic Review’, Proc. Des. Soc., vol. 2, pp. 1303–1312, May 2022, doi: 10.1017/pds.2022.132.

[24]????M. Adnan, S. Kalra, J. C. Cresswell, G. W. Taylor, and H. R. Tizhoosh, ‘Federated learning and differential privacy for medical image analysis’, Scientific reports, vol. 12, no. 1, pp. 1–10, 2022.

[25]????S.-M. Park and Y.-G. Kim, ‘A Metaverse: Taxonomy, components, applications, and open challenges’, Ieee Access, vol. 10, pp. 4209–4251, 2022.

[26]????S. Thiebes, S. Lins, and A. Sunyaev, ‘Trustworthy artificial intelligence’, Electronic Markets, vol. 31, no. 2, pp. 447–464, 2021.

[27]????M. Ryan, ‘In AI we trust: ethics, artificial intelligence, and reliability’, Science and Engineering Ethics, vol. 26, no. 5, pp. 2749–2767, 2020.

[28]????A. M. Turing, ‘Computing machinery and intelligence’, in Parsing the turing test, Springer, 2009, pp. 23–65.

[29]????H. A. Simon and A. Newell, ‘Human problem solving: The state of the theory in 1970.’, American psychologist, vol. 26, no. 2, p. 145, 1971.

[30]????P. Saariluoma, A. Karvonen, and L. Sorsam?ki, ‘Human Digital Twins in Acquiring Information About Human Mental Processes for Cognitive Mimetics’, in Frontiers in Artificial Intelligence and Applications, 2021.

[31]????Y. Li and W. Gao, ‘MUVR: Supporting Multi-User Mobile Virtual Reality with Resource Constrained Edge Cloud’, in 2018 IEEE/ACM Symposium on Edge Computing (SEC), Seattle, WA, USA, Oct. 2018, pp. 1–16. doi: 10.1109/SEC.2018.00008.

[32]????Y. N. Harari, Homo Deus: A brief history of tomorrow. Random House, 2016.