Communicating With The Departed

By Russ Banham

Realize magazine

When you’re dead, you’re dead,” actress Marlene Dietrich said in the 1980s. “That’s it.” In the 2020s, this conclusive statement is a bit more nuanced. Through a combination of audio-video recordings, speech recognition software, and machine learning, the departed can return as virtual beings, creating the illusion they’re still alive and well.

Not everyone will be delighted at the prospect of conversing with the ghostly vestiges of late friends and family members, perceiving it to be a bit—creepy. This possibility, however, hasn’t curtailed the development of technology platforms that provide intimate “discussions” and other experiences with people no longer with us.

Some platform developers have created digital avatars that resemble actual people, book, and movie characters—and even oneself, a clone that can spend time with the kids while you’re at play or work. Others are providing the opportunity to “converse” with a former CEO whose business experience and expertise were preserved on video before the executive passed. Most of the platforms offer a way to continue relationships with people who have transitioned into another realm of being or nothingness, depending on your view.

Creepy or comforting? Michael Graziano, Ph.D., professor of psychology at Princeton University where he heads the Princeton Neuroscience Institute, leans toward the latter. “People forget the pyramids in Egypt weren’t just monuments. They were funerary complexes for people to remember the dead and reflect upon their lives,” he says. “This is just another iteration of something that ahs been going on for a very long time.”

NICE TO MEET YOU, JACK

The marketing maxim of HereAfter sums up the company’s mission: “Never lose someone you love.” James Vlahos, co-founder and CEO, had a basic understanding of artificial intelligence (AI) when he learned his 80-yearold father was dying from stage IV lung cancer in 2016. Vlahos created a software program called Dadbot to “converse” with his avatar likeness.

HereAfter, developed in 2019, builds upon this platform, capturing a person’s oral history while they are still alive and then converting it into an interactive chatbot accessible through smart home devices. Upon the person’s passing, friends and relatives can ask the chatbot a question and hear a reply in the interviewee’s actual voice.

“Hey, great-grandpa, what was it like in L.A. in the years before cars were self-driving?”

“Well, kiddo, we fried on the freeways stuck in traffic.”

Vlahos soon started brainstorming more extensive uses for the chatbot. “[I was thinking] about expanding the market for HereAfter, from stories that have family interest to a bot with more universal appeal, such as the oral history of someone important in a particular field,” he says.

That “someone” might be a renowned scientist, top athlete, or great business leader like Jack Welch, the formidable CEO of General Electric (GE) in its heyday, Vlahos explains. Assuming Welch had submitted to a series of interviews on his business challenges and triumphs while leading GE, this information could feed the development of a virtual bot that current and future employees could query and otherwise chat with. “‘Bot Jack’ would respond in Mr. Welch’s actual voice with very specific and useful advice on business situations,” Vlahos says.

As a person’s stories are collected, they are annotated—essentially labeled as different data. Using word-recognition software and machine learning algorithms, the person’s words and phrases in the interviews are correlated to words and phrases asked in the employee’s or loved one’s question. “We plan to market our first offering to families and then to prominent individuals,” Vlahos says.

Hossein Rahnama, Ph.D., a visiting professor at the MIT Media Lab, has built a platform with a similar objective referred to as Augmented Eternity. The platform is a spin-off of his doctoral research, which resulted in the creation of Flybits, a customer experience platform for the financial sector. Augmented Eternity focuses on understanding the context of large data sets to personalize information for an end user. “The medium of delivery can be a conversational interface like a chatbot, a voice assistant like Siri, or a set of stories or narratives,” says Rahnama, also an associate professor and director of research and innovation at Toronto’s Ryerson University.

Once stories are collected on the subject’s experiences, they are mapped into a “decision tree”—a tree-like model of decisions and possible outcomes. “By mapping the person’s stories into a taxonomy, we can align them with questions involving similar semantics,” Rahnama says.

The decision tree grows as more experiences populate the “limbs.” “The responses of the avatar can be useful in guiding a decision, but are not to be confused with an actual decision,” says Rahnama. “We see our work on Augmented Eternity and Flybits as a decision support system, not a decision-making one.”

I AND ME

Like Vlahos, Marius Ursache was motivated to develop his platform by the death of a loved one: his grandmother. Ursache regretted that he hadn’t spent as much time with her as he would have liked, and retained only a few fond memories and photographs. He created Eternime so others could preserve fuller stories of their late loved ones.

To do this, the company collects data on a person’s digital footprint—words, photos, videos, texts, and content captured online and through social media. The raw material is digitized to create a clone-like avatar, “a ‘Tamagotchi’ version of the user that learns from and grows with the user throughout the person’s life,” Ursache says, referencing the once-popular electronic toy that displayed the digital image of a creature that had to be looked after by its owner like a real pet.

By creating a mirror version of oneself, the clone can observe one’s actions and behaviors while still alive, illuminating personal failings and suggesting ways to improve. It also could perform certain chores “like advising people and talking with journalists,” Ursache says. “We all have a dream of cloning ourselves to be able to do more than time allows.”

LUCY IN THE SKY

Fable Studio has created a virtual being to provide similar assistance, instruction, and company. The avatar’s name is Lucy, and she is based on a character with the same name in The Wolves in the Walls, a picture book by author Neil Gaiman and illustrator Dave McKean that teaches children about irrational fears.

Children who have read the book strongly identify with Lucy, says Pete Billington, co-founder of Fable Studio. Billington wants to extend this relationship. “Our heritage is storytelling,” he explains. “We’re fascinated by the opportunity to create ongoing relationships with virtual characters over long periods of time who learn, grow, and teach us as we teach them.”

Wearing lightweight virtual reality (VR) goggles, a child learning a song on the piano would “see” Lucy sitting next to her on the bench. If the girl hits a wrong note, Lucy would point out the mistake. If she plays a song well, Lucy would applaud. Over time, the relationship blossoms.

AI is the nucleus of this relationship. Fable embeds biographical information about the user into the platform. Each time the user and Lucy interact, the platform captures the experience, cultivating the relationship. The bond is further strengthened by having Lucy reach out to the user through text, email, social media, and voicemail. The company plans to release an early version of its Lucy VR experience by the end of the year. A premium version will follow.

While Lucy is a fictional character, Billington says the same platform also could be used to create a virtual clone of a family member to communicate with. “If grandma passes away, she could come back as a hologram in avatar form,” he says. A relationship that seemingly ended persists as the user ages.

What if someone wanted to commune with a really long-lost relative? Assuming images of the person remain, the visual aspects are attainable. As for the audio, a January 2020 New York Times article cited the use of a CT scanner by scientists at the University of London to create a 3D-printed version of the vocal apparatus of a 3,000-year-old mummified Egyptian priest. Using an electronic larynx, the scientists reproduced a single vocal sound, paving the way to reanimate some people’s voices.

Marlene Dietrich might disagree, but Graziano says there’s nothing “inherently wrong or unethical” in wanting to communicate with the departed. “We have a vigorous culture ‘out there’ of people who have died, but remain online, their photos and comments still on Facebook, Instagram, and YouTube,” he explains. “Since you know the person is gone, you accept the virtual equivalent for what it is—a comforting vestige.”

Russ Banham is a Pulitzer-nominated journalist and author who writes frequently about emerging technologies.