Being There

Your smartphone is about to become obsolete.

As a tech strategist, I explore the evolution of technology and how it will change our world. In this article I’ll explore how emerging technology will enable us to “be there” through virtual teleportation.

The term “Zoom Fatigue” entered our lexicon over the last year as people adapted to a world where meeting by video became the predominant way of connecting with colleagues, friends, and family. It’s hard to be on camera all day, especially because social conventions require participants to be continuously focused and engaged with each other. As with in-person conversations, you can’t zone out without appearing rude. This “always on” experience is draining, especially for introverts.

Take a moment, however, and consider how remarkable it is that we can communicate face to face with anyone across the planet in real-time with high-quality audio and video. We can see each other, share our environments, and hear sounds half a planet away. And we can do that anywhere there’s good network bandwidth, with a device you can hold in one hand. This is only going to get better as connectivity and bandwidth improves with the roll-out of 5G wireless networks and satellite broadband. This is a remarkable tectonic shift in the quality of distance communication and collaboration, even if Zoom over-use has become an issue.

Where will communications technology take us next? Today, we can see and hear “there”, but what if we could also “be there”?

It's time for teleportation

Star Trek offered the promise of a transporter that enables people to beam themselves physically from place to place. Although technology has either caught up or surpassed many Star Trek innovations, it has yet to solve the teleportation problem. Unfortunately, I don’t expect that will be possible in the next century, if at all. So, being there, means traveling there. That might be as short as a walk down the street, or as far as a day-long flight across multiple time zones. What if we could bring “there” here, or transport ourselves “there” in a second?

I got a sense of what teleportation might be like when I was working on Open Wonderland, a 3D virtual world my team developed at Sun Labs. We’d developed an innovative real-time, multi-channel audio mixing service for a previous project and integrated it into Wonderland. This added spatialized, immersive, CD-quality audio. As you walked around the virtual world, sounds were attenuated with distance, and spatialization enabled you to tell where the sound was coming from directionally, just like in the real world. I remember being in-world and spinning around in my chair when I heard my phone ring…except it was someone else’s phone who happened to be standing behind me in the virtual world, but 3,000 miles away in the real world.

While working on Wonderland, I was mentoring an engineer in Sun’s Beijing office. One week we decided to try hosting our mentoring meetings in the virtual world. When I “teleported” in, I suddenly felt like I was in her office in Beijing. I could hear the street sounds of the city and her voice as clear as if I were physically there. It was a remarkably immersive experience. As the cliché goes, you could hear a pin drop. Phone audio has been a 12 year-long disappointment ever since.

Being there acoustically is a solvable problem today; you just need two or more microphones and the ability to spatially encode and decode the audio stream. And with AirPods our experience is binaural, just like the real world.

We can “be there” acoustically, but what about visually?

If you’ve experienced VR, you’ll understand just how immersive the experience can be. As my friend Brian Jackson noted:

“I had one of the strongest moments of “presence” in VR in years. Last night in the new Oculus Home app, I was watching YouTube on a virtual TV in my personalized space. I took off my headset and it was like a cold smack. My brain just forgot that [it] wasn’t really “my room”.”

VR applications enable virtual visits to synthetically generated environments, or 3D models of the real places such as Machu Picchu. That’s great, but it’s not the real world in real time.

Realistic teleportation requires real-time video streaming and freedom of motion. For the experience to feel real, requires the ability to look around and move at will in any direction. To make this concrete, imagine teleporting to Disneyland, walking around the park, and going on rides…remotely.

A 360-degree camera can stream all directions simultaneously, and with head motion tracking, you could look around the remote location as if you were there, albeit only where the camera is positioned. If you wanted to move around, you’d need a way to move the camera. To really feel like you’re there, your camera would need complete freedom of movement. So, let’s make the camera fly! Now you’re inside a drone with the ability to move anywhere you want. Fly off the North Rim of the Grand Canyon? No problem! Dive into the ocean to find Nemo? You could do that too, subject to underwater Internet connectivity!

What this requires is a “presence drone” that you can “dial” into and establish a streaming video and audio connection to a remote location. Since the drone has freedom of motion, a 360-degree camera might not be required, but to minimize motion sickness, such a camera would minimize the look-around-latency and hysteresis of rotating the drone. The Door Robotics VISTA drone has embedded 360-degree cameras and already claims to support live VR streaming.

Equipped with a VR headset with immersive audio and 360-degree streaming video, you could teleport and really feel like you were there. Your ability to travel would only be limited by the availability of presence drones.

Let’s say you wanted to have coffee with your cousin in New Zealand. As long as they had a presence drone, you could just “call” into it and, voila, you’re right there in their living room.

Unfortunately, your cousin’s experience would be of a drone hovering in front of them. Ring’s Always Home Cam is a preview of what they might see (the device on the left).

That’s not great; the fidelity of the experience ought to be symmetrical. Each person should feel like the other person is equally “there” with them. For believable virtual co-location one of the participants (the “visitor”) would teleport into the other’s location (the “visited”), and the visited person would remain in their world. If both teleported, they’d swap worlds, and not be present in the same place at the same time. Just like with travel in the real world, someone has to travel. Alternatively, both could travel to a third place, “I’ll meet you in Florence, by the Duomo, at noon”.

Each person would need a presence drone to live stream themselves to the other participant.

To complete the immersive experience, they would need to see the other in their environment. The visitor would see the visited person in their environment via live streamed video without any reality augmentation. The visited person would see the traveler projected into their environment, standing virtually in front of them, like this:

While a VR headset would enable the visitor to experience the remote location with a high-degree of immersion, the visited person would see the visitor wearing a brick on their face. Making both participants wear headsets would just make the experience worse, creating a headset-enforced disconnect in the experience.

I don’t expect VR headsets to miniaturize to the point where they’re feasible for use for the teleportation use case in the next 5 years, but we are going to see that happen with AR, with ever lighter, more sunglass-like, designer glasses that project video directly onto your retina. With that type of rendering, it will be possible to swap out the real world you’d normally see, with another equally believable view of a remote location or synthetic world. Direct-to-retina rendering isn’t sci-fi, it’s already in development.

Implications of “Being There”

If you could teleport anywhere in the world and really feel immersed in that environment, what would become of travel? Why would you get on a plane, spend all that money, and deal with all the ever increasing hassle of security and pandemics? Not to mention the CO2 impact on the planet of flying. You could experience anywhere without leaving your home: lunch on a beach in Maui today; an afternoon walk in a rain forest between meetings tomorrow; a late night coffee in a Vienna coffeehouse listening to a live Jazz performance with your cousin from New Zealand who teleported there with you. The world would finally be your oyster.

What would happen to destination venues like theme parks, music concerts, theaters, sports stadiums?

Venues are still needed as “sets” where a concert, play, sporting event, or other in-person entertainment experience can be staged. With “being there” technology, however anyone could teleport to experience it. In fact, everyone would get the best seat in the house. No doubt event organizers will find a way to charge for premium “presence locations”. Overall, entertainment will be more immersive, more accessible, and incredibly more convenient.

During the pandemic, I was hoping Disney would offer 360-degree live streamed video tours of Disney Parks. Who wouldn’t pay $20 to visit Disneyland with a Disney tour guide and ride the attractions of their choice? Sadly, Disney missed that opportunity, and Disneyland remains shuttered.

Looking further ahead, the fate of venues becomes more questionable. Sim motor racing means you don’t need real world racetracks. Movies are being filmed with backdrops rendered using gaming engines, so you don’t need physical sets, except as a place to put human actors, but they’re also at risk as virtual beings become increasingly realistic. Sports venues are perhaps least at risk, although if the whole crowd is remote, you won’t need seating. As for theme parks, they're perfect augmented reality venues, but also vulnerable to VR.

The demise of smartphones?

I began this article with the statement that smartphones are about to become obsolete. I made this prediction for two reasons:

First, teleportation will replace phone and video calls. This is as significant a tectonic shift as the move from landlines to mobile phones, and arguably will deliver an order of magnitude improvement in experience. You’ll still need some device to broker the communication, but it doesn’t need to be a smartphone.

Second, if a high resolution image can be projected directly on your retina, you no longer need an external display to look at. The presentation surface that’s been with us in various shapes and sizes since the invention of the television in 1927 will be obsolete. And it’s not just the display in your pocket that disappears, it’s that stunning 90-inch TV on the wall. It’s the laptop, tablet, or computer monitor that you’re reading this article on. It’s the dashboard in your car, and so on. Suddenly our worlds will be uncluttered by screens, because our eyes will be our screens. The world will be constantly augmented. Samsung's future AR Glasses Lite show what an augmented display-less world might look like.

Closing thoughts

We’re nearing an inflection point in how we communicate. All of the technologies I’ve mentioned in this article exist in some form today. A proof-of-concept teleportation device could be prototyped now. Like the first mobile phones, it’ll be clunky, expensive, and flawed, but the rate of technological innovation is accelerating. Commercially viable implementations at a consumer-acceptable price point might be 5 years away, with the death of the smartphone within 10 years.

The future communication and entertainment is immersive, and it’s coming to a pair of glasses on you.