Transcending the Screen: Why Virtual Reality Depends on Volumetric Video

With Ready Player One making its theatrical debut over the Easter weekend, many audience members must be asking themselves how long it'll take for real virtual reality to reach the level of sophistication depicted in the film. And also, will everything in VR look video gamey, like in Spielberg's movie?

As it happens, the short answer to the first question is "not long". The long answer to the second question is the rest of this article, so read on.

It’s amazing to think how little has changed about the format of cinema since the dual revolution of sound and color, beginning in the 1920s. After nearly a century of technical innovation, we still find ourselves sitting in front of a rectangular screen in a dark room. But if we look for a common thread to all of the technical innovation in film-making over the years, it’s invariably been about increased immersion. That is, making the experience of cinema more fully engaging and closer to the experience of reality.

Filmmakers have always experimented with aspect ratio. From widescreen to IMAX and more recently Barco Escape, the goal has been to completely fill our peripheral vision—just like the real world does. Dolby continues to advance their surround sound technology to enhance the illusion that we, the audience, are physically present in a film’s world. Meanwhile stereoscopic films trick our brains into perceiving the images in 3D, while so-called 4D technology adds physical, haptic cues, to invite more of our senses into the cinematic experience. High frame rate filmmaking is another attempt to craft a cinematic experience, incrementally closer to reality.

Which brings us to Virtual Reality, the ultimate and inevitable extrapolation of this trend. VR promises an entertainment experience that is both fully-immersive and hyper-realistic. But right now, in the early stages of the medium's evolution, we generally have to choose between fully-immersive and hyper-realistic. Video game graphics—like those in Ready Player One—can provide full immersion because they are optimized for interactive playback in real-time. To pull that off, they must sacrifice the fidelity and detail we expect of reality. Conversely, the main approach to filming reality for VR, 360 degree video, falls short of full immersion. By simply letting you look in every direction from a single viewpoint, 360 video offers very little beyond the level of presence we already have in 3D cinema, with the downside of making more people nauseous.

The reason that 360 video isn’t more immersive and the reason it causes nausea is that it is not actually a three-dimensional representation of reality. It’s essentially an optical illusion that has a limited lifespan, as audiences inevitably become more sophisticated. Also, because of the nature of the illusion, it is a format that remains bound to two-dimensional screens. Remember, today’s crop of VR headsets are built around two-dimensional screens. That won’t be the case for much longer. New light field display technologies being developed by companies such as Magic Leap and Avegant, are a fundamental departure from the idea of representing the world as flat images.

Enter volumetric (aka. light field) video. You can think of it as a means of recording and playing back fully three-dimensional live-action scenes, that you can freely move around inside. By being truly three-dimensional, the content is dramatically more realistic and also future-proof. Unlike 360 video, volumetric video contains enough information that it can be played back on the exotic new light field devices mentioned above—and on whatever comes after them.

At the moment, volumetric video falls into two broad categories: outside-in and inside-out, corresponding to the orientation of the array of cameras that are used to capture it. Outside-in volumetric is a way of filming (often a person) with an inward-pointing array of cameras, resulting in a hologram that you can view from different angles. Examples of this include freeD, a bullet-time-type effect, which you may have seen lately in sports broadcasting, as well as the near lifelike holograms produced by Microsoft, 8i and Uncorporeal Systems (for whom I used to work). If you're interested in experimenting with volumetric video yourself, DepthKit is a great place to start.

Inside-out volumetric, on the other hand, is a way of filming an entire scene from a certain vantage point, in such a way that when it’s being played back it's at real world scale and you’re able to move your head around and experience parallax, like you would if you were actually there. This is a natural extension of the 360 degree video format, but it overcomes all of the drawbacks described above. Google recently made two significant moves in this space, sharing some of their own research and apparently acquiring (or acq-hiring?) established light field camera startup, Lytro. Facebook and OTOY have teamed up to tackle the same challenge, while companies like Hype VR and Uncorporeal have also been making strides in the space.

Hybrids of the two different types of volumetric video are relatively trivial, since the data is 3D and can therefore easily be combined—also with fully CG assets. There are currently limitations and trade-offs, both in visual quality and interactivity, as technologies that were intended for other uses are repurposed into prototype volumetric tech. But camera manufacturers are now actively working to create video and depth sensors specifically for recording volumetric video and the algorithms powering photogrammetry, object tracking, optical flow, etc—collectively referred to as machine vision—are advancing on an even quicker iteration cycle. As the hardware and software mature, the extent and fidelity to which reality will be able to be captured and recreated will only increase.

And fidelity is essential to overcoming the "video game feel" and allowing authentic human emotion to be conveyed in virtual reality. There has been some impressive progress recently in interactive real-time CG Characters and it's fair to say that the uncanny valley is narrowing. But we are still a long way from the point where a real-time synthetic human can take the place of a real one. Because of this, I believe that volumetric video will form a critical component in the next decade of advancement in VR.

And with it, a whole slew of new narrative forms become possible. Imagine a whodunnit in VR immersive theater (Murder Mystery meets Sleep No More), in which you, the viewer, are able to control time. You could watch the narrative play out from one point of view and jump to the most obvious conclusion about what happened. Or you could pause and rewind the action, walk around to review it from multiple points of view, to pick up on hidden clues and discover what really happened—knowing that each of your fellow participants may arrive at different conclusions.

Or how about a mashup of Woody Allen neurosis with the Sci-Fi premise of Being John Malkovich? A first person narrative in which you're looking out through the eyes and hearing the stream-of-conscious thoughts of the protagonist. Think of the potential this kind of storytelling has to break down cultural and political barriers between people. We'll be able to evaluate others' actions in the context of their intentions, guided by their experiences and biases, instead of simply judging the outcomes of their actions through the lens of our own experiences and bias.

And speaking of positive impact, just imagine the benefits of using volumetric video to record animals in the wild. Forget zoos, cages and safaris, we could immortalize our fellow Earthlings with minimal impact to them. This would allow current and future generations of humans the opportunity to observe them in their natural habitats and to study their behaviour.

These are just a few of the possibilities that volumetric video enables. It’s taken filmmakers 120-odd years of concerted creativity to establish the conventions and to perfect the art of storytelling in the rectangular confines of a screen. We are now working to establish new conventions and perfect an entirely different kind of storytelling that transcends the screen and volumetric video is one of the keys to the realm.