The State of VR and AR

Understanding how Immersive and Ambient Technologies are Reshaping our World.

In this article I would like to summarize my point of view on the state of the art of virtual and augmented reality technologies. I have been working in this domain since 2005 and in the last 14 years I have been working on quite a few projects of virtual and augmented reality. While not at all in the gaming industry, my focus has always been the application of virtual reality technologies and the immersive 3D web for enterprises. 

While VR as a technology dates its birth back in the 1960s, after more or less 60 years from its inception, its adoption is still in the early stages. 60 years of “incubation” before becoming widely used, is a lot of time for digital technologies. This is essentially due to one factor: processing power. While in the 60s and even before then, we had a clear vision of what virtual reality could do, only a few organizations in the world could actually afford to assemble the technology and processing power needed to actually make this technology work. In fact it was within NASA that the first VR headsets came to life. If on the one hand, it is not unusual for NASA to incubate new technologies, on the other, the driver behind VR was really being able to simulate the conditions in space, and in particular on the moon, in order to better train astronauts. VR is born as a training technology. It was designed to render a more immersive, realistic simulation to have people learn better, faster and with better results on the job.

This is still its best use-case today.

HMDs

Head Mounted Displays (HMDs) are precisely born in the 60s but it’s really the Oculus DK1 to be the first model to appear on the consumer market in 2012. Still a development version, the Oculus DK1 begun to amaze users around the world, who could for the first time experience a full immersion in a digital environment, like only seen in movies before. Other headsets came right after, needles to say Oculus was not the only company thinking VR. While the Oculus was being acquired by Facebook for a ridiculous amount, in 2016 the HTC Vive was released, a Chinese model that actually was better and more advanced, with included hand controllers, something the Oculus did not have in early days models, but introduced with later versions of the Rift and later with the Oculus Go and Quest. Oculus also partnered with Samsung to produce the first mobile headset. While Oculus was getting the technology to work with mobile phones, Samsung flooded electronics stores and mobile phones shops with their new mobile and phone-paired HMD, the Samsung Gear. A great idea indeed and natural evolution of our smartphones, the Samsung Gear really took for the first time computers (or smartphones) on our heads, marking a fundamental point in the history of screening. This was quite a point as, until then, the only way to experience VR was to buy an HMD headset and a powerful computer. Mobile VR was just the right concept. As computers moved from the IT rooms in basements, to our desks and to our pockets, the Gear HMD - merging smartphones with a pair of glasses - was really the first computer to be worn on our head.

Since then the Oculus, the Vive and other producers such as Lenovo, Sony, Pico, Xiaomi and various others, raced to produce the best headsets, with different approaches and focusing on different markets, for example Sony for gaming or Lenovo for healthcare. Which is the best headset for you really depends on the use and application you need to run in VR, but that is another story…

Today VR headsets have developed into mobile independent units, with processing power and wifi connection, able to run VR application on the go. As computers got smaller and smaller, until they reached our pockets, HMDs natural evolution is also becoming the smallest computer we can wear on our head. And perhaps in the future, inside of it. The Oculus Quest, the latest model from Oculus has marked a fundamental moment in the VR market in 2019. The Quest fulfills four fundamental requirements for this type of technology to go mainstream: it has an accessible pricing, it works independently, it is wireless and mobile, it is comfortable to use. This means that HMDs can now be truly introduced to the market as a new platform for content development and dissemination. This is why 2020 is considered to be the first year of true adoption of VR.

Content

While VR hardware producers are scrambling to get the best HMDs out on the market, the content and software side of VR is really crucial to be able to experience VR in a satisfying and meaningful way. Adding a third dimension to the visuals, to the digital experiences and the internet in general, opens a new territory for creating a new breed of user interactions. 

Content in VR has always been around gaming and some early enterprise applications, like training employees. By studying the VR software market it is interesting to observe how it has first developed in the gaming industry, but has quickly moved into the enterprise world and actually will grow more in the enterprise use, than in gaming in the next few years. Gamers were the first explorers of this technology, but enterprises are the ones that will be making the most out of it in the coming years. In this, content plays a big part. Gamers are used to complex, graphically realistic games with many levels, in which they can play collaboratively and even connect and chat with other gamers. VR is not ready yet to render this type of experience. Or better yet, the technology is ready, but it still has to become cheaper, more adopted and widespread. Gamers play for hours and they require the highest level of quality in content, which VR cannot give yet. Moreover, the size of a VR game in terms of megabytes can be 10 times the size of current games. Developing VR content, especially for gaming, is complex, time consuming and expensive. On the other hand, VR enterprise applications have different characteristics and have to fulfill different needs. Typically, the VR enterprise market is represented by training applications. These are typically short, focused experiences aimed at simulating something or training someone on something, The graphics are not as important, but the process is. It is what you learn to do or what data you visualize that is important. The development times and costs of such an experience are acceptable and company can have a clear benefit from using this technology to train employees more effectively and efficiently.

More on VR hardware

It has to be said, VR is not only head mounted displays and content. A myriad of other devices are populating the VR world. They all aim at essentially one thing: make the experience more immersive, more realistic, more engaging. Fundamentally mirroring our body into our Avatar. From hand controllers to hands-free haptics, gloves and other experiments, the VR hardware market is highly scattered and formed by essentially a few very large players and many small producers of innovative devices, that are generally better than the large brands ones, but with more “fragile” technologies and support. The VR hardware market will offer many innovations and new devices in the near future and will follow the pattern of evolution of other digital technologies.

One interesting aspect and actually real market evolution in the computing world, is the shifting of processing of data from the CPU to the GPU. The GPU is the graphical processor in a computer, mounted on what is called the video card. As graphics and content is getting more complex, video processors and graphics cards evolved into more powerful units, even more powerful than the central CPU of a computer itself. Companies like Nvidia are at the forefront of this and have been driving this change in the recent years. Essentially - besides mobile VR headsets - a VR ready computer has a graphics card that costs typically as much as the rest of the computer parts all together. Part of the data processing has been entirely moved to the GPU.

Haptics and eye tracking

While there are various attempts to achieve good haptics already available on the market, what is most interesting is the development of eye tracking functionalities into various HMDs. Tracking the movement of the eyes is actually an enormous source of data and can also give us useful information on how to make VR experiences better and less tiring for long use. The first immediate application of eye tracking is off-course knowing what people are looking at, for how long and possibly even with what degree of attention. This is especially useful in retail environments in 3D, where users can move around and look around freely in a 3D digital space. With eye tracking we will able to see what products users are mostly looking at, what features, what interests them or not. In the training use case, eye tracking will give us information on what people are paying attention to, what they focus on and what they have difficulty with. A complete tracking of eye movement will lead to understanding user emotions during their virtual experiences. I believe these are the main factors driving the adoption of eye tracking on many devices in the near future. Marketing folks and data analysts will love this feature!

VR/AR/XR/MR

A hell of a lot of Rs for essentially the same technology.

It’s funny how each time there is a new tech on the edge of mass adoption, the carnival of names pop-up. It’s like a new baby we need to give a name to. Technology likes funky names and often tech startups invented new streams of names, just like the name fashion for babies. The difference in this case, is the we are making things hard for people to understand what the hell we are talking about. Let’s get it right, VR and AR are technologies coming from the same basket. Then they can pair up, for example, VR and AI combined can generate an “Immersive Smart Environment”. Without AI it would be only a VR environment. But the term mixed reality is really coming from mixing the virtual/digital and the real.

There are two main streams of this technology: VR, the immersive technologies stream, where you use HMD headsets and are entirely immersed in the virtual environment; and AR, Which belongs to ambient technologies, or techs that integrate with the environment around us.

These techs come from the same origin, the same development processes and they just differ by the devices used to visualize content and the way data is assembled. But there is only one main element of differentiation. Augmented Reality to work needs sensors. Because AR is integrated with the physical reality, it needs to sense, measure and give constant feedback on our surrounding environment. In this aspect the IoT come in as the porting infrastructure needed to make AR tech work. The IoT is in other words, the backbone for augmented and mixed reality experiences. This happens today with IoT sensors and the new advanced sensors introduced by new high end smartphones, which are able to literally scan in 3D the environment around us, being able to position themselves as an object in a cloud of physical objects. The ability to scan reality around us and position digital elements integrated with it, is what makes AR really possible. 

AR Glasses

As virtual reality need HMDs to be experienced, AR today is mainly experience through the use of a mobile device, such as high end smartphones and tablets. But we can already see on the market the first AR glasses. Since a few years brand like Microsoft, Google, Magic Leap and others have produced their own version of AR glasses. They are all still early stage models, with limited capabilities and bulky dimensions, let alone the super-high costs. Like often, the most interesting products are coming out from the innovation market of small startups. Companies like DreamWorldVision for example are producing more interesting alternative to large brands, lightweight devices, more usable, mobile, and with 2k 90 degrees field of view. This last detail may be difficult to interpret by most not technical… but it essentially mean that the field of view of the device offers the experience having a full rectangular screen overlayed onto reality. In other models, like the Hololens from Microsoft, the visual area of the glasses is not the whole lens surface, but only a small squared portion. Which is not good enough to truly experience AR.

AR needs glasses to really kick-off and become mainstream. We need our hands free and light devices to wear on our heads for AR to really work. AR on mobile phone is good, but you can’t expect people to look at the world through their smartphone camera at all times.

If we compare AR glasses to mobile phones, we are now in the era of the bulky large mobile handsets of the first days, with a battery bag to carry around. But faster than what happened with mobile phones, AR glasses will become as lightweight and usable as an iPhone in not more than 3 years from now. 

The Internet’s Third Dimension

The real big disruption that is behind ambient and immersive technologies is really that we are adding a third dimension to the internet. We are doing this at two levels, a virtual level with VR and AR and the physical one with the IoT. These technologies, apparently very different from each other, really carry a similar structure and design, as they are thought to be deeply integrated. The IoT gives infact the backbone with ubiquitous computing, for spatial, ambient and immersive experiences to take place. With ubiquitous computing and the evolution of the IoT into a network of hyper-powerful smart objects, we are able to call upon processing power on demand, any amount of processing power. Precisely what is needed to shape an immersive, real time, pervasive, 3d Internet. 

The Applications

So yes, finally here we are, the part that I guess interests most. What the hell do we do with it? So this is exactly what people like me have been working on for the past 15 years. Trying to test all possible applications and the ones that best fit corporate needs. VR and AR in the various declinations can be applied to a variety of sectors and in many ways. As this technology progresses, people accept it more and companies invest in real long term projects, VR and AR are being applied to more and more use cases. In the past 15 years I have worked on cases for industrial training, defense training and simulation, risk training, innovative digital retail experiences and more. I can safely say the following are the applications of VR and AR that are definitely market-ready and for which the technology has proven to be effective not only in the wow factor, but also in terms of data gathered in various use cases of successful applications.

These are:

1. Training and learning, especially in industrial cases, difficult situations to replicate, dangerous hazards.
2. Immersive Collaboration, creating collaborative interactive real time online environments in 3D, where people can work together, but also engage in learning sessions. 
3. Data Visualization, enabling immersive 3D visualizations, also in collaborative mode and displaying real-time data and information contextualized also geographically.
4. Retail, with new concepts of immersive digital shopping, 3D online stores, AR shopping experiences integrated with the real environment. 
5. Gaming, in all forms and ways.

Platforms

As much as devices, the backbone of this technology is really what happens at software level. Distributed computing and faster processing power and bandwidth now allow us create platforms to sustain a 3D web environment needed to experience a shared virtual reality experience. Probably not as many as players producing devices, the ones working on platforms are a few, well experienced and funded… These companies range from Google and other large players, to small inventors and leading evangelists and technologists who took the lead in this technology. 

Besides the software engines, we can today use to create a VR/AR project, from Unity to Unreal to Havock and others, there are platforms born to allow people to socialize in VR and create their own spaces and virtual simulations. Highfidelity, for example, founded by Philip Rosedale and Ryan Karpf, is a platform that integrates powerful distributed computing paradigms, VR and the blockchain, to allow the creation of independent and interconnected virtual worlds than can scale up to thousands of simultaneous users. 

Other platforms are populating the universe of what is called today Social VR or VR online environments where people can socialize and interact in 3D. The true precursor of social VR was actually Second Life and the many other virtual world platforms that were born (and died) between 1999 and 2010. Social VR is nothing new, but today we have the possibility to experience it with a better level of interaction and realism; while new devices will allow a more immersive and engaging experience.

The Future of VR?

We all know the imaginary future of VR by now, widely shown in movies like The Matrix or more recently with Ready Player One. Essentially endless parallel dimensions in which to live our future existence, separated from our original body, possibly preserved or maybe even not. If you are interested in this type of hyper-futuristic visions, you may find something you like in my other articles here on LinkedIn and on Qbit's VR Blog.

Going back with the feet on the ground and sticking to our current dimension of reality, one of the companies that is at the forefront in terms of vision and roadmap in developing environments and tools that leverage the new era of the 3D Web and immersive and ambient computing, is Dassault Systeme. I have to mention this company, because they were the first to conceive and apply the vision of a virtualized version of a whole organization or company, with all its departments, facilities and processes, virtualized in 3D. The entire value chain is mirrored in a 3D virtual environment, starting from the 3D model itself of any product or even service, visualized with new 3D capabilities. Dassault integrates emerging immersive technologies with enterprise resource planning, evolving the concept of ERP into the new paradigms of Software Defined Organizations. Through virtualizing every process of a company, we can work with an immersive real time model, in which we can work, learn, monitor, evolve and simulate changes in our systems, products, services.

Imagine everything you do today on the Internet and more, all in 3D. An environment where for the first time you actually see the people (Avatars) behind the screens that separate us today from each other.

Thank you for reading and for your comments and contributions. For more articles on VR and immersive and ambient technologies, please read our VR blog.