Augmented Reality for Automotive Industry

In a recent survey, McKinsey highlighted a huge gap between piloting and deploying digital use in Automotive Industry. According to this survey, within Automotive Industry, 92% of OEMs think that they are ahead of or on par with their competitors.

Within this Industry 4.0, Augmented Reality (AR) is seen as a modern tool to move relevant information to the right people in real time. But where are we today? Is AR just another trendy gadget for Millennials? Some Marketing gimmicks? Or can it offer some true industrial and business advantages in development and in manufacturing within the Automotive Industry?

Augmented Reality - What is it?

Augmented Reality (AR) belongs to Internet of Things (IoT). It is defined as the expansion of physical reality by adding layers of computer-generated information to the real environment. Information in this context could be any kind of virtual object or content, including text, graphics, video, sound, haptic feedback, GPS data, and even smell.

This expansion of the real world can be displayed over various supports: smartphone, tablet, smart glasses.

Please, note that a more advanced technology is the Mixed Reality (MR). In addition of merging real and virtual worlds, MR is able to make interactions in-between both.

The objective of this article is to illustrate some on-going projects using Smart Glasses AR in the Automotive Industry.

Augmented Reality Glasses – What do they offer?

AR Glasses are a new way of using a computer. In the past, we had dedicated screens to display information (computers, laptops, tablets and smartphones). The next step is to display computer-generated information in front of your eyes through the use of “smart glasses”. These AR glasses offer several advantages:

1. Hands-free viewing of in-situ information. It is possible to receive information while carrying out simple Tasks
2. Real-time user guidance (possibility to indicate the precise part of the item to be manipulated)
3. Use of enriched content (images, 2D diagrams, 3D objects, QR-code reading, animation, …)
4. More immersive experiences
5. Possibility to memorise an operation by recording a movie or a screenshot of the Scene

Augmented Reality – More than 20 years of experience in Automotive Industry

The first attempt to develop AR in an industrial context has been carried out by Boeing in the early 90s [1]. In the Automotive Industry, a German project called ARVIKA, led by Siemens and including most of main OEMs (Daimler, BMW, Ford, VW, Audi, etc.), aimed at applying AR to Engine and Vehicle design, production and servicing, as early as in the 2000s.

BMW's proprietary AR smart glasses from 2000s - Video available here

Example of the use of Augmented Reality in Automotive Industry

Engine Maintenance

Nowadays, all cars provide system interface that allows quick diagnostics from the ECU. Nevertheless, for the operator, most of the documentation is still stored on an external computer slowing down immediate access to the relevant information. AR has the potential to bring this information in real time, enabling the diagnosis to be displayed in immediate proximity to the Engine as the operator is working. In this project, Daimler chose to track user’s position and orientation using a marker-based approach. Daimler realised 4 tasks:

1. Maintenance and repair instruction from garage book and sketch displayed
2. Pre-recorded video instruction
3. Overlaying of 3D models and animated videos
4. Audio link with an expert to remote technical assistance 

Vehicle Interior Design

AR can offer numerous benefits in product design and development by improving efficiency and cutting development time.

For example, when designing the interior of a vehicle, multiple processes need validation at each step. Use of CAD for general design (placement of the seats, appearance of accessories, ergonomic displays, etc.) is widespread but feedback from numerical analysis of, for example, air conditioning or crash test originates from different software impeding interaction. AR allows to immerse the stakeholders by superposing the different levels of virtuality, so that a better “feel” of the final product is reachable.

Development of an interior. Image extracted from Sang-Gyun An et al [2]

Collaborative design review

Designing a complex product made of numerous parts requires many iterative engineering steps to ensure a proper assembly and readiness for production and servicing. This is by nature a collaborative task involving frequent meetings, traditionally based on CAD and rapid prototypes. CAD lacks a real feel of 3D while prototypes are expensive and time-consuming to build. AR can allow to integrate 3D data into the meeting environment by the way of holograms [3,4] without the need for constructing an actual physical model.

Holographic display in a meeting. Image extracted from Regenbrecht et al [4]

Fuse placement

In truck manufacturing, no two vehicles are really identical because of the variety of possible options. The consequence is that nearly all trucks need a specific placement of fuses and relays in the cockpit. The classical way to do this is to print on paper the scheme that the operator follows to install the fuses. The AR solution is here quite convenient. The information can be overlaid directly in the operator line of sight through AR glasses.

Fuse plan (left), working place (center), augmented view (right). Image extracted from Regenbrecht et al [4]

Vehicle Service

BMW developed BMW Augmented Reality techniques using special data goggles and wireless access to support maintenance work for complex technical innovations and vehicles service [5]. A BMW 7-series engine was used to test the system. The solution for AR-based repair guidance consists of a markerless CAD-based tracking system able to deal with different illumination conditions during the tracking stage and to automatically recover from occasional tracking failures [5].

Field Testing inspection

When OEMs or petroleum companies run Field Trial all over the world, they usually require visual inspection of specific Engine parts. Sometimes these parts cannot be dismantled or replaced (it is especially the case on large Engine for industrial plants) and expensive experts are required to travel from one place of the world to another just for few hours of inspection. Thanks to AR glasses, you can get an operator on site wearing the glasses and allowing an expert somewhere else in the world to "view" and nearly touch whatever he wants. In this case, AR can save a lot of time and Money.

Training

Contrary to Virtual Reality, AR offers the possibility of on-site experiences for the trainee. You do not have some abstract environment. For example, for driving lessons, it is possible to wear AR glasses emulating the sudden appearance on the road of a child running after a ball. Both the trainer and the trainee wear AR glasses for full immersion.

Driver and trainer wearing AR glasses (left) and augmented scene (right). Image extracted from Regenbrecht et al [4]

Conclusion

IoT has definitively expanded traditional barriers of automotive industries. Augmented Reality is already offering quite a lot of applications not only in manufacturing but also in product development process. But of course, the final customer (vehicle driver) is also expecting quite a lot from this new Automobility 4.0. One of the most exciting feature is the Augmented Reality Windshield recently patented by Apple: a unique way to understand and follow your SatNav without your eyes quitting the road. Apple is said to work on a this technology with VW

Augmented Reality Windshield by Apple (source)

If you want to know more, please refer to the following references:

References

[1] Mizell (2001). Boeing’s Wire Bundle Assembly Project. In Barfield and Caudell ed. Fundamentals of Warable Computers and Augmented Reality. Lawrence Erlbaum & Associates, New Jersey, 447-467 

[2] Sang-Gyun An et al (2016), Collaborative Experience Prototyping of Automotive Interior in VR with 3D Sketching and haptic Helpers, AutomotiveUI’17, Oldenburg, Germany

[3] Regenbrecht et al (2002). Magic Meeting – a Collaborative Tangible Augmented Reality System. Virtual Reality – Systems, Development and Applications, Vol 6(3), 151-166

[4] Regenbrecht et al (2005), Augmented Reality Project in Automotive and Aerospace Industry, IIEE Computer Graphics and Applications

[5] Dini and Mura (2015), Application of Augmented Reality Techniques in Through-life Engineering Services, Procedia CIRP Vol 38, 14–23

To go further:

• Alostad and Abdul Azi (2018), Augmented Reality Application in Maintenance Process and Inspection for Automotive Industry, J Fundam Appl Sci., Vol 10(5S), 739-745    
• Fast-Berglund et al (2018), Testing and validating Extended Reality (xR) technologies in manufacturing. Procedia Manufacturing Vol 25, 31-38
• The use of Augmented Reality by SAFRAN Group
• UtilityAR: Augmented Reality for Utilities and Industry

• Anyone Can Be a Mechanic With I-Mechanic Augmented Reality App: