Simulations & Virtual Reality, Digital Twins and Augmented Reality Supporting Automation and Production Solutions (3/ 3)

AAE (Grauel) pushes technical boundaries. We provide high tech printing & assembly solutions. We support smart printing and manufacturing equipment with Industry 4.0 technologies and solutions. This series of articles provide background information on how these developments are supported.

Introduction

In the previous?articles?we?looked at?simulations and digital twins?and how they support industry 4.0. Another interesting technique that is employed supporting industry 4.0 is augmented reality?(AR).??

Using?augmented?reality,?the?communication in product?and machine?design as well as?production development?improves.?It?helps to identify?and avoid design errors in early stages of the development process.??

Looking at augmented reality more generally, it can be used to?support?increasing?production?by?enhancing the user?experience (during the?development,?manufacturing, maintenance,?etc.).??

In this article we?will look at the history and definition?of?augmented reality (AR)?in the first paragraph. After?that?we take a “deeper look” at the technical aspects of Augmented Reality.?The last two paragraphs deal with the benefits for both the automation supplier and the manufacturing company?when using Augmented Reality.??

Augmented Reality (AR), history and definition

Augmented Reality?has a new feel to it,?but it actually?dates back to?1957;?depending a bit on the definition that is used.??

At that time?Augmented?Reality was first achieved,?by a cinematographer called Morton Heilig. He?developed?the?Sensorama?which?included visuals,?vibrations?sounds?and smell to the viewer.?This did not include the use of?a?computer?(or AR glasses)?but it was the?first?example?and?attempt at adding additional?(enhanced) information?to an experience.??

Then a?more modern approach appeared in?1968?where?Ivan Sutherland?(American computer scientist and early Internet influence)?invented the head-mounted display. The display offered?a sort of (viewing)?window into a virtual world.??

But it is a little hard to?provide a clear date?for the birth of Augmented Reality?as?back in those days?there was?not yet?a clear?discrimination?between?“virtual reality”?and?“augmented reality”.??

This?discrimination,?between the two (2) phrases,?appeared around 1990 where Virtual?Reality was?invented?by Jaron Lainer of VPL Research in 1989 and?by?Thomas P.?Caudell of Boeing in 1990.?Boeing is still a leading company in the use of Augmented Reality.??

An overview of the history of Augmented Reality is shown below?where?Augmented?Reality is assumed to have started in 1968.??

This brings us to the definition of?Augmented?Reality where?Virtual Reality and Augmented Reality?have the same basis (two sides of a coin). Simply put, Augmented Reality?is Virtual Reality but then with one foot?in the real world.??

Augmented Reality?places?artificial objects?(or data)?in?a?real?environment,?where Virtual Reality creates?an?artificial environment.?

We can formulate the definition of Augmented Reality?as:?

A technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.

A system can be projected, using a tablet in this case, to present an FMS?(Flexible Manufacturing System)?onto an?existing?environment.?The image we presented earlier is shown below ?*1?where a system is shown on a production floor (see?the?paragraph?“Model Presentation, first steps” below?for the actual app).??

• *1: development for a Microsoft?Hololens?is similar?

Augmented Reality, a deeper look

There are various software packages available to support the creation of augmented reality.?Schematically?the “components” needed?can be presented as follows:

The environment is scanned using a device (phone in this case) and the environment (trigger) is recognized.?Based on the recognition, information is pulled from a database and projected on top of the?scanned image?as captured by the camera.??

The first we need is something that triggers the information. This is?often called a?“trigger point”?or?“marker.”?.

Trigger Point

A trigger point will execute the superimposed information and show it on the?screen. Once?the trigger is?recognized the model and / or information is retrieved?from the database?and shown.?The basic trigger point includes:??

• Image recognition?(information appears when an?(flat)?image is recognized)?
• Model recognition?(information appears when a model is recognized)
• Positioning recognition?(information appears when a certain location is reached, ‘Pokémon Go’ is a well-known example).??

Augmented reality that always superimposes the information?on the screen is?also?present;?these are also referred to as?“markerless” trigger points. This is typically used when you want to “try” furniture at a specific location in?your?house. By pressing a button, the image is superimposed on the screen (and thus projected?onto the world,?e.g.,?showing a new tv on the wall).??

?Model Presentation, the first steps

Once we have a trigger point, we want to show?useful?information.?In our case we want to?show a model of?a?FMS (with robot?and handling units).?

Unfortunately, a CAD model cannot be used directly and needs to be modified (through?tessellation, see also previous article).?Fortunately, developments in this field are progressing fast, more software becomes available allowing for the transition?of CAD models?to AR models.??

Note:?implementation of this digital transition requires a new approach?for?engineering departments. But once implemented,?the communication with customers and other?departments becomes?much more effective.?

An interesting article on this?engineering?topic?is?“Augmented Reality system for aiding Engineering Design Process of Machinery Systems”?by Marcin Januszka and Wojciech Moczulski?for those interested in this subject.?

We receive the models from the engineering department?which?are?then tessellated *2?(optimized for use in an AR app). When completed, the models can be imported and used for the development of the Augmented Reality?*3?(we use a?“markerless” trigger point in this case).?

*2:?tessellation?becomes more complex when?e.g.,?animations are to be included with accurate?movements.

*3 the example below shows the model on an Android tablet; the app can be generated for multiple devices.

The model that is shown in the image above is pulled from a database. This can be stored locally or in the cloud. Cloud storage is preferred also looking at revision control and display options for multiple users.?

Adding More Functionality

With the model?included in the app,?we get a good understanding of?how the model fits?in the environment.?Additional information can be added?for?an even?more immersive experience. This?may include movements,?technical data?or even a complete software menu.?For additional technical support?or presentations, parts of the machine can be removed or made transparent to see?and understand?the operation.??

In the?Augmented?Reality app,?we can zoom in?on?the handling units (by clicking on them), an additional view?is generated which?shows more details.??

When the Augmented Reality app is used after the machine is installed, spare parts and technical drawings can be retrieved to present?them?on top of the actual machine. Also,?the assembly of dedicated parts can be shown.?A commercial approach is shown below where images are pulled from the Internet?(cloud)?and shown when the?robot is selected.

Note:?implementation of the augmented reality strategy requires planning and good communication between the different departments. This is a new approach for many companies but results in great benefits for all parties involved.?

Augmented Reality (AR) Supporting Automation Decision

We looked at an example in the previous?paragraph,?but?AR offers more benefits for?automation?suppliers.?We will take a look at the?benefits in general.??

AR?supports?communication?during the?machine designing process.?The design process can be?streamlined by the?collaboration between?the?parties involved.?

Developing a product or machine (from concept?through?development)?is a long and resource-intensive process. It requires frequent?communication between?different?parties and involves numerous revisions of the initial concept. This all takes place before?the product or machine?reaches?production and manufacturing.??

By using augmented reality all parties involved?can see the product?or machine?in development in real-time?(supported by actual machine operation?when required).??

Once developed, AR supports pre-installation requirements where AR can?be used to?project the machine onto the work floor allowing to check the installation requirements.??

In case of?operation and?maintenance, the experience and knowledge of the machine builder may be required. This?normally involves travelling to the site where the machine is?operation.?With various machines installed, this involves?a lot of travelling.?Using?AR, the expert(s) can look through the eyes of the technician?who is?physically?doing the maintenance (a HoloLens is a typical example to support this).??

The expert can leave notes or annotate on the field of view of the technician, so they can point out particular features of interest in what the technician is seeing.?

In addition to maintenance?support, the?spare?parts to be installed or replaced can be projected onto the machine?to ensure?correct installation?or?to provide?assembly?instructions.

Augmented Reality (AR) Supporting Manufacturing Decision

Also, for manufacturing companies, AR offers great advantages.?Modern manufacturing involves complex automation solutions. A wide range of components are put together using a wide range of processes.???

AR is used to?display relevant process information, technical drawings?and even videos onto the AR?app.?This can be used for?a wide range of purposes including?training, quality?or even?validation.?

An example is included below showing a product with the proposed?areas of interest (e.g.,?bar code printing?location?and colors?and pre-treatment area).?The product and print?are?projected onto the real world?(this?example is created using the?Hololens).?

When actual production data is "overlayed" onto the AR?app, operators can verify the current operation with the?defined?parameters.?For?example,?looking at a robot, operators can see information such as temperature, current load profile, speed of the various axis at a glance.??

An example is included below where data from the robots (and FMS) is projected on top of the real world?(this example projects the information on a booth at a trade fair).

For operators using?an?AR?headset,?this?provides an additional benefit as they can keep their hands free while performing operational or maintenance tasks. Using AR this way?it enables?manufacturers streamline predictive maintenance programs. It?helps workers to?see the equipment’s potential failure points, letting them quickly see if there is a problem and, crucially, identify which parts are at fault. Replacement parts can then be ordered from a supplier without the need for costly unplanned downtime.?

For quality inspectors, AR?can be used?to provide a?virtual overlay of an assembled product. This overlay can be used to?verify if?ha products has been assembled correctly and meets quality demands. Especially for complex products this provides a useful tool.??

For more information please also check Grauel, a brand of AAE, on LinkedIn?for weekly updates and extra content or check aaebv.com.

Literature

De Pace, F.,?Manuri, F., Sanna, A. (2018). Augmented Reality in Industry 4.0, Available at:?https://www.researchgate.net/publication/324565731_Augmented_Reality_in_Industry_40?[Accessed October 15, 2020].?

Liu, X., Sohn, YH., Park, DW (2018),?Application Development with Augmented Reality Technique using Unity 3D and Vuforia, Available at:?https://www.ripublication.com/ijaer18/ijaerv13n21_33.pdf?[Accessed October 10, 2020].?

?S??ski, J., Salonen, T.,?Liinasuo, M., Pakkanen., J., Vanhatalo, M.,?Riitahuhta, A. Augmented reality efficiency manufacturing industry: A case study.?NordDesign?2008, pp 1 –11.?

Januszka, M., Moczulski, W., (2011). Augmented reality system for aiding engineering design process of machinery systems, Available at:?https://www.researchgate.net/publication/225901566_Augmented_reality_system_for_aiding_engineering_design_process_of_machinery_systems?[Acessed?October 10, 2020].?

Mourtzis, D., Doukas, M.,?Riitahuhta,?Bernidaki, D (2014),?Simulation in Manufacturing: Review and Challenges, Available at?https://www.sciencedirect.com/science/article/pii/S2212827114010634?[Accessed October 10, 2020].?

https://prespective-software.com/white-paper/?(2020),?Digital Twin Tech and your?mechatronical?system?[online]. Available at?https://prespective-software.com/white-paper/?[ Accessed October 12, 2020].?

Eyre, J., Prof. Dodd, T., Freeman,?C.,Lanyon-Hogg, R., Dr. Lockwood, A., Prof. Scott, R. Demonstration of an Industrial Framework for an Implementation of a Process Digital Twin. ASME 2018 International Mechanical Engineering Congress and Exposition, 2018. Page 1 – 8.?

The future Factory. Industry 4.0 and Digital Twins: Key lessons from NASA [Online]. Available at:?https://www.thefuturefactory.com/blog/24?[Accessed September 20, 2020[.?

Exorint. What is the Difference Between a Simulation and a Digital Twin? [Online]. Available at:?https://www.exorint.com/en/blog/what-is-the-difference-between-a-simulation-and-a-digital-twin?[Accessed September 25, 2020].?

Simul8. What is Industry 4.0 and could simulation help unlock its potential? [Online]. Available at:?https://blog.simul8.com/what-is-industry-4-0-and-could-simulation-help-unlock-its-potential/?[Accessed September 21, 2020].?

Behrtech. Digital Twins for Industry 4.0: Applications, Benefits, and Considerations [Online]. Available at:?https://behrtech.com/blog/digital-twins-for-industry-4-0/?[Accessed September 21, 2020].?

Onirix. Augmented Reality for Industry 4.0 [Online]. Available at:?https://www.onirix.com/learn-about-ar/augmented-reality-for-industry-4-0/?[Accessed October 15, 2020].?

Advantech. Enabling Industry 4.0 with Integrated Automation & Cloud Innovations [Online]. Available at:?https://www.advantech.com/industrial-automation/industry4.0/emo#my_cen. [Accessed October 05, 2020].?