Digital Twins – Present & Future

#digitaltwin #causality #inversedigitaltwin #graphcausalmodel #algorithm #kalmanfilter

Digital Twin is a framework by which humans can interact with IoT Systems in a natural way. The best way to think about Digital Twins is as the seat of Machine Learning and Artificial Intelligence in IoT systems.

By now in 2020, there is general acceptance that IoT is a “must have” technology for industry, enterprise and home. What is less clear to buyers and sellers of IoT is how to get business value out of IoT deployments!

Let us agree that the way businesses get value from the current AI and Machine Learning boom is through “PRESCRIPTIVE Analytics” – we will use it as a “suitcase” term meaning software that will tell business leaders what to do and when (and what NOT to do also) . . .

Step back and think about a 1970’s and 80’s scenario . . . Your old car is running funny and you take it to a mechanic. After the junior technicians have poked around, the master mechanic steps up; he puts a stethoscope to the running engine, listens carefully for a minute or so and proclaims, “increase the timing belt tension by 2 clicks”. Once done, the engine purrs . . .

IoT is the stethoscope, (ear is the Edge compute), master mechanic forms a mental model (or mental “twin”!) in his head, does his what-if analysis in his mind and then states his Prescriptive Analytics (timing belt adjustment).

Back in the 1990s, NASA kicked off Numerical Propulsion System Simulation or “NPSS”. Objective was stated as flows: “The analysis is currently focused on large-scale modeling of complete aircraft engines. This will provide the product developer with a "'virtual wind tunnel" that will reduce the number of hardware builds and tests required during the development of advanced aerospace propulsion systems.” In subsequent years, every jet engine manufacturer (such as GE and Rolls Royce) and many other independent parties (research institutes and universities) became part of this open source effort. Now NPSS (“gas path“) model of a jet engine is used by manufacturers for ML-based fault detection by proprietary methods which has extended NPSS to suit their “gas path” combined with deep-learning. NPSS is one type of digital Twin.

In simple terms, Digital Twin is a software counterpart of a Physical Twin. Here are some examples.

On the left-hand side, we see a motor, a CNC machine and a flexible manufacturing system (FMS). Physical Twin examples can be as varied as a widget or a patient or a whole city. The right-hand side indicates the forms that Digital Twins can take; it could be a simple dash-board that displays real-time and historical data from the Physical Twin or a NPSS-like simulation of a jet engine color-coded for heat buildup or a smart city simulation.

Digital Twin finds multiple uses in technology. Considering two types of Physical Twins, Assets and Systems, application can be for the creation of the “product” itself (a motor CAD/CAM example) or the focus may be on the use of these products for producing other widgets (use of a motor to turn a CNC machine to turn a metal rod) or both. The use of Digital Twins for the design of machine is very advanced and prolific. However, our focus is on *operational* aspects that produce widgets for sale and revenue generation.

As you can see there are different varieties of Digital Twins (DTs) and their proliferation has created a lot of confusion among practitioners! We will present a clear taxonomy as follows.

At the outset, we should note that each of the Digital Twins in the chart are not distinct entities but each are built upon the previous ones!

Display DT is a software image that is populated by data that are generated from its Physical Twin (PT). Just this simple level of DT can be very useful in an IoT solution! Human eye is very good at detecting patterns and a Display DT can quickly tell us if its PT is operating properly.

Forward DT is built on a software simulation of PT. Consider a physical printed circuit board (PCB) of an electronic device; thermal distribution across the PCB is usually of great interest (do you need a fan in the enclosure to cool the PCB for proper operation?). Heat transfer equations are well-developed and given the geometry of the PCB, pretty accurate heat distribution map of the PCB can be calculated on the computer. This heat map can be displayed on a screen to set the baseline; when the electronic device is in operation, actual measured temperature(s) can be overlaid on the calculated heat map; any difference in patterns and colors indicate a malfunctioning PCB (or wrong simulation calculations!).

Inverse DT is a new and a necessary form of digital twins. We will discuss this topic in greater detail below. In addition to the information that Display and Forward DTs provide, we want to know the *causes* of what we see on the DT in real-time; in that sense, Inverse DT can also be called “CAUSAL” digital twin. The word “inverse” arises because of the desire that from the sensor and other real-time measurements we make, what we really want to know is what is happening “inside” the system, be it inside a machine or a retail outlet or a city! Clearly, this is a very difficult task. Fortunately, ML and AI community has started to focus on Causality and methods are being developed. We have developed our own algorithm with the specific objective of capturing the *dynamics* of the underlying system that generated the sensor and other real-time “surface” data.

Current Digital Twins:

Digital twins are taking center stage in IoT, especially in Industrial IoT. And for good reason! We can “see” the motor or pump on your PC and manipulate its “avatar”. This opens up many possibilities . . .

?      See the current condition of the equipment

?      Compare to stored past

?      Compare to similar

?      Understand failures & improvement possibilities

?      Try out changes in software – effects of interventions

?      Assess gains & risks

?      Aggregate them together to assess system-wide performance

?      . . .

There is a progression of Digital Twin versions as we saw earlier that deliver more and more value (do more and more of the list above).

Let us step through the first two Digital Twin versions.

1.   Digital Twins that display data:

Seeing field measurements on your PC displayed on a rendering of the machine is revelatory – after all, vision is our dominant sense for absorbing information. You can eyeball the data overlay on a picture of the machine and make visual or ML-assisted fault detection, for example. A very useful start!

2.   Role of Software Simulation:

Once you have a “companion” software simulation of the machine, we can compare the field measurements with the simulated “measurements”. Any deviation is an indication of an issue – either that the simulation is inaccurate (and needs update) or more importantly, something is wrong with the machine.

Since you have a complete simulation in hand, you can investigate further. What changes to the simulation will match the field measurements? Are these changes within simulation “limits”? If not, you have a problem in the field which requires a “truck-roll” and root-cause analysis in the field. Such a digital twin saves money and time from disrupted operations and the deployment of expert troubleshooters who are in short supply.

Measurements by themselves are NOT important; there is nothing controversial in this statement – when we measure vibration signals from a bearing, it is NOT the vibration itself that is of interest; we want to know if the inner or outer race or balls have problems!

The real purpose of developing and deploying a Digital Twin is to understand what the “parameters of the underlying system” are so that we can develop a CAUSAL understanding of what the measurements are telling us.

Causality:

All of today’s ML is correlation-based. And we know that “correlation is NOT causation”! A recent article, “How Causal Inference can lead to Real Intelligence in Machines”, does a good job of explaining this predicament leading one of the “AI Three-Amigos” to make serious efforts to incorporate Causality into AI . . . lest another “AI Winter” emerges! Most of the credit for “causality crusade” goes to Judea Pearl (Turing Prize winner) who started his drive into Causality Calculus from early 2000.

There is a new awakening in AI & ML that incorporating Causality is essential to move the field forward. This is because while association or correlation-based ML has brought us many low-hanging fruits, it is not sufficient to get us the *valuable* fruits that lead to prescribing specific action to achieve specific business results!

When you take a medicine, randomized controlled trials have been conducted to prove the cause-effects of that medicine – this is the gold standard method to prove *causality*. A physician will not *prescribe* it if it was only correlated with your illness! Taking a small leap, *Causality* is required for *Prescriptive* analytics.

You cannot do randomized controlled trials for industrial machinery! And Causality is hard . . . when you encounter an “immovable object” such as Causality, you have to employ multiple strategies to get through. We are at that juncture now with IoT and Digital Twins.

Since Digital Twin is simply a code word that encompasses AI, Machine Learning or Data Science when the input is from IoT, Digital Twin is where cause-effect determination has to happen. This is the place of INVERSE Digital Twin (IDT).

Inverse Digital Twin:

What Inverse DT does is to systematically incorporate software simulation and add Machine Learning and System Identification (for estimating the underlying parameters) on top. This is the power of Inverse DT – the ability to incorporate independent sources of information generated by the simulation into traditional estimation methods.

The concept of Inverse (Causal) DT twin can be best explained by a medical scenario. Say, you go to a Neurologist since you are feeling unwell. She will ask you to get an EEG (electroencephalography) study where the technician will place multiple electrodes on your head and measure and display the electrical activity (a Display DT of your brain!). The specialist can make certain conclusions based on past experience such as “spindles” will appear in the measured data when sleepy which is a use of Correlation studies done in the past. But what the Neurologist may want to know is the origin inside the brain of the epilepsy signature that had appeared in the measured EEG.

So, from measured data on the surface, the challenge is to deduce the CAUSE of it. This is INVERSE modeling. Friston and colleagues have worked on this problem of “Dynamic Causal Modeling” of the brain; here is a figure from one of their publications modified for our purposes.

You see the EEG data measured on the surface of the brain on the right; but the specialist wants to know the epileptic focuses INSIDE the brain which in a live human being is not easy to do! This figure also clarifies a distinction that has bedeviled the Digital Twin community – what is the REAL difference between “simulation” DT and “causal” DT. To avoid this confusion, we use the terms FORWARD and INVERSE digital twin, respectively.

In the IoT digital twin case, the Forward DT is a software simulation such as NPSS or PCB heatmap. Forward DT has a strong history and is pursued vigorously by various companies today. However, the Inverse DT has not been developed by anyone till now (in 2020), as far as I know. That is because the Inverse problem is an “ill-posed” problem in Mathematics and considered impossible to solve generally other than by using multiple constraints!

Finding the CAUSES given the EFFECTS (surface measurements) is hard – but the pay off is great! We will know the real internal causes of issues in an operating machine and we can prescribe the right actions to solve it at the root level.

In Mathematics, the constraint that is usually used to solve ill-posed problem is called “Regularization” (of Tikhanov). Without going into the details, the simple idea is to choose the Solution Vector with smallest norm (in L2 regularization). However, such a “solution” may have no bearing on reality while being Mathematically satisfying . . . But we want solutions that will lead directly to identifying the location of an epileptic focus or a bent shaft inside a motor!

INVERSE Model framework:

Application-domain constraints make the solution to the ill-posed problems meaningful. Consider the IoT Industrial Machinery applications. Let us start with a general framework.

The meter indicates a surface measurement device (like the EEG machine). Below the surface (shown in brown), let us imagine a “generator” that is a collection of 3 signal sources of varying strength and location. Based on the radiation of signals from these 3 sources, the meter will measure the additive combination of these 3 sources in the simplest case.

Now assume that these 3 sources are also “coupled” (via wired connections perhaps). As the coupling strength changes, so does the DYNAMICS of the system and the signal measured by the meter. Coupling changes can give rise to very complex measured surface data from which deducing the number, location and coupling dynamics of the sources can seem virtually impossible! This is why we call this an “ill-posed” problem.

In the rest of this discussion, we will assume that our main interest is in the DYNAMICS of the system. In the specific case of “machine design”, we are interested in the kinetics and kinematics of the machine and NOT the Statics or Structural aspects. (By the way, as mentioned in the introduction, there are DT applications where Static and Structural aspects are important – one case is the design of a product where using CAD/CAM techniques where we can visualize the stress and thermal distribution and improve the design.)

Consider a real-life example of a motor to be monitored. You see a cut-away view of the motor on the left-hand side with a vibration sensor attached to the body (on top) of the motor. The surface measurement of the vibration is indicated on the right-hand side (top). Considering the dynamics of the motor that we want to track, the 2 bearings and the shaft are the key sources of vibration when the motor is operational. Much like the previous made-up “meter” example, vibration will travel to the sensor and combine to provide surface measurements.

When the inner or outer race of the left or right bearing is faulty, vibrations will increase. But these changes will affect the shaft. If the shaft is misaligned or slightly bent, both bearing will be affected and the vibration from all 3 components may increase. This is captured in the “couplings”, K1, K2 and K3. As these couplings change dynamically over the life of the motor, the vibration patterns will also change. Note that couplings, K1, K2 and K3, are sometimes called “Virtual Sensors”.

Graph Causal Model:

The next step in the development of Inverse DT is the creation of Graph Causal Model. This is where domain experts come in who can tell us how best to form the “forward” model. For example, in the motor case, a mechanic who has spent a lifetime repairing and rebuilding motors can tell you the key sources of vibration and how the sources are causally related!

The resulting overall model of the motor for our *dynamics* analysis purpose may be as shown. The load input, “c”, can be measured as the current draw or by a torque meter. K1, K2 and K3 are the “couplings” of interest. p1, p2 and p3 are the less important (for our study of motor dynamics) transmission strengths of vibrations to the sensor on top of the motor. This Graph Causal Model is the Forward simulation model for our Inverse DT solution and involves the interplay between the Forward and Inverse models within our Inverse DT algorithm.

Inverse Digital Twin Algorithm:

To get familiar with the Inverse Digital Twin (IDT) architecture, consider a Plant whose digital twin is of interest to your business. Plant is a KNOWN entity because either we made it (say, a motor) or we have a model with known parameters, {p}, and “field measurements” (say, soil moisture and nitrogen from a wheat field, taking an AgriTech example). Our purpose with IDT is to TRACK the variations in {p} in real-time from field measurements. We can also consider a special case when a field measurement is NOT available – such as when a sensor failed or that measurement is practically impossible to make (perhaps due to cost).

The traditional role of the Kalman Filter is to act as the Observer which reveals the Plant parameters. With just the input and output of the Plant available to a Kalman Filter, the estimated States are not, in general, estimates of {p} when certain System matrices are unknown. The added blocks of Machine Learning and software simulation of the Plant are the key to recovering the Plant parameters. After sufficient learning, {q} approx = {p}.

The challenging part of IDT algorithm is the development of the COMBINATION of a neural network and the software simulation,‘psi’, of the Graph Causal Model.

A MIMO neural network as shown below with 1 Hidden Layer was implemented using a matrix version of backpropagation algorithm. The ‘M’ inputs are the ‘M’ States from the Kalman Filter. The {x2}’s are the {q} software simulation parameters which when converged are the estimates of Plant parameters, {p}.

Assume that the Plant output ‘y’ and all the field measurements are available. The software simulation, ‘psi’, uses these inputs and the running simulation produces estimates of Plant output and the other field measurements. The differences between the actual and those produced by ‘psi’ gives us the MIMO error vector. Once you have the error vector, one has to backpropagate the error across ‘psi’ and through the neural network layers all the way to the neural network MIMO input layer. All of them are standard work except the backpropagation of the error vector across the software simulation, ‘psi’.

Propagation of the error vector back across ‘psi' and the later stages of the neural network requires a re-derivation of the algorithm (nothing more than applying the Chain Rule carefully).

Conclusions:

We have introduced a new class of Digital Twins called Inverse DT. IDTs provide the facility to play with the Digital Twin on our PC, answer “what-if” questions and even try out counterfactual scenarios so that we can provide valuable PRESCRIPTIVE Analytics to business owners.

We find that the IDT algorithms are able to track parameter variations as well as offer some work-around when some sensors fail. This is the power of IDT – being able to incorporate an independent source of information generated by the simulation into traditional estimation methods. We are NOT “manufacturing” information from nothing – we are exploiting the knowledge (and the information) contained in the simulation structure and some guesses of governing parameters to enhance “on the fly” real-time estimation for TRACKING the Plant (machine, crop field, etc.).

The truly exciting technology aspect of Inverse Digital Twin for Data Scientists is the interplay between the software simulation and parameter estimation methods. There are many ways to construct the simulation, ‘psi’, which will lead to clever ways to infuse valuable a-priori information into the estimation step! Data Scientists can use their ingenuity and creativity to develop many combinations of simulation+estimation solutions in partnership with domain (Plant) experts . . . the possibilities are endless!

Our approach is rooted in the firm conviction that Causality is paramount in digital twins – knowing what underlying dynamics caused the surface IoT sensor measurements; this “inverse” problem of unearthing underlying causes from measurements is a hard “ill-posed” problem. We bring additional information in the form of Graph Causal Model simulation to constrain the inverse solution and make it tractable and meaningful. The Graph Causal Model is developed in collaboration with domain experts who from their vast experience can help identify a “chain” of causative factors. This information is used in a sophisticated solution involving Kalman Filter and neural network modified with the Graph Causal Model. End result is an Inverse Digital Twin that allows business owners to perform “what-if” analysis, assess counterfactual scenarios and prescribe actions that improve business performance.

PG is a technology & business native grown up within multiple careers as Startup Founder, Corporate Officer, technology leader and university professor. He is currently the Chief Acceleration Officer of Faceopen, Inc.

·      One of the first Industrial IoT applications in equipment monitoring & predictive maintenance (Rockwell Automation)

·      Wireless technologies at Microsoft (Windows Wi-Fi, BT, UWB) & Lucent (smartphone chipsets & software)

·      Recognized as a “Rock Star” data scientist by Data Science Central; Machine Learning for Engineers book – “SYSTEMS Analytics” (2016, 2021 edition)

PG sees his role as a Data Science, Machine Learning & IoT business & technology leader & guide.

www.dhirubhai.net/in/pgmad

?