Generating Synthetic Data Using Graph Neural Networks (GNNs)

Elon Musk's CES Revelation: AI's Knowledge Limit Reached, But What About Digital Twins?

At CES in Las Vegas, Elon Musk made headlines with his declaration, "We've now exhausted, all of the, basically, the cumulative sum of human knowledge has been exhausted in AI training," in an interview with Mark Penn, posted on X on January 8, 2025.

My Response? Welcome to the party, pal.

For those of us in the Digital Twins field, this is hardly news. We've been wrestling with these constraints for years. Here's the stark reality: when dealing with physical devices, there's never "enough" observational data to train neural networks to the precision needed for Digital Twins. These models require a level of certainty that's simply unattainable for scenarios too expensive, dangerous, or impractical to test in real life.

But here's where it gets even more complex: we need data not just to mirror reality but to push the boundaries of Digital Twins. We must simulate conditions or scenarios that are rare, non-existent, or future possibilities - the edge cases and what-ifs.

As explored in my article, "Reverse Engineering Complex Dynamical Systems Using Artificial Intelligence and Digital Twins," the challenge is immense. Accurate modeling of complex, high-dimensional dynamical systems often borders on the impossible. This limitation not only questions the economic viability but also the practical application of Digital Twins in real-world contexts.? In our experience we never have the freedom of enough data to generate Digital Twins for Complex Dynamical Systems and having such costly data would defeat the purpose of creating Digital Twins for the purpose of Virtual Prototyping

The Bottom Line? Musk's statement might be a wake-up call for AI, but for Digital Twins, it's an ongoing battle to overcome the data desert we're navigating. We're not just at the party; we're still trying to find the map to navigate through it.

Key Takeaway:

A viable Engineering tool is available that synthesizes training data from scratch—i.e., using Generative AI in the form of Graph Neural Networks (GNNs) to perform synthetic data augmentation.? This synthetic data can be used to build and test Digital Twins for Complex Dynamical Systems. This method for defining the graph structure in GNNs involves the use of port-Hamiltonian networks, which integrate elements from graph theory, neural networks, and port-Hamiltonian systems.

Background:

Due to the lack of sufficient real-world data for training neural networks to model Complex Dynamical Systems, we must resort to creating Synthetic Data that simulates actual data. This synthetic data can be generated through three primary methods. Simulation: by using mathematical models or physics-based simulations to generate data that represents potential real-world conditions or behaviors. Algorithmic Generation: where we directly program rules or patterns to generate data based on known distributions or characteristics of the dataset. Or Generative Models where we use techniques like VAEs (Variational Autoencoders), or the subject of this article GNNs (Graph Neural Networks) to create new data points that look similar to the training data.

For Generative Models, it's possible to incorporate encryption, allowing the synthetic data to mimic the statistical properties of sensitive data without revealing the original information. This approach helps in maintaining privacy and adhering to data protection regulations.

However, the Quality of synthetic data should not be assumed; it must be rigorously evaluated to ensure it accurately reflects the complexity and unpredictability of real-world scenarios.

Bias is another critical concern. If the methods or initial datasets used for generating synthetic data are biased, these biases can not only be preserved but potentially amplified in the synthetic data.

We have established Validation methods to assess how well synthetic data mirrors real-world scenarios in Complex Dynamical Systems, which will be discussed in a separate article.

Approach:

In the process of creating Digital Twins for complex dynamical systems, we leverage Graph Neural Networks (GNNs). Our method for defining the graph structure in GNNs involves the use of port-Hamiltonian networks, which integrate elements from graph theory, neural networks, and port-Hamiltonian systems. A key property of port-Hamiltonian systems is that they remain port-Hamiltonian when interconnected with other port-Hamiltonians, making them particularly suitable for network modeling.

Methodology Overview: Our method assumes that the graph structure is composed of multiple lumped parameter port-Hamiltonian systems, each described by an explicit state-input-output form.

Understanding Port-Hamiltonian Systems:

• Port-Hamiltonian systems are a category of dynamical systems where energy transactions between system components are explicitly modeled via ports. These systems are beneficial for modeling due to their natural preservation of physical properties like energy conservation, which is vital in networks where energy dynamics are critical.
• Hamiltonian: Represents the total energy within the system.
• Port: An interface for energy exchange.

Graph Representation of Port-Hamiltonian Systems:

• Nodes: in Graph Representation terms, nodes represent specific variables as either values or vectors of part of a complex Dynamical System’s, (i.e. a sub-systems) state, it’s Hamiltonian function, its dissipation matrix, its control input matrix, its multi-dimensional control input vector, (a port), and its corresponding output vector, (also a port) as required to build a network representation of the system.? Nodes can also represent entire port- Hamiltonian subsystems where the various ports can be connected to form larger port- Hamiltonian systems.
• Edges: In port-Hamiltonian systems, Dirac structures represent the interactions or energy flows between nodes such as energy storage, dissipation, and external ports. In Graph Representation terms these are edges which model the power-conserving interconnection of these elements.

Multiple port-Hamiltonian subsystems can be interconnected to form larger systems. In Graph Representation terms Edges depict the interactions or energy flows between nodes:

• Directed Edges: Used when energy flow has a direction, which is common in physical systems.
• Weighted Edges: The weight on an edge indicates the intensity or capacity of energy transfer, analogous to coupling strength in Hamiltonian dynamics. This reflects how strongly different parts of the system interact in terms of energy.
• Intuitive Explanation: The Hamiltonian function quantifies the energy of the system based on its state. The interconnection matrix details how energy is conserved and exchanged between system components. The dissipation matrix explains how energy is lost or dissipated within the system.

This graph representation allows for a structured understanding of complex port-Hamiltonian systems through the lens of graph theory, particularly useful in the application of Graph Neural Networks for modeling and simulation.

Scope:

For each node, or perhaps for the network as a whole, we define the scope as either a local or global Hamiltonian function. This scope influences how the neural network handles data, focusing on energy conservation or dissipation.

Network Layers:

• Message Passing: ?As the GNN updates each step in the Hamiltonian dynamics the nodes exchange messages (akin to energy exchanges) based on their current states and the structure defined by the graph.
• Update Rules: These are inspired by Hamiltonian equations, where the update of node states considers energy conservation laws.

Implementation Considerations:

• Feature Representation: How do we translate the state of each node into features that a neural network can utilize? This involves encoding physical states or quantities into suitable formats for neural network inputs.
• Learning the Structure: How can we dynamically learn or adjust the graph structure during training? This includes modifying edge weights or topology based on how well the network performs in tasks mimicking the physical system.
• Stability and Conservation: Given the energy conservation aspect of port-Hamiltonian systems, how do we configure loss functions or constraints in the GNN to maintain these principles?

This approach not only leverages the mathematical rigor of port-Hamiltonian systems but also harnesses the learning capabilities of GNNs to model and simulate complex dynamical systems effectively.

Synthetic Data

The standard approach to generate Digital Twins for Complex Dynamical Systems is to train deep convolutional neural network (CNN) models using large-scale datasets which are representative of the target task.

We use Graph Neural Networks (GNN) rather than CNNs because the data inherently forms a network or graph, the data pattern is more relational than spatial, and because CNNs have a fixed architecture once defined; GNNs adapt to the structure of the graph, i.e. port-Hamiltonian) they work on.

We perform data synthesis based on realistic 3D transformations of the motion trajectories of a physics-based 3D model. These trajectories create the required volume and variability of training data necessary to achieve the desired Feature Representation, learning the Structure and check Stability and Conservation.

Our approach blends the rigorous structure of port-Hamiltonian systems with the flexibility and learning capabilities of GNNs, potentially leading to models that respect physical laws while being adaptable to data-driven insights.

Visualization

This method can be thought of as an enhanced version of an exploded view diagram in engineering. In a traditional exploded view:

• Components are shown spread out but in the positions they would occupy when assembled.
• Lines or arrows indicate the paths these components would take to reach their assembly positions.

Our method modifies this concept:

• Instead of placing components in their relative assembly positions, we start with each component at a random position and orientation (this is the Initial State).
• Using a Graph Neural Network (GNN), we simulate the movement of these components. The GNN uses rules defined by a port-Hamiltonian network to guide each component along specific trajectories towards their assembly position (the End State).
• The paths or trajectories (which are essentially vectors representing movement) that these components follow to get from their initial random positions to the final assembly are what we use as synthetic data.

By altering the starting points and orientations of the components, we can generate a vast amount of synthetic data:

• Many of these trajectories might be physically implausible or inefficient and are thus discarded.
• However, some trajectories might reveal novel or non-obvious assembly solutions that wouldn't be apparent through conventional methods.

This approach not only generates data for analysis but also potentially uncovers innovative assembly processes.

Implementation

Our approach to creates synthetic data for Digital Twins targeting Complex Dynamical Systems by blends the rigorous structure of port-Hamiltonian systems with the flexibility and learning capabilities of GNNs, potentially leading to models that respect physical laws generating large-scale datasets which are representative of the target task.

Linear interpolation assumes that trajectories are simply linear, making it challenging to be applied to the Complex Dynamical Systems that are our products.? Kinematic methods are overly idealized, overlooking the complex factors existing in real-world environments and demanding highly from the model. Probabilistic methods typically rely on assumptions about the data distribution, and if these assumptions do not align with the real data, it may lead to the decreased performance of the model. Compared to these three algorithms, our GNN deep learning methods can adapt to large-scale data and learn complex patterns.

Our algorithm comprises four main components: feature extraction, subgraph construction, spatial interaction graph, and trajectory regeneration module.

In step 1, the feature extraction stage, essential information required by the algorithm is extracted from a small amount of data and then vectorized.

In step 2, the extracted features are used to generate distinct sub-graphs for each specific port-Hamiltonian systems, ports to other port-Hamiltonian systems in the network, and energy sources & sinks.

Subsequently, all subgraphs are connected to form a spatial interaction graph, where attention mechanism operations are applied to extract spatial interaction features.

Finally, the spatial interaction features, along with known trajectories, are input into the trajectory regeneration module to generate completed trajectories by combining temporal features which is used to generate the synthetic data.

There are three immediate challenges in applying this method to Complex Dynamical Systems:

A critical and time-consuming aspect of this analysis involves establishing the initial state of the system. To address this, we employ Monte Carlo methods to generate synthetic training data for Graph Neural Networks (GNNs). These methods help us identify a variety of plausible starting points within the data space.

The next challenge is to Build an Initial Graph Representation efficiently. This is the initial graph representation of the system, which is composed of interconnected port-Hamiltonian networks.

1. We've identified that the most cost-effective approach to building these port-Hamiltonians involves: System Identification: Using techniques that convert input-output responses to identify the underlying port-Hamiltonian systems. This process utilizes frequency response data. Data Acquisition: We obtained this frequency response data by: Importing a 3D assembly into Creo MDO for dynamic motion analysis, focusing on friction and forces, to generate impulse responses for each mode and state. We then convert these impulse responses into to port-Hamiltonian systems using MATLAB.
2. Handling Non-linearities: Given the system's non-linear elements, we segmented it into multiple port-Hamiltonian subsystems: Non-linear components are isolated and treated specially. System Identification for these non-linear elements uses neural state-space models within a nonlinear state-space framework in MATLAB which is then converted into to port-Hamiltonian systems.

This approach allows us to construct an initial Graph Representation based on the interconnections of these multiple port-Hamiltonian networks, forming the backbone of our complex dynamical system analysis.

The third challenge relates to the shear complexity of the Dynamical Systems we are working with.

We developed a propriety masking algorithm and methodology which masks select portions of each trajectory, only inputting only the unmasked trajectories based on the rules defined by the port-Hamiltonian network into the model. The GNN algorithm is subsequently trained to complete the masked trajectories as accurately as possible, aiming to approximate the original trajectories. Consequently, the trained algorithm is finished to complete missing trajectories in real-world scenarios.

To maximize the utilization of known information, the feature extraction algorithm must process a substantial volume of data, filter out the relevant data required by the model, and convert it into feature vectors suitable for model input. Beyond masking and converting the trajectory data of the target node into feature vectors, the crucial aspect of the feature extraction algorithm involves filtering out other non-target node data that significantly impact the target nodes under specific conditions. These filtered data are then transformed into feature vectors, reducing the consumption of computational resources by eliminating redundant and ineffective data, thereby minimizing interference with completion performance of the model.

Censoring

After generating synthetic graphs, it is crucial to address the potential security and bias issues before using them for training Graph Neural Networks (GNNs).

Censoring Features: Once synthetic graphs are created, features such as node attributes, edge properties, or overarching graph-level metrics can be extracted. This step is vital for ensuring that the synthetic data mirrors the complexity and structure of real-world data without reproducing sensitive information.

Censoring GNNs Training: With these features, GNNs can be trained for various tasks. However, before feeding this data into models, one must consider censoring or anonymizing any identifiable patterns or features that could inadvertently disclose secure information.

Parameter Tuning: The tuning of transition probabilities and acceptance criteria not only shapes the graph characteristics but also serves as a mechanism for data sanitization. By adjusting these parameters, you can minimize the risk of overfitting to specific, potentially identifiable patterns.

Summary

This approach to synthetic data generation allows for controlled experimentation and model training for Digital Twins, especially when real data is limited or biased.

This article outlines a viable Engineering tool that synthesizes training data using Generative AI in the form of Graph Neural Networks (GNNs) to perform synthetic data augmentation.? This synthetic data can be used to build and test Digital Twins.? For Complex Dynamical Systems our method for defining the graph structure in GNNs involves the use of port-Hamiltonian networks, which integrate elements from graph theory, neural networks, and port-Hamiltonian systems.

Ensuring that synthetic data does not leak sensitive information is paramount, thereby safeguarding security while maintaining utility for machine learning tasks.