AI for Engineering: GPT-Powered Numerical Methods to Solve Engineering Problems

Motivation

The ability to solve complex problems methodically and systematically is of utmost importance in the practice of Engineering. There is a growing interest in exploring potential contributions of artificial intelligence (AI) to this process. One such AI system that has gained significant attention lately is the Generative Pre-trained Transformer (GPT).

GPT has demonstrated remarkable capabilities in natural language processing and generation, but its ability to utilize knowledge, logic, and common sense to solve intricate engineering problems remains relatively unexplored. ?This LinkedIn post aims to bridge this gap by presenting several experiments to investigate GPT's aptitude and attitude in tackling complex engineering challenges using 5 problem-solving scenarios.

• Scenario 1 – Translation of MATLAB numerical methods to Python
• Scenario 2 – Systematic solution generation to solve ordinary differential equations (ODE) with initial boundary problems (IVP)
• Scenario 3 – Modeling of numerical methods for a dynamical system
• Scenario 4 – Autonomous AI for engineering systems
• Scenario 5 – Numerical methods for partial differential equations in traffic flow modeling

The most common uses of GPT currently are customer service chatbot, question-answering based on documents, text summarization and sentence completion. Hopefully, the outcome of these experiments will shed light on the extent to which GPT goes beyond these applications by leveraging on its pre-trained knowledge, logical reasoning, and common sense to tackle much more complex problems, more impactful and far-reaching outcomes in advancing technologies in the industry, economy and humanity.

These experiments were carried out in a Python development environment prepared using Anaconda. The GPT model used in these experiments is OpenAI GPT-3.5 Turbo.

The Yardsticks

Three textbooks were used as authoritative sources of Engineering Mathematics as benchmarks against GPT-generated output:

Why Numerical Methods?

Numerical Methods are crucial in solving engineering problems as they allow for complex calculations and simulations that cannot be solved analytically by breaking down the problem into smaller, solvable steps. Additionally, they enable engineers to account for uncertainties, non-linearity, and dynamic behavior, making them indispensable in practical engineering applications. On the other hand, conventional analytical methods rely on exact mathematical equations, which may not exist or be too complex to solve.

Real-world engineering problems often involve intricate systems with numerous variables, making analytical solutions impractical or even impossible. Even the simplest engineering problem such as the dynamics of a skydiving can result in rather complex if not impossible analytical solution as demonstrated below adapted from Chapra, S (2017) Book.

Scenario 1 – GPT-generated Translation of MATLAB to Python

MATLAB

MATLAB has been trusted by Engineers for decades due to its reliability, versatility, and efficiency tackling complex engineering problems.?its extensive library of mathematical functions and algorithms made it a powerful tool for creating numerical methods to solve complex engineering problems. As a result, it is widely used in industry and engineering for data analysis, modeling, simulation in solving complex problems, design and optimize systems, and develop algorithms, signal processing, control systems, image and video processing, and machine learning.

Rationale of Translating MATLAB to Python

In recent years, Python has become the preferred programming language due to its simplicity, versatility, and extensive libraries. Its ability to handle complex calculations, data analysis, and automation tasks has made it indispensable for data scientist, and now engineers across various disciplines. Python is an open-source language, making it more accessible and cost-effective for engineers. A large community and developers provides a wider range of tools and resources for solving engineering problems. ?Moreover, majority of machine learning (ML) applications are created in Python.

The following illustration shows MATLAB code from page 40 of Ghasem (2024) Book.

In the following illustration, Python function translate_matlab_prompt takes the MATLAB code as argument to construct an instruction prompt. The prompt is used by GPT to translate the provided MATLAB to Python.

From the above illustration, Python codes provided by Ghasem Book were written using sympy package. Sympy provides symbolic mathematics capabilities, making it useful for algebraic computations and equation solving.

The GPT-generated Python code is a function with the name Cramer to conform to the MATLAB function that it was learning from. It involves calculating the determinants of matrices to find the values of the variables in the system. This method is useful when the number of equations is small, but it becomes computationally expensive for larger systems.

Latex Expressions for Mathematical Equations

Latex is a widely used typesetting system for mathematical expressions in programming and publishing. It allows users to create accurate representations of complex mathematical formulas and equations with ease, providing a high level of precision and flexibility. Overall, latex simplifies the process of displaying mathematical expressions in a professional and visually appealing manner.

The following show a list of latex scripts in Python. The way to write latex representation for a mathematical expression is by first flanking the script with ‘$’ on both sides. You can view the math expression in its natural form by using a Markdown parser.

The generated Python function was tested using A and b matrices. Let’s?investigate if GPT was able to understand the equation and in linear algebra form of Ax = b and extract matrices A and b correctly as numpy arrays.?The 2 equations were written in latex forms and stored as Python variables. They were then assembled into a single variable as system_eqn. This variable was used by translate_system_eqn_numy function to construct a prompt for GPT to carry out its task.

Thoughtfulness and Common Sense of GPT

To my pleasant surprise, not only GPT extracted A and b correctly, and in anticipation of my intention, it went above and beyond to provide a numpy linear algebra solution to solve the equation. This behavior demonstrates GPT’s ability to act thoughtfully and use common sense by relating A and b to Ax=b as well as proactively recommend a solution.

In the correctness tests, both GPT-generated numpy linear algebra and Cramer function produced in the previous scenario the same correct results of x1 = 1, x2 = 3.

Scenario 2 – GPT-generated Solutions for Ordinary Differential Equations

This experiment is similar to the previous one except that we now solve a system of ordinary differential equations (ODE) instead of linear equations with initial boundary values. Two ODEs and the initial conditions were provided as latex and expressed in Python. The matrix algebra form of dY/dt=AY was also provided to allow GPT to use as guide to formulate its solution. The equations in their natural forms are displayed in the illustration below.

A strategy was put in place to help GPT to methodically and systematically to generate solutions. Prompt was designed following recommendations in Ghasem book as follow:

• Step 1 – Extract matrix A
• Step 2 – Determine eigenvalues
• Step 3 – Determine eigenvectors
• Step 4 – Produce the general solution

Step 1 – Extract Matrix A

The generated matrix A is identical to Ghasem book.

Step 2 – Determine the Eigenvalues of Matrix A

Generated eigenvalues were identical with Ghasem book. They were both iteratively consistent.

Step 3 – Determine the Eigenvectors of Matrix A

Generated eigenvectors were consistent on both.

Step 4 – Generate the General Solution

Finally, the moment of truth, generated General Solution was consistent with Ghasem book’s.

Generate Python Codes

The prompt to solve the system of ODE was modified to generate a Python solution by replacing steps 3 and 4 as follow:

The generated Python codes were used to present the solution using a time-course graph as follow and compared with Ghasem book:

Scenario 3 – GPT-powered Numerical Modeling

Let’s revisit the design equation for skydiving. Let’s investigate GPT’s ability to design a numerical method for this equation. The generated outcome will be compared to Chapra Book’s solution.

The skydiving design equation was expressed using Latex form as diff_eqn variable in Python. By doing so, it can be reused for publishing without having to rewrite. GPT was then notified about the desirable flavor to model the numerical method which is v(ti+1) form instead of v(t+h) form. Both variables were injected into model_prompt as input to get GPT response.

The following illustration shows GPT generated numerical methods for the skydiving design equation on the right panel.?The generated output is compared with Chapra Book, the authoritative source.

GPT solutions appeared to be identical with Chapra Book.

The prompt for generated numerical method was modified by removing v(ti) style as requirement. As a result, GPT used h which is a common notation in math textbooks.

Scenario 4 – GPT-powered Autonomous AI

The 4th and final application of this post investigates the ability to perform independent analysis on data to draw insights from it autonomously. In order accomplish this, LangChain Agents were used.

LangChain Agents work according to the ReAct paradigm. This is a novel approach developed by Langchain, a language model powered by OpenAI. It aims to enhance the conversational abilities of AI systems by incorporating reactive behavior. Unlike traditional models that generate responses based solely on input, ReAct introduces a feedback loop that allows the model to adapt and improve its responses over time.

The ReAct paradigm consists of three main components: recognition, adaptation, and generation. In the recognition phase, Langchain identifies the user's intent and context from the input. In the following experiment, it is the prompt that told GPT to analyze data and find some interesting insights.

The adaptation phase then utilizes a feedback mechanism to learn from user feedback and adjust its behavior accordingly. Finally, in the generation phase, Langchain generates a response based on the refined understanding of the user's intent.

The ReAct paradigm shares similarities with reinforcement learning. It involves an iterative process where the model interacts with an environment, receives feedback, and adjusts its behavior accordingly. This feedback loop helps the model improve its performance over time. Similarly, reinforcement learning algorithms also learn through trial and error, receiving rewards or penalties based on their actions.

The Experiment

The associated data for this experiment was obtained from D3BATT project at Massachusetts Institute of Technology in collaboration with Toyota Motors. The associated paper was published in Cell Joule shown in the following illustration.

D3BATT data was parsed and converted from MATLAB format to Python pandas data frame. A new column remaining_pct was added and used as target for Machine Learning prediction for the percentage of remaining useful life of battery.

Since GPT was left alone to do its works autonomously, several tools were provided to help GPT to carry out its responsibility more effectively. These tools allowed it to search the web, read Wikipedia and write its own codes under the hood as illustrated by dataframe_agent Python function.

The instruction prompt below was constructed to ask GPT to analyze data autonomously and produce some graphs. No further guidance was provided except for the tools mentioned previously.

Autonomous analysis of D3BATT data yielded some statistical profile of several important attributes. Correlations were also computed for several pairs of attributes.

Next, GPT was asked to perform a different task. This time to build a machine learning model using Random Forest Regressor and rank the importance of data attributes from contributing the most to the prediction outcome to the least.

We are also quite interested to know how well the ML model perform by measuring how far the predicted values are from actual. Mean absolute percentage error (MAPE) and R-squared (R2) are 2 commonly used metrics to measure ML Regressor models. The prompt was modified by instructing GPT to calculate the metrics mentioned.

Scenario 5 - Numerical Methods for Partial Differential Equations for Traffic Flow

Finally, no discussions about Numerical Methods is ever complete without talking about Partial Differential Equations (PDE).?PDEs play a crucial role in modeling traffic flow by capturing the complex dynamics and interactions of vehicles on road networks, allowing for the development of accurate and efficient numerical methods to analyze and predict traffic behavior. In this experiment, GPT was asked to model a numerical method for traffic travelling in a straight line along a highway.

Various mathematical expressions were identified and expressed in Latex in Python as follow:

The problem statement of this traffic flow was used as prompt to get response from GPT:

Comparison of GPT-generated numerical method with Mattheji, et al (1987) Partial Differential Equation book:

Conclusion

Food for Thought

In light of recent scandals that rocked the Academia and Scientific research communities, perhaps GPT can lend a hand to help with peer review, check-and-balance and probably policing for its knowledge, discipline, thoughtfulness, forthrightness, common sense, and autonomous traits. Instead of just thinking how to regulate AI, doing the opposite might be equally important.