A Brief Summary of the PINNsFormer Paper

The numerical resolution of partial differential equations (PDEs) has been widely studied in science and engineering. Traditional methods, such as the finite difference method, are common but suffer from issues like high computational cost, often referred to as the "Curse of Dimensionality."

Recently, research on Physics Informed Neural Networks (PINNs) has gained prominence, aiming to develop more effective methods for solving problems involving PDEs. With the increasing use of Large Language Models (LLMs) in academia and industry, the Transformer architecture has experienced a boom in the field of Machine Learning. Originally applied to words, the concepts of embeddings have been adapted to represent physical properties in this context.

Based on this, Zhiyuan Zhao and B. Aditya Prakash from the Georgia Institute of Technology , along with Xueying Ding from 美国卡内基梅隆大学 , published an article titled: PINNsFormer: A Transformer-Based Framework for Physics-Informed Neural Networks.

Below, I provide more details about the architecture and the idea behind the model. The link to the original article can be found at the end of this post.

The Proposed Architecture

• Raw Input: The independent variables of the PDE are passed as parameters. For example, if X and t are the independent variables, the input array should have the shape [n, m], where n represents the number of samples and m the number of independent variables.
• Pseudo Sequence Generator: The Transformer architecture is designed to identify long-term relationships in sequential data. In contrast, conventional PINNs use non-sequential data as inputs. To integrate these two models, it is necessary to transform the point inputs into temporal sequences. In the article, for a spatial and a temporal input, the Pseudo Sequence Generator performs operations such as:
• Spatio-Temporal Mixer (MLP): The architecture uses a fully connected Multilayer Perceptron (MLP) to map the spatio-temporal inputs into a high-dimensional space, enriching the data representation.
• Encoder-Decoder: The Encoder-Decoder layer in PINNsFormer is similar to that of the original Transformers, with one difference: the embeddings used in the encoder are reused in the decoder. This occurs because the focus of PINNs is to approximate the solution in the current state, unlike models for language or time series prediction.
• Output Layer (MLP): Although Transformers typically use nonlinear activation functions such as LayerNorm and ReLU, these may not be ideal for solving PINNs. The article proposes the Wavelet activation function, defined as:

[x,t]→{[x,t],[x,t+Δt],…,[x,t+(k?1)Δt]}

For more details, check out the full article: Link to the paper.