The Transformative Power of Algorithms and Computing: A tale of two fields!

During this holiday season, as I reflect on my two decades in High-Performance Computing (HPC), working in Deep Learning (DL) and Computational Fluid Dynamics (CFD), one realization stands out: the striking parallels between backpropagation and the SIMPLE (Semi-Implicit Method for Pressure-Linked Equations) algorithm. These two breakthroughs, though from different domains, share a common thread in turning challenging theoretical ideas into practical tools that transformed their respective fields by harnessing HPC.

Neural networks, systems inspired by how the human brain processes information, had existed for decades but struggled to learn effectively from data. Similarly, the Navier-Stokes equations, which describe how fluids like air and water move, are fundamental to understanding fluid dynamics but were historically too complex to solve for real-world applications. Backpropagation and SIMPLE solved these fundamental challenges by breaking down complex coupled problems into manageable, iterative steps. While these algorithms were ingenious in their design, their transformative impact was realized through the parallel advancements in high-performance computing (multi-core CPUs and accelerators connected with a high-speed network in large clusters), which made their application to large-scale, real-world problems possible.

Backpropagation, formalized in the 1980s by David E. Rumelhart, Geoffrey Hinton, and Ronald J. Williams [1], changed the game for artificial neural networks. It efficiently solved the problem of training deep neural networks by computing gradients and adjusting weights systematically, making them a practical tool. Backpropagation is why deep learning applications like image recognition and language models are ubiquitous today.

Similarly, the SIMPLE algorithm, developed in the 1970s by Suhas V. Patankar and Brian D. Spalding [2], revolutionized CFD. The Navier-Stokes equations, known since the 19th century, were difficult to solve due to the pressure-velocity coupling issue. SIMPLE introduced an iterative correction process, allowing simulations of flows in applications ranging from airplane wings to HVAC systems, and many more in engineering. Like neural networks, the Navier-Stokes equations found their practical use only after a breakthrough method to solve those and the availability of sufficient computing power.

What’s striking is the shared methodology of these breakthroughs. Both decouple complex systems—backpropagation handles layers of weights, while SIMPLE separates pressure and velocity fields. Both rely on iterative error correction to converge toward a solution, whether it’s minimizing loss in a neural network or satisfying continuity in fluid flow. And both became transformative only when computing power advanced enough to handle the iterative calculations efficiently. These methods not only solved their immediate challenges but also laid the groundwork for future innovations, tackling problems we couldn’t dream of solving before. Personally, working across both the fields has been an incredible experience for me.

Have you seen parallels between advancements in seemingly unrelated areas?

[1] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.

[2] Patankar, S. V., & Spalding, D. B. (1972). A calculation procedure for heat, mass and momentum transfer in three-dimensional parabolic flows. International Journal of Heat and Mass Transfer, 15(10), 1787-1806.

Image credits: Conceptual illustration generated using DALL·E by OpenAI

Note: This article reflects my personal observations and is intended to highlight conceptual parallels between deep learning and CFD. While backpropagation and SIMPLE have been foundational in their respective fields, this is not to imply that these two methods are theoretically comparable.