Compute Powering AI & Huang’s Law

It is a modern (tech) saying: Data is the new oil (attributed to British Mathematician Clive Humby, 2006). Oil revolutionized the world by its widespread use as an energy source. Energy from oil runs vehicles, heats homes, generates electricity, etc. The energy hidden in oil is released on combining with oxygen. Similarly, “intelligence” hidden in big data is released only when combined with Computing. Computing is then the new oxygen for AI.

A dominant fraction of this Oxygen comes from one source: GPUs by Nvidia. NVIDIA is the toast of the tech-world and of the stock market today. Announcement of Nvidia’s results is the most anticipated financial event. On February 22nd, 2024, Nvidia’s shares surged to create the largest-ever single-day increase in market capitalization of $277 billion! Nvidia became the largest corporation by market-cap of $3.3 trillion on June 18th, 2024, ahead of Microsoft, Apple, Google, Amazon, and Meta. These numbers move down and up, but Nvidia remains in the top 3. The quarterly announcement of Nvidia’s financial numbers has become the most anticipated event! The latest report that came on Aug 28th showed Nvidia revenue of $30B in the Apr-Jul quarter, more than twice that of the same quarter is 2023 and 4.5 times of 2022. How did they get to be in such a dominating position? AI boom and its insatiable hunger for computing caused this. Nvidia’s focused mission of making high-performance computing (HPC) accessible at modest budgets really brought it to this position. It is a story of foresight, persistence, and luck which always favors the prepared!

The Story of Nvidia

Nvidia started making GPUs (Graphics Processing Units) or add-on cards for 3D graphics and computer games in 1993. The demand for enhanced speed, resolution, and quality of imagery called for special hardware. “Moore’s Law” was in full force then, with the transistor-count in a chip doubling roughly every two years. The CPUs in the PCs got faster and cheaper by riding the Moore’s law wave in the last millennium. ?That growth hit a wall by the year 2000 as their complex designs couldn’t effectively exploit the additional transistors. The GPUs had simpler architectures that could productively utilize more transistors as they perform simple, near-identical calculations on large numbers of elements, like pixels of an image. Nvidia was not the only company making GPUs. They had competition from 3dfx, 3Dlabs, ATI (now part of AMD), S3, etc. Today, AMD and Nvidia are perhaps neck-and-neck in GPUs for gaming while others disappeared.

As the computing power in a GPU increased, parts of it were made programmable for interesting visual effects. Clever researchers saw the connection of GPUs with the specialized array processors built in the 1970s which used a SIMD (Single Instruction on Multiple Data) model. Prior experience was effectively recycled to implement foundational operations like matrix multiplication, FFT, sorting, etc., on the GPU by mid 2000s.

Democratizing High-Performance Computing

Jensen Huang , Founder-CEO of Nvidia, saw an opportunity to democratize high-performance computing. He envisioned Nvidia to provide computing for specialized/specific applications, 3D graphics being the first. GPUs released in 2006 had multiple identical processing units, instead of different ones to process vertexes and pixels. Nvidia positioned the GPUs as economical and accessible parallel processors, delivering about 350 GFLOPS at $400!!

Any hardware is only as good as the software and tool support available on it. At the core of the strategy was the CUDA parallel computing platform to exploit the GPU via high-level APIs. This came with state-of-the-art compilers, runtimes, debuggers, drivers, etc., for easy adoption as parallel processors. The end-to-end or full-stack approach is the real factor behind Nvidia’s phenomenal success in HPC. Though success was far from guaranteed, Jensen persisted with it. GPUs using CUDA started to power protein folding, oil & gas exploration, etc., in addition to graphics and media processing. Many in academia started to develop algorithms and techniques to use GPUs for different problems: Computer Vision, Ray Tracing, Graph Algorithms, Sorting, etc. By 2012, GPUs were being used as compute accelerators widely; 13 of the top-100 supercomputers used Nvidia GPUs, including 2 of the top 10. (Today, 53 of top-100 and 6 of top-10 use them. GPUs use less energy for computations and appear in 70 of top-100 and 7 of top-10 Green-500 supercomputers today.) Nvidia’s focus on HPC was apparently going to pay off over the coming years.

HPC to AI Computing

Life got more interesting, with luck meeting the prepared. Deep Learning burst into the scene to transform AI and, along with it, the compute landscape and Nvidia. Artificial Neural Networks had been around for decades as shallow, multi-layer perceptron networks. They, unfortunately, didn’t scale well to larger problems. A few researchers Geoffrey Hinton, Yann LeCun , and Yoshua Bengio were playing with Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) with many layers. They need huge amounts of data and compute power to train. Huge amounts of text, speech, and image data under varied conditions became available with the explosion of the internet, inexpensive sensors like cameras, microphones, and smartphones, etc. Compute was still a problem, however.

In 2012, AlexNet revolutionized AI landscape by sweeping ImageNet recognition challenge by a huge margin! Images are bulky and need large networks to process. Alex Krizhevsky trained a 60+ million parameter CNN on two Nvidia GTX580 GPUs in about 7 days. This couldn’t be done without GPUs. Very quickly, deep networks became the only game for most AI tasks, with GPUs supplying the needed oxygen. Deeper and bigger networks as well as newer architectures emerged later; they need more data and compute.

NVIDIA was quick to spot the potential and pivoted itself to an “AI first” company, concentrating strongly on compute for AI. Architectural features that suit AI computations, such as 16-bit and 8-bit floating point numbers, were added to the hardware. Equally notably, software tools and libraries were developed to exploit the GPU for Deep Learning. CUDA libraries like cuBLAS and cuDNN integrated seamlessly with the high-level platforms like TensorFlow and PyTorch built by other companies. As heavy architectures like Transformers came along, the demand for compute went through the roof. Ripples created by the introduction of LLMs like ChatGPT made AI a global buzzword. And, up went the ravenous demand for data and compute!

Comprehensive Approach beyond Hardware

Advances in chip technology makes GPUs faster each year. Nvidia constantly enhances the architecture, number representation, memory, etc, to suit major applications. Their latest Hopper GPUs have transformer engines to help train foundational models. More critically, Nvidia enhances the CUDA platform constantly for users to take advantage of the hardware painlessly. Combining all these, the overall performance of Nvidia GPUs has doubled yearly in the past decade. This is informally referred to as the “Huang’s Law”, to highlight Jensen’s focus on full-stack improvements.?

Compute power is a critical resource in today’s world. Nvidia sits comfortably with its dominant GPU offerings. Nvidia’s datacenter or hyperscaler market is 10 times larger than gaming today, with no credible competition. Other companies are trying to play catch-up. Google builds its own Tensor Processing Units (TPU) to accelerate AI. Other big companies and several startups are building AI processors as alternatives. Cerebras follows a radical approach to build Wafer-Scale Engines with a million compute cores. None of them are available widely in an easy-to-use manner.

Foresight, perseverance, and top talent elevated Nvidia to its highly envious position! There is much to learn from Nvidia for everyone. I write this piece in genuine appreciation of Nvidia’s strategy and success. I along with a few International Institute of Information Technology Hyderabad (IIITH) faculty and students were early adopters of GPU computing from 2007, for Computer Vision, Machine Learning, Graph Processing, Ray Tracing, String Sorting, etc. We worked with Nvidia researchers for it. I was also made a “CUDA Fellow” when Jensen Huang visited IIIT-H in 2008. Several IIIT graduates are in key roles in Nvidia, as they do in other tech companies. My high appreciation for Nvidia’s accomplishments and future potential is due to this exposure I have.

Points for the Future

Before I conclude, I want to study Nvidia from its latest financial results. The company earned $30B in the first quarter of 2024.? A whopping 87% of it (or $26.3B) came from sales to datacenters of big players like Meta, Google, Microsoft, and OpenAI. They use thousands of GPUs for in-house AI development; some also rent them out. Microsoft and OpenAI are reportedly establishing an AI supercomputer with millions of GPUs, at a rumoured cost of $100B. The share of GPUs sold to datacenters will shoot up given all the AI hype. Are we back to the old world with only big budgets getting access to critical computing? The latest H100 GPUs cost $40K a piece; server versions cost a lot more. Is the original dream of democratizing supercomputing still alive? Even the top universities struggle in AI due to insufficient computing resources. What, then, is the fate of other universities and startups? Nvidia must come with ideas to democratize computing again for the coming decades.

The environmental impact of modern AI is a grave matter. The share of energy used by datacenters has shot up in recently. Further explosive growth may be imminent, going by the plans of the major players. Will compute as the new oxygen reduce access to the old oxygen for humanity? We must factor energy consumption when advancing AI responsibly. Can Nvidia lead on this “green” front also? With its dominant position in AI computing, improvements in Nvidia’s efficiency will positively impact the environment. The world expects revolutionary advances and not modest ones. I hope Nvidia will take this as their responsibility as well as a factor for competitive advantage.

?