Decoding Nvidia: Part Two

CUDA and Scientific Computing

Nvidia began its significant engagement with the scientific community and subsequently the AI researchers with the introduction of its CUDA (Compute Unified Device Architecture) technology. Launched in 2006, CUDA was revolutionary as it allowed researchers and developers to utilize Nvidia GPUs for general-purpose computing beyond traditional graphics rendering. This was a pivotal move for Nvidia, as it enabled their GPUs to be used in parallel computing, where massive numbers of computations can be performed simultaneously, a feature greatly beneficial for scientific and later, AI research.

The introduction of CUDA opened up Nvidia's GPUs to a broader range of applications, particularly in scientific computing fields that require intensive computational capabilities, such as simulations, fluid dynamics, and later, deep learning and AI. CUDA's launch marked Nvidia's initial foray into these areas, providing the tools that researchers in AI would soon utilize to accelerate their computations and model training.

Before the launch of CUDA, Nvidia's journey toward programmable capabilities in its chips had several significant milestones. This journey shows the intentionality towards Accelerated Computing and the exploration of new markets that were not yet mature, potentially not existent at that time.

GPUs and Programmability:

• GeForce 256 (1999): Often referred to as the "world's first GPU", the GeForce 256 introduced hardware transform and lighting (T&L) capabilities. It was primarily a graphics-focused chip but laid the groundwork for more complex computations by relieving the CPU from these specific tasks.
• GeForce 3 Series (2001): This series introduced programmable vertex shaders, marking a significant step towards greater programmability in graphics hardware. Vertex shaders allowed for more complex geometric calculations to be processed on the GPU, enabling more sophisticated visual effects and detailed 3D environments directly programmable by developers.
• GeForce FX Series (2002-2003): This series introduced programmable pixel shaders with the Cg (C for graphics) language, co-developed by Nvidia. Cg allowed developers to write programs that would run on the GPU to control how individual pixels are processed, which is crucial for creating more detailed textures and lighting effects in graphics.

The Convergence Towards General Purpose Computing

• G80 Architecture (2006): The release of the G80 GPU architecture was a turning point. This architecture powered the GeForce 8800 GTX and was the first to support unified shaders, which meant that the same shader units could process both vertex and pixel data. Unified shaders allowed for more flexible programming and resource allocation based on workload demands, which is crucial for non-graphical computations.
• Introduction of CUDA (2006): CUDA was launched as a parallel computing platform and application programming interface (API) model that allowed developers to use C++ (and later other high-level programming languages) to write programs that could execute on the GPU. It abstracted much of the hardware complexity and provided a more accessible means for exploiting the parallel processing power of Nvidia GPUs.
• AlexNet's Breakthrough Using Nvidia GPUs (2012): A team from the University of Toronto used Nvidia’s GTX 580 GPUs to train their convolutional neural network, which led to a significant reduction in error rates from 25% to 15%, a monumental improvement at the time. This success not only proved the effectiveness of GPUs for deep learning tasks but also marked a seminal moment for Nvidia in the AI space .

CUDA represented the culmination of Nvidia’s strategic shift from a graphics-centric company to a broad-based computing platform company. By making GPUs accessible for general-purpose computing, Nvidia not only expanded its market but also laid the foundational technology that would drive future innovations in AI and machine learning. CUDA enabled the democratization of parallel computing, making high-performance computing capabilities accessible to a broader range of scientists, engineers, and researchers, thereby accelerating innovation across various disciplines.

Zero Billion Dollar Markets

What is so striking about Nvidia generally and Huang in particular, though, is the extent to which this capability is the result of the precise opposite of an emergent process: Nvidia the company feels like a deliberate design, nearly 29 years in the making. The company started accelerating defined graphical functions, then invented the shader, which made it possible to program the hardware doing that acceleration. This new approach to processing, though, required new tools, so Nvidia invented them, and has?been building on their fully integrated stack ever since. - Stratechery

Huang describes "Zero Billion Dollar Markets" as areas "where there's no market yet but we believe there will be one." This forward-looking strategy involves positioning Nvidia in markets that are unformed and untested, where others are trying to understand why Nvidia has chosen to enter. This is evident in Nvidia's early moves into industries like automotive with GPU-based computing solutions, anticipating the future significance of software in cars .

Throughout the evolution of Nvidia, there are several examples of how the company has explored and expanded into these emerging markets:

• PC Gaming: Nvidia was early in recognizing the potential of PC gaming, shaping the market as it grew.
• Workstation Design: They reimagined workstations, which are now predominantly powered by Nvidia technology.
• Supercomputing and AI: Nvidia also led the transformation of supercomputing, democratizing its access and redefining its use toward AI and machine learning applications .

Huang highlights the creation of markets as a strategic advantage, noting that by the time these markets mature, there are few competitors structured to compete effectively against Nvidia. Beyond just entering markets, Nvidia focuses on creating ecosystems around their technologies, involving developers, customers, and partners. This ecosystem development is essential for sustaining leadership in these new markets and is likened to building a network rather than just a product or service .

Parting Thoughts and Looking Forward

I believe that you can only connect the dots in a straight line when looking backwards. Despite this, Jensen's strategy appears to have remained consistent over several years. While it's hard to believe that everything was entirely clear at each stage, it's evident that an exceptional visionary and strategic approach has been present since the early stages of Nvidia's history.

In the final article of this series, I will further explore the concept of Accelerated Computing. I will discuss how this has not only driven Nvidia's past success but also shapes its entire forward-looking strategy.