The Unsung Heroes of AI: GPUs, TPUs, and NPUs Explained

Hi There.

I am Manjit Singh and Welcome to the 3rd edition of FutureFrame.

Today, we will uncover the unsung heroes powering one of the most transformative technologies of our time—Artificial Intelligence, the term that’s now reshaping industries and everyday life alike.

Everyone's heard of ChatGPT, but have you ever wondered about the powerhouse hardware working behind the scenes to make such advanced AI possible?

While everyone’s amazed by tools like ChatGPT, not many stop to think about the incredible hardware that makes it all possible. It’s not just the clever algorithms at play—it’s the GPUs, TPUs, and NPUs working tirelessly to handle the massive computations behind the scenes.

Before diving into these processors, let’s first understand why traditional CPUs fall short when it comes to handling the demands of AI.

CPU: The Ultimate Jack of All Trades in Computing

For years, CPUs (Central Processing Units) have served as the cornerstone of computing, running everything from operating systems to daily applications. Known for their versatility, they can manage a broad array of tasks effectively and reliably—a true jack of all trades in the computing world.

Why CPUs Struggle with AI

When it comes to Artificial Intelligence, CPUs hit their limits. AI tasks like training neural networks or running real-time inference demand massive computational power and parallelism—areas where CPUs aren’t optimized. Their architecture is built for sequential tasks, meaning they process one instruction after another, whereas AI workloads require simultaneous processing across millions of data points.

Running AI on CPUs comes with these key drawbacks:

1. Limited Parallelism

CPUs are built for general-purpose tasks, excelling at sequential operations like running applications and managing operating systems. However, AI tasks, especially deep learning, demand simultaneous processing of vast datasets. With a limited number of cores, CPUs struggle to efficiently handle such parallel computations.

Example: Training a neural network involves extensive matrix multiplications. CPUs process these sequentially, taking significantly more time compared to hardware designed for parallel tasks.

2. Inefficiency in Specialized AI Tasks

AI operations like matrix multiplications and convolutions are repetitive and require hardware optimization. CPUs lack the specific architecture to handle these efficiently, resulting in slower performance and higher energy consumption.

Example: Running large language models like GPT on CPUs consumes more power and time compared to GPUs or TPUs, which are tailored for such tasks.

3. Challenges with Big Data

AI tasks often deal with massive datasets like images, videos, and text. CPUs struggle to process these quickly, creating bottlenecks in both training and inference.

Example: Real-time video processing for autonomous driving would face delays on a CPU, making it impractical for safety-critical scenarios.

4. Scaling Constraints

Scaling AI workloads on CPUs requires adding more processors, which quickly becomes inefficient and expensive. In contrast, GPUs and TPUs leverage parallel architectures, making them far better suited for scaling AI applications.

For those who are new to Machine Learning -

Inference in AI is the process where a trained model makes predictions or decisions based on new data. Think of it as "using what the AI has learned" to solve a problem or answer a question.

Example:

Imagine you trained an AI model to recognize cats and dogs by showing it thousands of pictures of each. During the training phase, the AI learned what features (like shapes, patterns, or colors) distinguish cats from dogs.

Now, when you show the trained model a new picture it hasn’t seen before, the inference process kicks in. The AI analyzes the picture and applies what it learned during training to determine whether it’s a cat or a dog.

In simple terms:

• Training = Learning.
• Inference = Using what was learned.

Both training and inference require significant computational power and parallel processing capabilities because neural networks are inherently embarrassingly parallel. This means that many of the calculations involved, like processing inputs and weights in a neural network, can be done simultaneously.

This is where CPUs fall short, and GPUs step in to take AI processing to the next level.

GPU : More than a graphics processor

The Graphics Processing Unit (GPU) transformed the gaming industry in the early 1990s, originally created to improve graphics rendering for video games. This breakthrough enabled more immersive gameplay experiences, laying the groundwork for iconic franchises like Grand Theft Auto (GTA), which leveraged the enhanced graphical capabilities GPUs provided. A major milestone came in 1999 with the release of the Nvidia GeForce 256, the first dedicated GPU. This innovation revolutionized gaming by allowing developers to design expansive open worlds and intricate character animations, as seen in GTA III and its successors.

As GPUs advanced, they transitioned from being solely gaming tools to indispensable components for general computing, including artificial intelligence.

Their ability to perform parallel processing enables them to handle numerous calculations at once, making them perfect for training complex neural networks in AI. This evolution has positioned GPUs as a driving force behind technological progress across diverse domains, from gaming to cutting-edge scientific research.

NVIDIA unlocked the true potential of GPUs, pivoting from a hardware company to an AI powerhouse—now worth trillions and leading the AI revolution!

Understanding Flynn’s Taxonomy: A Guide to Computing Architectures

Flynn’s Taxonomy is a classic way to classify computer architectures based on how they process instructions and data. It’s divided into four categories:

• SISD (Single Instruction, Single Data): A single instruction operates on a single data stream at a time. Traditional CPUs fall under this category, making them ideal for sequential tasks.
• SIMD (Single Instruction, Multiple Data): A single instruction operates on multiple data streams simultaneously. GPUs thrive here, making them highly efficient for parallel processing tasks.
• MISD (Multiple Instruction, Single Data): Rarely used in practice, this architecture processes multiple instructions on a single data stream, mainly for specialized use cases.
• MIMD (Multiple Instruction, Multiple Data): Modern CPUs and distributed systems use this for multitasking and complex workloads.

Why GPUs Excel: SIMD Architecture

GPUs primarily operate under the SIMD (Single Instruction, Multiple Data) paradigm. This allows them to apply the same instruction, such as a matrix multiplication, to multiple data points simultaneously. This parallelism is ideal for repetitive, large-scale computations, like those needed in AI and graphics rendering.

Example: SIMD in GPUs for Neural Networks

Let’s consider a neural network where we calculate the dot product of two matrices during training:

1. Each row of the first matrix represents an input feature, while each column of the second matrix represents a weight.
2. The GPU processes multiple rows and columns simultaneously, performing the same multiplication operation on different data points.
3. This parallelism drastically reduces the time required for training compared to a CPU, which would handle each operation sequentially.

Real-World Application: Imagine processing a 4K video in real-time for object detection. Each pixel represents data that needs analysis. A GPU can apply the same instruction (e.g., edge detection) to all pixels at once, making real-time processing possible.

To truly understand how GPUs became the backbone of AI, let’s dive into the hardware architecture that makes them so uniquely powerful for AI workloads

Let’s dive into the details and break down this powerful GPU architecture step-by-step :

1. Streaming Multiprocessors (SMs)

• What They Are: The GPU is divided into multiple Streaming Multiprocessors (SMs), which are the workhorses responsible for parallel processing. Each SM contains multiple cores that execute tasks simultaneously, leveraging the SIMD (Single Instruction, Multiple Data) paradigm.
• L1 Cache: Each SM is equipped with a small, dedicated L1 cache, which stores data that is frequently accessed by that particular SM. This minimizes delays caused by repeatedly fetching data from slower memory levels.

2. L2 Cache

• What It Does: The L2 cache acts as a shared memory resource across all SMs. When data cannot be found in the L1 cache, the request moves to the L2 cache, which is faster than the global memory but shared among all SMs.
• Why It’s Important: The L2 cache ensures that data can be accessed quickly across multiple SMs, improving efficiency for tasks requiring communication or data sharing between them.

3. DRAM (Global Memory)

• What It Does: DRAM, or global memory, is the primary memory of the GPU. It stores all the data required for computation, including input datasets, intermediate results, and the final outputs.
• Speed vs. Size: While DRAM has high capacity, it is slower compared to L1 and L2 caches. Efficient memory management is crucial to avoid bottlenecks during GPU-intensive tasks.

Flow of Data

1. Data Retrieval: Data begins in DRAM, which holds the dataset for the task at hand.
2. Caching Process: Frequently accessed data is moved to the L2 cache, which is shared by all SMs, and then further stored in the L1 cache of the specific SM performing the computation.
3. Parallel Execution: SMs process the data in parallel, with each core working on a small portion of the workload.

While GPUs revolutionized AI with their incredible parallel processing power, Google took it a step further with a processor built specifically for deep learning—enter the TPU, the AI specialist.

TPUs: Google’s AI Game-Changer

Tensor Processing Units (TPUs) are specialized processors designed by Google to handle the unique demands of AI and deep learning. Unlike GPUs, which were originally built for rendering graphics and later adapted for AI, TPUs were purpose-built from the ground up to accelerate machine learning workloads.

But , what is a Tensor?

A tensor is a mathematical structure used to organize and represent data in machine learning and deep learning. Think of it as a generalization of vectors and matrices to higher dimensions.

How Tensors Work:

• Scalars (0D Tensor): A single value, like a number (e.g., 5).
• Vectors (1D Tensor): A list of numbers, like [1, 2, 3], representing a single dimension.
• Matrices (2D Tensor): A table of numbers with rows and columns, like a spreadsheet.
• Higher Dimensions (nD Tensor): Tensors can extend to 3D, 4D, or more dimensions, making them ideal for representing complex datasets like images, videos, or text.

TPUs are optimized to handle tensor operations, such as matrix multiplications and additions, which are fundamental to neural networks.

For instance:

• During training, tensors represent inputs, weights, and biases.
• The TPU performs rapid calculations on these tensors to adjust the network and improve its predictions.

In essence, tensors are the backbone of data representation in AI, and TPUs are designed specifically to process them efficiently. This synergy is what makes TPUs so powerful for machine learning and deep learning tasks.

Understanding TPU's Architecture

Here's a step-by-step breakdown of its components and workflow:

1. PCIe Gen3 x16 Interface (Off-Chip I/O)

• This interface connects the TPU to the host system (like a CPU).
• Data and instructions for machine learning tasks are sent from the host to the TPU through this interface at a speed of 14 GB/s.

2. DDR3 DRAM Chips (Data Buffering)

• These external memory chips store large datasets and intermediate results.
• The DDR3 interfaces transfer data between DRAM and the TPU at speeds of up to 30 GB/s.
• This ensures the TPU has access to the required weights, activations, and inputs needed for computation.

3. Unified Buffer (Local Activation Storage)

• This buffer serves as local memory for the TPU to temporarily store activations (intermediate results during neural network processing).
• It ensures faster data access compared to repeatedly fetching data from DRAM.

4. Systolic Data Setup

• A systolic array is a specialized hardware block designed for matrix computations.
• It streams data in a synchronized manner to maximize efficiency and minimize latency.
• This setup ensures that data flows smoothly through the TPU’s processing pipeline.

5. Matrix Multiply Unit

• The matrix multiplication unit is the heart of the TPU, capable of performing 64,000 multiply-accumulate operations per cycle.
• This unit is optimized for tensor operations like those used in neural networks, such as matrix multiplications required for forward and backward passes during training.

6. Weight FIFO (Weight Fetcher)

• FIFO (First-In-First-Out) storage fetches weights (parameters of the neural network) from the DDR3 memory.
• These weights are then fed into the matrix multiplication unit at high speed (30 GB/s).

7. Accumulators

• After matrix multiplication, the results are sent to accumulators to combine intermediate results.
• This is crucial for summing up values during tensor operations.

8. Activation Functions

• After the accumulator stage, activation functions (like ReLU or Sigmoid) are applied to introduce non-linearity into the model.
• This step allows neural networks to learn complex patterns.

9. Normalize/Pool

• Normalization ensures data consistency, while pooling operations (like max pooling or average pooling) reduce the size of feature maps.
• These steps help make computations more efficient and reduce the model’s size.

10. Control Units

• Various control units manage data flow between different components, ensuring smooth communication between the host system, DRAM, and TPU processing units.
• They optimize the utilization of the systolic array, buffers, and memory interfaces.

Data Workflow Example

1. Data and instructions are sent from the host via the PCIe interface.
2. Inputs, weights, and activations are fetched from DDR3 memory into the unified buffer.
3. The systolic array processes the data in a pipeline, performing matrix multiplications.
4. Results are accumulated, activation functions are applied, and outputs are normalized/pooled.
5. Final results are sent back to the host for further use.

TPUs and TensorFlow

Tensor Processing Units (TPUs) were purpose-built by Google to accelerate TensorFlow, their open-source machine learning framework. TensorFlow relies heavily on tensor operations like matrix multiplications and convolutions, which TPUs are optimized to perform efficiently. This makes TPUs exceptionally fast for training and inference tasks compared to CPUs and GPUs.

TPUs seamlessly integrate with TensorFlow, allowing developers to switch from CPUs or GPUs to TPUs with minimal code changes using TensorFlow's built-in APIs.

Additionally, TPUs enable scalable AI workloads, handling large models like BERT or GPT with distributed processing across multiple TPU cores or pods.

Through Google Cloud, TPUs are also accessible as a service, providing cost-effective, high-performance AI development without requiring physical hardware.

Example Workflow

1. A machine learning developer creates a model in TensorFlow.
2. With just a few lines of code, the developer switches the backend to TPU by using tf.distribute.TPUStrategy
3. The model trains or performs inference on the TPU, significantly reducing processing time while maintaining TensorFlow’s flexibility.

If GPUs revolutionized AI and TPUs took it to the next level, you might wonder—why do we need NPUs? Let’s explore what makes Neural Processing Units a game-changer in their own right.

NPUs: The AI Specialists for Everyday Devices

Neural Processing Units (NPUs) are purpose-built processors designed specifically to handle AI tasks, particularly for real-time inference. While GPUs and TPUs are exceptional for large-scale AI workloads like training deep learning models or handling massive datasets, NPUs focus on making AI accessible and efficient in everyday devices like smartphones, IoT gadgets, and laptops.

NPUs are optimized for inference tasks, where the AI model applies what it has learned to new data, such as recognizing faces, processing voice commands, or enhancing images.

Unlike GPUs or TPUs, which often require higher power and thermal management, NPUs are lightweight and energy-efficient, making them ideal for portable devices.

• Parallelism at the Edge: NPUs process tasks in parallel but are tailored to smaller-scale workloads compared to GPUs.
• Low Power Consumption: Designed to run efficiently without draining the battery, they are perfect for edge devices like phones and wearables.

Although, I have already discussed about NPU's architecture once on Linkdein , let's revisit it again for our understanding of this powerhouse.

Understanding NPU's Architecture

Here's a step-by-step breakdown of its components

• Neural Compute Engines: Feature 2 SHAVE DSPs and MAC arrays to handle intensive tasks like matrix multiplications and tensor operations, essential for neural networks.
• Inference Pipeline: Includes modules for activation functions and data conversion, enabling efficient processing of non-linearities and data type transformations.
• Scratchpad SRAM: Provides high-speed temporary storage for frequently accessed data, reducing reliance on slower system memory.
• MMU and DMA: Streamline data transfer between SRAM, system memory, and compute units, minimizing bottlenecks.
• Global Control System: Manages the coordination of all components, ensuring efficient operation.
• IOMMU: Ensures secure memory sharing across processes, enhancing system efficiency and security.

Key Players in NPUs

Several companies have integrated NPUs into their products:

• Apple’s Neural Engine powers Face ID, real-time photo enhancements, and augmented reality in iPhones.
• Qualcomm’s Hexagon DSP in Snapdragon processors enhances AI in Android smartphones, enabling features like voice assistants and on-device translation.
• Intel’s Meteor Lake NPU brings AI capabilities to PCs, handling tasks like speech recognition and background noise cancellation.

Real-World Use Case: Smartphone AI

Imagine you’re taking a photo on your phone. The NPU instantly analyzes the scene, detects faces, adjusts lighting, and enhances the image—all in real-time without needing an internet connection. This kind of on-device AI is only possible because of NPUs.

Which is Better: GPU, TPU, or NPU?

The answer to whether a GPU, TPU, or NPU is better depends entirely on the use case, as each processor is uniquely designed for specific AI workloads.

GPUs are the most versatile, excelling at training large-scale AI models and handling a wide range of tasks like gaming and deep learning, thanks to their parallel processing capabilities. However, their higher power consumption makes them less efficient for real-time inference.

TPUs, on the other hand, are highly specialized for TensorFlow-based workloads, offering exceptional speed and energy efficiency for tasks like training massive models (e.g., BERT and GPT) in cloud environments, but their flexibility is limited outside TensorFlow.

NPUs shine in edge computing, providing lightweight, energy-efficient solutions for real-time AI inference on devices like smartphones and IoT gadgets, enabling tasks like facial recognition and voice processing without cloud dependency.

And that’s all for today, folks!

Hope this article gave you a clearer understanding of GPUs, TPUs, and NPUs, and how they’re shaping the future of AI.

If you found this insightful, don’t forget to like, share, and subscribe for more tech deep dives. Let’s keep the AI conversation going!