GPUs versus TPUs, which is better?

Lately, I have been thinking about how LLM processing might evolve, the associated implications and the advent of LLM application-specific integrated circuits (ASICs). Google TPU (Tensor Processing Units), is the obvious example as the most established ASIC. Other Hyperscale companies, including Meta, are investing in developing their own LLM ASICs. I will publish a follow up article on what the other Hyperscalers, Meta, Tesla and startups are doing including our exciting portfolio companies SambaNova Systems , Recogni and other up and coming startups.

• Google Trillium TPU: The Trillium TPU offers a 4.7× increase in peak compute performance compared to its predecessor, the TPU v5e. It is equipped with expanded matrix multiply units (MXUs) and increased clock speeds, which contribute to its enhanced performance.
• NVIDIA Blackwell GPU: The Blackwell GPU promises a performance improvement of seven to 30 times (marketing numbers, more like 2.5 to 5x assuming not bottlenecked by system level considerations) over NVIDIA’s previous H100 GPU, depending on the task. This significant boost is facilitated by the architecture's 208 billion transistors and a 10 terabyte-per-second interlink between two large chip dies.

Energy Efficiency

• Google Trillium TPU: It is 67% more energy-efficient than the TPU v5e, making it suitable for large-scale AI deployments with reduced power consumption.
• NVIDIA Blackwell GPU: Despite substantial performance gains, Blackwell GPUs are designed to use up to 25 times (marketing numbers, 1.7x higher efficiency when normalized to FP16 performance and more when utilizing FP4) less energy than their predecessors, enhancing their suitability for energy-efficient AI computations.

Architecture and Memory

Google Trillium TPU: Features double the High Bandwidth Memory (HBM) capacity and bandwidth compared to the previous generation. It also doubles the Interchip Interconnect (ICI) bandwidth, which allows for efficient scaling in data centers. The Trillium can scale up to 256 TPUs in a single pod.

• Pod-Level Scalability: Trillium can scale up to 256 TPUs in a single high-bandwidth, low-latency pod. This configuration allows for efficient processing and communication among the TPUs, enhancing performance for AI tasks.
• The memory capacity of Google's Trillium TPU is 32 GB of High Bandwidth Memory (HBM). This represents a doubling of the memory capacity compared to its predecessor, the TPU v5e, which had 16 GB of HBM. Additionally, Trillium features an HBM bandwidth of 1.6 TB/s, also double that of the previous generation, enhancing its ability to handle larger AI models and improve overall performance in training and inference tasks.
• Multislice Technology: Utilizing multislice technology, Trillium TPUs can operate across multiple slices within a pod or across multiple pods. This feature allows for data parallelism, enabling the training of large models by distributing workloads effectively across the available TPUs.
• Hundreds of Pods Connectivity: Beyond individual pods, Trillium TPUs can connect to hundreds of pods, forming a supercomputer capable of supporting tens of thousands of chips. This setup is interconnected by a multi-petabit-per-second datacenter network, facilitating extensive scalability for demanding AI applications.
• Integration with Titanium IPUs: The integration of Titanium Intelligent Processing Units (IPUs) further enhances scalability by allowing for offloading processing tasks, thus freeing up resources and improving overall compute performance.
• Linear Performance Scaling: The performance of Trillium TPUs scales nearly linearly (marketing claim) with the number of pods and slices deployed, making it suitable for large-scale AI model training and inference.

NVIDIA Blackwell GPU: Built with an improved version of TSMC’s 4-nanometer process, Blackwell GPUs contain 208 billion transistors and offer a 1.8TB/s bidirectional throughput via upgraded NVLink technology.

• Chip-to-Chip Interconnect: Blackwell GPUs utilize a unique architecture that combines two GPU dies into a single unified GPU, effectively doubling the performance capabilities. This design allows for a massive increase in computational power while maintaining coherence between the two chips. The High-Bandwidth Interface (NV-HBI) facilitates 10 TB/s of bidirectional bandwidth, enhancing data transfer rates between the chips.
• NVLink 5.0: The fifth generation of NVLink provides 1.8 TB/s of bidirectional bandwidth per GPU, supporting interconnectivity among up to 576 GPUs. This high-speed communication is crucial for scaling AI models that require extensive computational resources, such as trillion-parameter models.
• Micro Tensor Scaling: Blackwell introduces micro tensor scaling, which allows for fine-grained adjustments in tensor precision. This capability enables the GPUs to efficiently manage and optimize performance across various workloads, enhancing their ability to scale in response to the demands of complex AI tasks.
• Support for Large Models: The architecture is designed to support larger model sizes, with the ability to handle trillions of parameters. This scalability is essential for modern AI applications, which increasingly rely on large language models and other complex neural networks.
• Memory Capacity: The memory capacity of NVIDIA's Blackwell GPUs is 192 GB of HBM3E(High Bandwidth Memory 3E). This capacity is designed to facilitate substantial data handling, particularly for complex AI models and high-performance computing tasks. The architecture supports an impressive memory bandwidth of 8 TB/s, allowing for rapid data transfer rates essential for training large AI models efficientlyIn summary, the Blackwell architecture is engineered for exceptional scalability, enabling organizations to deploy extensive AI models efficiently while maintaining high performance and reliability.

Use Cases

• Google Trillium TPU: Optimized for AI workloads in Google Cloud's AI Hypercomputer, it supports large-scale AI models like Gemini and Imagen, with applications in large language and recommender models.
• NVIDIA Blackwell GPU: Designed for generative AI and data center applications, Blackwell GPUs are intended to power AI training and inference at scale, with adoption by major tech companies like Microsoft, Meta, and OpenAI.

How do you choose between Trillium TPU versus the Blackwell GPU?

In summary, if your focus is on large-scale AI models and you are leveraging Google Cloud, the Trillium TPU may be the better choice due to its optimized performance and scalability. Conversely, if you require a more flexible solution that can handle a variety of workloads, particularly in on-premises or hybrid environments, the Blackwell GPU might be more suitable.

Another key consideration is CUDA versus TPU software. One would need to consider framework compatibility (potential lock in to Google Cloud), performance needs and developer experience. Obviously, CUDA was introduced in June 23, 2007 to the world and has a long lead in terms of compatibility, fine-tuned control and more general capabilities.