Who Will Claim the AI Chip Crown?

AI has sparked a new arms race in the tech industry, not only for software, but also for hardware. Specifically, AI chips, the engines powering everything from self-driving cars to intelligent assistants and advanced data analytics. AI applications require massive amounts of data and computation, which demand specialized AI chips that can handle the complexity and diversity of AI workloads. As we venture deeper into the AI era, the question on everyone's lips is: Who will claim the AI chip crown?

Before delving into this question, let's first understand what types of chips there are in the semiconductor industry and which ones are relevant for AI.

What Type of Chips Are Relevant for AI?

There are different ways to categorize chips based on their functionality, design, or application. One common way is to classify chips into 3 categories: 1) logic 2) memory and 3) analog.

1. Logic Chips: Chips that process data

Logic chips perform the computational and processing tasks in electronic devices. The most familiar types of logic chips are Central Processing Units (CPUs) and Graphics Processing Units (GPUs). They execute instructions and process data in computing devices. Logic chips also include microprocessors, digital signal processors (DSPs), Application-Specific Integrated Circuits (ASICs), and Field-Programmable Gate Arrays (FPGAs). System on a Chip (SoC) can also fall into this category, as they integrate various functional components, including logic processing units, onto a single chip.

Below are how logic chips relate to AI:

• CPUs (Central Processing Units): While not as efficient as GPUs or ASICs for AI tasks, CPUs are still used for AI, particularly in scenarios where the AI models are less complex or where resources are limited.
• GPUs (Graphics Processing Units): GPUs are crucial in AI, especially for deep learning and neural network training. Their ability to perform parallel processing makes them ideal for handling the large and complex calculations required in AI models. Both consumer and professional-grade GPUs are used, though professional GPUs are more common in AI research and development due to their higher performance capabilities.
• ASICs (Application-Specific Integrated Circuits): ASICs designed specifically for AI tasks, such as Google's Tensor Processing Units (TPUs), are optimized to execute AI algorithms more efficiently than general-purpose processors. These chips are custom-built for specific AI workloads, offering enhanced performance for tasks like neural network inference.
• FPGAs (Field-Programmable Gate Arrays): FPGAs are used in AI for their flexibility and adaptability. They can be reprogrammed to suit different AI tasks and algorithms, making them useful for prototyping and applications where requirements might change over time. FPGAs are particularly valuable in situations where the AI workload might need to be frequently updated or customized.
• DSPs (Digital Signal Processors): DSPs are used in AI for processing audio and visual data. They are efficient at handling the types of repetitive, mathematical computations required in signal processing, which is a key component of many AI applications, such as voice recognition and computer vision.
• SoC (System on a Chip): SoCs, particularly those used in smartphones and embedded systems, increasingly incorporate AI capabilities. These chips integrate CPU, GPU, and other components, including dedicated neural processing units (NPUs) or AI accelerators, to handle AI tasks like image and voice recognition directly on the device.

2. Memory Chips: Chips that remembers data

Memory chips are used for storing data. They are essential for virtually all computing systems, from simple microcontrollers to advanced supercomputers. Below are the main types of memory chips and how they relate to AI.

• DRAM (Dynamic Random Access Memory): Typically used for main system memory. Used extensively in AI for processing large datasets due to its high speed, though it is volatile.
• SRAM (Static Random Access Memory): Often used in cache memory for CPUs and GPUs, improving the speed of data access in AI computations.
• Flash Memory: Used in solid-state drives (SSDs) and USB flash drives and known for its ability to retain data without a constant power supply. Important for the storage of AI models and datasets, especially in edge AI applications where data might need to be stored locally on the device.

3. Analog Chips: Chips that process real-world signals into digital formats

Analog chips process real-world signals such as sound, light, temperature, or pressure and convert them into a digital format that can be processed by digital systems, or vice versa. While not directly involved in AI computations, analog chips can play a supportive role in AI applications. For example, in sensor processing or converting real-world analog signals into digital data that can be processed by AI algorithms.

In summary, the most relevant categories of chips for AI are primarily found within the logic and memory categories. Logic chips, particularly GPUs, are at the forefront of AI processing. Memory chips are equally essential for the storage and quick retrieval of AI data.

Who Are the Key Players of AI Chips?

Now we know what chips are the most important for AI applications, let's delve into the competitors in this arena and evaluate who might emerge as the ultimate victor. The key players largely fall into 3 categories: 1) Established Chipmakers 2) Big Cloud Companies 3) Emerging Start-ups.

Note: The companies listed in the table above are representative of the AI chip market but the list is not exhaustive.

Established Chipmakers

1. NVIDIA: Long known for its dominance in the GPU market, NVIDIA has seamlessly transitioned into the AI space. Their GPUs are not only popular in gaming but have also become the de facto standard in AI training due to their high processing power and efficiency. NVIDIA’s offerings, like the A100 and H100 chips, are specifically designed for AI workloads. A100 series was launched in 2022 and has been widely used for AI applications. The H100 contains 80 billion transistors, which is 6 times more than its predecessor, the A100 chip. This makes it even more capable of processing large amounts of data. Even though the H100 is not yet widely available, Nvidia has already announced its successor, the GH200. The GH200 is expected to be even more powerful than the H100 and will be available in 2024.

2. AMD: Traditionally NVIDIA's rival in the GPU space, AMD is also making strides in AI. On Dec 8, 2023, AMD announced the launch of MI300 series that aims to compete with Nvidia’s H100 chips in the AI space, offering more memory capacity and compute power. AMD’s MI300X platform has similar training performance to Nvidia’s H100 HGX platform, but it has lower costs due to its improved memory capabilities. It also has better memory specs than Nvidia’s upcoming H200 chip, which is expected to launch in Q2 2024. However, Nvidia has an advantage over AMD in terms of its software ecosystem, which includes CUDA programming languages and software, which help retain some customers who have invested in Nvidia’s GPUs. AMD is improving its software suite called ROCm, but Nvidia still has a head start on the software front.

Big Cloud Companies

1. Google: Google’s entry into the AI chip market with its Tensor Processing Units (TPUs) showcases its innovative edge. TPUs, used internally in Google's data centers, are designed specifically for high-speed machine learning tasks. Google's ability to optimize its software and hardware stack gives it a unique advantage.

2. Amazon's AWS: Amazon AWS Trainium series of chips are designed to optimize AI tasks. In Nov 2023, Amazon announced its new Trainium2 AI chips. Trainium2 chips are built for training AI models, including what AI chatbots like OpenAI’s ChatGPT and its competitors run on.

3. Microsoft Azure: In Nov 2023, Microsoft launched Azure Maia 100 chip, optimized for AI tasks and intended to compete with Nvidia's top-line AI chips. Maia 100 is the first in a series of Microsoft's Maia accelerators for AI.

Emerging Startups

A swarm of startups and emerging tech firms are also making significant strides in developing AI chips. Companies like Graphcore and Cerebras Systems are reimagining chip architectures to optimize for AI workloads, posing a serious challenge to established players.

Cerebras Systems created the world’s largest and most powerful AI chip, called the Wafer Scale Engine (WSE). The WSE is a chip that has 1.2 million neurons and 16 trillion synapses, which can perform up to 200 trillion operations per second. The WSE is designed to enable massive-scale AI training and inference for various domains, such as healthcare, finance, and education.

Graphcore develops intelligence processing units (IPUs), which are specialized chips for deep learning applications. Graphcore’s IPUs have a novel architecture that uses programmable logic to accelerate matrix operations on tensors. Graphcore’s IPUs can achieve up to 100 times higher performance than conventional GPUs for certain types of deep learning models.

What Factors Influence the AI Chip Race?

The winner of the AI chip crown will be determined by several key factors:

• Performance: The ability to process AI algorithms quickly and efficiently is paramount. This includes rapid data processing, low latency, and high throughput.
• Energy Efficiency: As AI models grow in complexity, energy efficiency becomes crucial. Chips that deliver high performance with lower power consumption will be more desirable.
• Versatility: The ability to handle a variety of AI tasks, from training complex models to executing real-time inference, will be a significant advantage.
• Ecosystem Support: Strong partnerships with software developers, access to a vast library of AI models, and robust developer tools will be essential for widespread adoption.

Looking Ahead...

As the AI chip industry evolves, the winning company will likely be one that not only develops high-performing chips but also builds a strong ecosystem around their technology. NVIDIA currently leads with its advanced GPU technology and comprehensive software support, but AMD is quickly catching up, challenging with cost-efficient and powerful alternatives. The entry of big cloud companies and innovative startups adds to the competition, making the outcome less predictable. Industry experts believe that versatility, energy efficiency, and strong partnerships will be key in determining the market leader. This ongoing race is set to revolutionize various sectors, heralding a new era in AI technology.

What do you think the future holds for AI chips? How do you see these developments impacting your industry or daily life? Please feel free to leave a comment below.