Amazon’s Strategic Move into Custom Chips to Excel in the Generative AI Race

In a modest office in Austin, Texas, Amazon is quietly revolutionizing the chip-making industry. Here, a small team of experts armed with microscopes, soldering irons, and tweezers is busy designing microchips to power data centers and the burgeoning field of Artificial Intelligence (AI).?

These chips, however, are not sourced from usual industry giants like Nvidia or AMD; they are Amazon’s brainchild. This move aims to reduce cost and enhance performance in response to Amazon's significant consumption of data center chips.

AWS CEO Adam Selipsky said, "The entire world would like more chips for doing generative AI, and whether that's GPUs or whether that's Amazon's own chips that we're designing, I think that we're in a better position than anybody else on Earth to supply the capacity that our customers collectively are going to want."

Amazon’s Custom Chips: Pioneering the AI Boom

Amazon Web Services (AWS), the cloud computing behemoth and Amazon’s most profitable segment, is leading this initiative. AWS CEO Adam Selipsky has pointed out the pivotal role these chips play in powering large language models essential for the AI boom. With a Q2 operating income of $5.4 billion, AWS has become a crucial funder for Amazon’s venture into custom silicon and an expanding suite of developer tools.

Mai-lan Tomsen Bukovec, VP of Technology at AWS, said, "Many of our AWS customers have terabytes or petabytes or exabytes of data already stored on AWS, and they know that that data is going to help them customize the models that they're using to power their generative AI applications."

In 2015, Amazon made a strategic acquisition of the Israeli startup Annapurna Labs, marking a significant step into the chip industry. Expanding its operations, AWS designs chips across multiple locations, including Silicon Valley, Canada, and a larger facility in Israel. Post-design, these chips are sent to manufacturers like TSMC in Taiwan for production.?

The journey into custom silicon began earlier in 2013 with the development of Nitro, a specialized piece of hardware that has become AWS’s highest-volume chip, boasting over 20 million units across AWS servers. Further intensifying its presence in the chip market, Amazon unveiled the ARM-based server chip Graviton at its 2018 re: Invent conference, positioning it as a competitor to x86 CPUs from established players like AMD and Intel.

Matt Wood, VP of Product at AWS, said, "We're into our third generation of our graviton chip that provides acceleration in terms of speed and cost efficiency and power for very general web-based workloads."

Competing with Tech Giants

Amazon’s strategy involves developing its own chips to compete with tech giants like Intel and Nvidia. It includes the creation of two AI chips: Trainium and Inferentia.?

Launched in 2021, Trainium is designed for training machine learning models, including those used in generative AI, offering a 50% price-performance improvement. Inferentia, introduced in 2019, delivers low-cost, high-throughput machine learning inference.

Matt Said, "With Inferentia, you can get about four times more throughput and ten times lower latency by using Inferentia than anything else available on AWS today."

Despite these advancements, Nvidia’s GPUs still dominate the market, especially in training large language models. AWS has introduced new AI acceleration hardware powered by Nvidia's H100s.?

Swami Sivasubramanian, vice president for data and machine learning services at Amazon Web Services, said this Nividia H100s-powered tool "accelerates performance by up to 6x and reduces training costs by up to 40% as compared to EC2 P4 instances."

Amazon is not the only non-chip giant entering the custom silicon field. Apple (M-series chips) and Google have also ventured into creating their own chips. Google has deployed its cloud Tensor Processing Units (TPUs) for nearly eight years.

Amazon’s Generative AI Strategy

Long before the current frenzy around Generative AI took hold, Amazon was already laying the groundwork for a comprehensive AI infrastructure. This strategic initiative was in place well before Amazon ventured into chip manufacturing or began using these chips to train Large Language Models (LLMs).

Swami Sivasubramanian stated, "In the late 1990s, we were the first one to actually leverage machine learning-based technologies to reinvent recommendation engines, and we leveraged machine learning to do things like better product search and then automating leveraging robotics and computer vision in our Amazon FCs or fulfillment centers to help products ship faster to actually reinventing completely new customer experiences with things like Amazon Alexa."

The generative AI race intensified with the launch of OpenAI’s ChatGPT in November 2022, followed by Google’s Bard. Amazon responded two months later by announcing its large language model, Titan, and Bedrock, a cloud service to aid developers in leveraging generative AI.?

Sivasubramanian reflected on the timeline of developments in the AI industry and pointed out that Amazon's journey in AI and chip engineering did not hastily begin as a reaction to the emergence of ChatGPT. Instead, the advent of such technologies only hastened existing customer dialogues and increased eagerness toward generative AI deployments. This statement underscores that Amazon's foray into AI and custom chip development was a calculated move, rather than a rapid response to shifting market trends.

Amazon has opted for a different approach than directly competing with ChatGPT, focusing instead on tools and services to enable its vast customer base to tap into generative AI.

Chirag Dikate, VP Analyst at Gartner, said, "So if you look at the Bedrock strategy that they are trying to focus on, they are betting the farm on the fact that enterprises might not necessarily be building out their own GPT models."

Bedrock gives AWS customers access to LLMs made by Anthropic, Stability AI, AI21, and Amazon's own Titan.

Matt Wood, VP of Product at AWS, explains that Titan is a family of foundational models. It has text-based models which are great for generative text, so creating marketing copy and advertising, chatbots, those sorts of things. And it has an embedding model which is great for personalization and ranking those sorts of use cases.

Amazon reports that many customers, including Philips, 3M, Old Mutual, and HSBC, are actively using its AI products. During its Q2 earnings call, the company highlighted that AI significantly drives AWS business, bolstered by over 20 machine learning services. This surge in AI-driven business underscores the growing impact and reach of Amazon's AI and machine learning offerings in the global market.

Sivasubramanian says that Amazon doesn't believe that one model will rule the world. The company wants its customers to have state-of-the-art models from multiple providers because they will pick the right tool for the right job.

Amazon added AWS HealthScribe to its AI portfolio in July. This service assists doctors in automatically drafting patient visit summaries, among other functions. Another significant addition to the AWS AI lineup is CodeWhisperer, which further broadens the scope and utility of Amazon's AI tools and services.?

While introducing CodeWhisper at an event, Sivasubramanian said that CodeWhisperer generates code recommendations from natural language prompts based on contextual information. Participants who use CodeWhisperer were 27% more likely to complete their tasks successfully, and they did it 57% faster on average.

The company also offers Amazon Sagemaker, a fully managed service that AWS provides. It enables developers and data scientists to quickly and easily build, train, and deploy machine learning (ML) models.

The Future of Amazon’s AI Ventures

In June 2023, AWS also announced a $100 million generative AI innovation center.

AWS CEO Adam Selipsky said, "We have so many customers who are saying, I want to do generative AI, but they don't necessarily know what that means for them in the context of their own businesses. And so we're going to bring in solutions architects and engineers and strategists and data scientists to work with them one-on-one."

Looking ahead, Amazon’s strategy in the generative AI space revolves around leveraging its dominant position in cloud services and its vast data storage capabilities. AWS’s massive customer base provides a significant advantage, as many companies are already familiar with AWS’s ecosystem and will likely use its AI tools and services.

While the generative AI race is still in its early stages, Amazon is positioning itself as a key player in this evolving field. The company is focused on responsibly and securely developing AI tools, aligning with regulatory frameworks, and addressing national security concerns.?

As the development of new generative AI applications and the chips needed to power them continue to accelerate, Amazon’s strategic investments in custom silicon and AI services could significantly impact the tech industry’s landscape.

Chirag Dikate from Gartner highlights the opportunity for Amazon in the AI landscape: "At the end of the day, Amazon does not need to win headlines. Amazon already has a really strong cloud install base. All they need to do is to figure out how to enable their existing customers to expand into value-creation motions using generative AI."