193. Innovate down the stack - AWS re:Invent 2024 recap Day 1

Peter DeSantis, Senior Vice President of AWS Utility Computing, upheld the Monday Night Live tradition by offering a behind-the-scenes look at AWS's latest innovations and the foundational technologies driving them.

It sounds like the keynote highlighted AWS's commitment to combining cutting-edge technology with robust security, particularly focusing on cloud security, custom silicon, and performance enhancements through the AWS Nitro System and Graviton4 processors. Here's a breakdown of the key themes:

Custom Silicon: Graviton4 Processors X Nitro System

- Optimized for Performance and Security: Graviton4 processors, designed in-house, promise enhanced efficiency and cost-effectiveness while maintaining stringent security measures.

- Nitro System Integration: AWS Nitro provides a hypervisor-free environment, offloading virtualization tasks to hardware, which enhances both security and performance. Nitro combined with Graviton4 enables:

a. Reduced attack surface.

b. Lower latency and higher throughput for compute-intensive applications.

The combination of custom silicon with AWS Nitro is a clear indicator of AWS’s focus on delivering performance with uncompromising security, particularly in handling AI, cloud-native, and enterprise workloads at scale. This keynote reinforces AWS’s leadership in cloud innovation while addressing the growing demand for secure, high-performance computing.

Latency-Optimized Inference for Amazon Bedrock

Amazon Bedrock is AWS’s custom service for building generative AI models. DeSantis highlighted how latency issues can impede real-time inference, especially in?agentic AI processes?that rely on sequential task completions.

With the new latency-optimized option, models like?Llama 3.1 405B?demonstrate remarkable performance improvements. Running on?AWS Trn2 chips, Llama 3.1 405B generates 100 tokens in just 3.9 seconds—significantly faster than competing platforms like Azure (6.2 seconds) and Google Vertex AI (13.9 seconds). This positions Amazon Bedrock as the go-to platform for real-time AI workloads.

Network innovation with 10p10u to manage massive AI clusters with ultra low latency

While virtualized compute is the foundation of cloud computing, enabling all that compute to transmit data is the job of the network.?So how does the world’s largest cloud scale its network to meet the increased demands of AI?

The demands on AI networks are particularly intense.?DeSantis noted that during training, every server needs to talk to every other server at exactly the same time.

The 10p10u network fabric is being specifically deployed in support of AWS’ UltraServer compute technology, which is being built out to run massive AI training workloads. Each Trainium2 UltraServer has almost 13TB of network bandwidth, requiring a massive network fabric to prevent bottlenecks.

“The 10p10u network is massively parallel, densely interconnected, and 10p10u network is elastic,” DeSantis explained. “We can scale it down to just a few racks, or we can scale it up to clusters that span several physical data center campuses.”

Patch panels are a common sight in many data center networks, with a stream of cables connecting into a panel. With the complexity of the 10p10u network, AWS found that its existing patch panel approach wasn’t going to be enough. So it created something new. AWS developed a proprietary trunk connector that combines 16 separate fiber optic cables into a single connector.

“What makes this game changing is that all that complex assembly work happens at the factory, not on the data center floor, and this dramatically streamlines the installation process and virtually eliminates the risk of connection errors,” DeSantis said. “Now, while this might sound modest, its impact was significant. Using trunk connectors speeds up our install time on AI racks by 54%, not to mention making things look way neater.”

Peter introduced 10p10u, which enables AWS to provide ten petabytes of network capacity to thousands of servers with under ten microseconds of latency.

Deep Roots of Towering Trees

DeSantis used this analogy to emphasize AWS’s long-term strategy: Just as trees with deep roots provide stability and growth over time, AWS’s continuous investment in its infrastructure forms a solid foundation for scalable, secure cloud services.

AWS is positioning itself to lead in AI by building security-first infrastructure for emerging AI workloads, ensuring trust and compliance in sensitive machine learning and data-processing environments.

By pulling back the curtain on AWS’s foundational technologies, DeSantis reinforced AWS’s dedication to both technological excellence and customer-centric security, setting the stage for the next era of scalable cloud computing solutions. His presentation underscored AWS's commitment to pushing boundaries in cloud computing while prioritizing security, performance, and scalability.

Source: aws re:Invent 2024, networkworld