Optimizing Networks for Billions: Scaling Efficiency and Speed in AdTech

To recap from my previous post, we process approximately 250 billion transactions daily, responding within 40ms-50ms. Of these, 220-230 billion are bidding endpoint requests, where we either bid or decide not to. Though we only convert a fraction of a percent into actual ads, this system favors scale, low latency, and cost over completeness, allowing us to disregard processing a small percentage of bid requests—i.e., if we don’t process 0.1% of our bid requests in any given time period, that’s largely fine.

Our pixel stack, responsible for impressions, clicks, conversions, and site events, handles tens of billions of transactions daily. Here, we prioritize completeness over latency and cost. While our software requirements differ between stacks, our network stack supports both.

Bidding Stack Network Architecture

Our bidding stack operates across four regions: US West, US East, Europe, and APAC, chosen for proximity to our markets and alignment with supply partners. We process far more traffic from supply partners (~220 billion requests daily) than from internet users (~10-20 billion requests daily).

We run our software on AWS, but AWS egress costs are high. Instead, we use our own backbone connecting all the AWS datacenters but sitting outside of the AWS infrastructure. Then we pull in a wire into the AWS network. This facility is called AWS DirectConnect (AWS DirectConnect)—a dedicated connection outside AWS infrastructure. For a high-scale network operation, a setup like this could save millions.

Peering: A Shortcut Through the Network Maze

One of the fascinating challenges in network design is reducing the number of hops between a source and destination. Each hop, or intermediary server, adds both latency and cost, which can accumulate quickly in a system as large as ours. Ideally, you'd want traffic between partners to flow as directly as possible, without bouncing across multiple networks. But the internet, by its nature, doesn’t always make this easy. IP routing tends to follow paths determined by factors like network congestion or peering agreements between internet providers, often causing packets to take unnecessarily long routes.

Enter Peering—an elegant solution to this problem. Peering allows us to bypass the chaotic routing of the open internet. Instead of sending packets through random intermediary hops, peering enables us to define specific routes—often as direct as a single hop from source to destination. Imagine drawing a straight line on a map between two points instead of zig-zagging through detours.

How does this work? When both we and our partners advertise our IP addresses within a Peering Exchange, such as those indexed in PeeringDB, requests between our systems are routed directly through the shortest path. If both parties are in the same peering exchange, the transmission happens across a single hop: source → peering exchange → destination. It’s as if we’ve built our own express lane for network traffic, avoiding the stoplights and traffic jams of the broader internet.

This isn’t just about reducing latency—though that’s a huge benefit. The fewer hops your traffic has to make, the fewer middlemen are involved, significantly reducing network transmission costs. For a system like ours, processing billions of transactions per day, the savings in both time and money are enormous. This elegant solution transforms what could be a chaotic, multi-hop network mess into a controlled, near-instantaneous communication channel.

By combining AWS DirectConnect, which reduces the cost of pulling data into our AWS environment, and strategic peering arrangements, we’ve optimized our network for both efficiency and scale. It’s a powerful, low-latency, cost-effective way to ensure that our systems remain responsive, even when handling an enormous volume of transactions.

Load Balancing

In the bare-metal world, we used Keepalived for load balancing at the border router. In AWS, we use AWS Route53, which supports up to 8 IP addresses per DNS entry. This setup balances traffic across our datacenters. While AWS supports load balancing through its own products, we found them too expensive for our use case. Instead, we built our own load balancing architecture using Nginx.

In the pixel stack, we stick to a standard Layer 7 architecture. However, in our bidding stack, we use Layer 4 load balancing. This means we can’t use Layer 7 features like routing by URL patterns, but it saves a ton of CPU and memory, making the load balancers lightning fast and cost-efficient.

Optimizing Communication: UDP over TCP

Our bidding stack consists of three layers: a load balancer, a Mux layer for operational tasks, and multiple bidders for bid computation. To minimize network costs and hops, we use AWS Placement Groups to keep these machines physically close together.

To make things further optimise, instead of using TCP for internal communication, we opted for UDP, eliminating the overhead of handshakes and retries. This significantly improves speed while maintaining an acceptable error rate. We’ve also fine-tuned UDP to handle larger packets more efficiently, enabling us to break and reassemble data with minimal overhead.

Put together, these optimisations save upwards of $5M a year compared to conventional designs and architectures. What such optimisation have you been performing on your end?