How Confluent acquired WarpStream for $220m after just 13 months of operation

In August of 2023, WarpStream shook up the Kafka industry by announcing a novel Kafka-API compatible cloud-native implementation that used no disks.

Instead? It used S3.

It was a viral HackerNews post named “Kafka is Dead, Long Live Kafka!”.

Just a year and month later - on September 9, 2024 - Confluent acquired them for $220M (!).

Why did they do that?

WarpStream’s innovative architecture gave them two major advantages that nobody could compete with:

• massive cost savings ??
• massive operational simplification ??

The only drawback?

Latency was high.

? p99 for writes is 400ms ??

? p99 e2e (from write to read) is 1 second ??

Since WarpStream writes directly to S3 and also has to buffer writes so S3 PUT costs don't explode - it suffers from latency.

This wasn’t a problem. They had one key finding. ??

Kafka workloads are either:

• high volume and latency insensitive ??
• low volume and latency sensitive ???

It’s precisely the high volume workloads that cost a fortune.

Customers were happy to make the trade off - increase latency, but save costs.

The cost savings were indeed the juicy part:

Here I compare a Kafka cluster with:

• 1 GB/s Produce Inbound
• 3 GB/s Consume Outbound
• 7 day retention on disk

using retail AWS prices.

WarpStream is fundamentally:

• 10x cheaper than an unoptimized deployment. ?? ($500k vs $5.25M)
• 4x cheaper than an optimized deployment. ?? ($500k vs $2M)

We're talking a ~$500k cost versus $2M.

The right architecture can be the difference between millions of dollars a year in infrastructure costs.

Like this in-depth cost analysis so far?

A lot more is to come. Make sure to follow me on all platforms to not miss a beat:

• ? LinkedIn: Stanislav Kozlovski
• ? Twitter/X
• ? 2minutestreaming newsletter (2 minute reads)
• ? BigDataStream newsletter (long-form reads & interviews)

Where do WarpStream's savings come from?

?? Network costs.

The cross-zone charges you sustain from a regular Kafka deployment are the largest expense in a high-volume deployment.

They can be 80% of the total cost!

As you can see, even in the optimized deployment here, we have major charges from:

• producers writing to leader brokers in other availability zones
• brokers replicating to each other across zones

A less-optimized Kafka can cost you more than twice - $5.2M/yr.

A large chunk coming from EBS disks (no tiered storage) and consumer networking.

Anyway. If you optimize Kafka as much as possible, you’ll get to a ~$2.1M annual cost.

Out of that - $1.68M (80%) are UNAVOIDABLE network costs.

But WarpStream avoided them. ??

WarpStream drives all these costs to zero.

• every agent can be a leader for every partition, so producers write to agents in the same zone, removing that cross-zone charge ??
• broker replication doesn’t exist, so that charge is zero (it’s done inside S3 for free) ??
• consumers can read from brokers in the same zone, just like Kafka ??
• no disks are necessary at all (everything is in s3) ??

They offered it through a hybrid BYOC model, where:

• the customer hosts the stateless agents in their cloud
• WarpStream hosts a SaaS control plane

How?

This is where the operational simplicity comes in.

The secret sauce is the control plane. ??

All the complex logic lives inside the control plane.

It’s essentially a sequencer that leverages DynamoDB and gives each agent the offset for each partition it wants to write.

This lets them keep the agents dumb and stateless. Their write flow is roughy:

1. accept produce requests and batch the data
2. persist the data in s3
3. commit the offsets to the control plane
4. the control plane commits the offsets and s3 file references to DynamoDB
5. the agent acks the request to the producer

All the state they get from the control plane, and the complex offset synchronization happens there.

Agents are therefore allowed to scale up and down effortlessly, like nginx.

Ingenious design. Literally worth hundreds of millions of dollars. All within a year.

Kudos to the team ??

Liked this article?

It took me hours to research and write.

I ask you for one favor (it'll take you 2 seconds) ??

??

• Share it to your network! ??