Kafka's Evolution: Zookeeper vs. KRaft

Apache Kafka has long been a cornerstone in the world of distributed systems, offering a reliable event streaming platform for applications ranging from real-time analytics to data pipelines. Historically, Kafka relied on Apache ZooKeeper for metadata management and cluster coordination. However, with the introduction of Kafka Raft (KRaft), the platform is transitioning to a more streamlined architecture.

This blog explores ZooKeeper and KRaft, their differences, and when to use one over the other. It also examines the implications of this shift on Kafka’s architecture and operations.

ZooKeeper in Kafka: The Legacy Coordination System

Overview of ZooKeeper

ZooKeeper is a distributed system that organizes data in a hierarchical namespace, much like a file system. Its core components include:

• Persistent znodes: Default znodes that persist until explicitly deleted.
• Ephemeral znodes: Temporary znodes that exist only while the session is active.
• Sequential znodes: znodes used to generate unique sequential IDs.

ZooKeeper operates as an ensemble of servers, maintaining a consistent in-memory image of state through the ZooKeeper Atomic Broadcast (ZAB) protocol. This protocol ensures reliable delivery and ordering of messages, supporting Kafka’s metadata management.

ZooKeeper’s Role in Kafka

ZooKeeper has been integral to Kafka’s architecture, managing:

1. Controller Election: ZooKeeper elects the Kafka controller by facilitating the creation of a temporary znode.
2. Cluster Membership: Tracks broker availability via ephemeral znodes.
3. Topic Configuration: Stores metadata, such as topic configurations, partitions, and replicas.
4. Access Control Lists (ACLs): Maintains permissions for Kafka topics.
5. Quotas: Manages resource allocation limits.

Challenges with ZooKeeper

While reliable, ZooKeeper presents several challenges:

• Complex Deployment: Requires a separate distributed system, increasing operational overhead.
• Scalability Limits: ZooKeeper struggles with high client loads, creating bottlenecks in large Kafka clusters.
• Consistency Dependencies: Strong consistency guarantees can lead to availability issues during network partitions or failures.
• Slow Failovers: Notifications about state changes may not propagate quickly, causing delays.

Kafka Raft (KRaft): The New Metadata Mode

What is KRaft?

KRaft, short for Kafka Raft Metadata mode, is Kafka’s new consensus protocol based on the Raft algorithm. Introduced as part of Kafka Improvement Proposal (KIP-500), KRaft eliminates the dependency on ZooKeeper, integrating metadata management directly into Kafka.

Key Features of KRaft

1. Event-Sourced Architecture: Uses Kafka’s internal __cluster_metadata topic to store metadata events.
2. Simplified Architecture: Reduces the need for a separate coordination system.
3. Leader-Follower Model: The quorum controller replicates metadata changes across brokers, ensuring consistency.
4. Improved Scaling: Supports larger partition counts and reduces metadata bottlenecks.
5. Enhanced Security: Simplifies Kafka’s security model by unifying it under a single system.

Advantages of KRaft

• Simplified Deployment: Removes ZooKeeper, reducing operational complexity.
• Higher Scalability: Supports more partitions and brokers with reduced overhead.
• Faster Failovers: Event-driven synchronization minimizes downtime.
• Streamlined Security: Eliminates mismatched security models between Kafka and ZooKeeper.

Comparing ZooKeeper and KRaft

Metadata Storage

• ZooKeeper: Stores metadata in a hierarchical structure, managed externally.
• KRaft: Uses Kafka topics (__cluster_metadata) for event-driven state transitions.

State Synchronization

• ZooKeeper: Relies on ZAB protocol for strong consistency.
• KRaft: Implements a Raft-based approach with event sourcing for eventual consistency.

Deployment

• ZooKeeper: Requires a separate ensemble and multiple daemons.
• KRaft: Integrates directly into Kafka, supporting single-node deployments.

Failovers

• ZooKeeper: Slower due to delayed message propagation.
• KRaft: Faster due to event replay and quorum-based synchronization.

Transitioning to KRaft

Migrating from ZooKeeper to KRaft involves:

1. Upgrading Kafka: Use Kafka 2.8 or later, where KRaft is available as an early-access feature.
2. Metadata Conversion: Transform ZooKeeper metadata into KRaft-compatible format.
3. Configuration Update: Enable KRaft mode by updating broker settings.
4. Testing and Validation: Ensure compatibility with existing Kafka clients and applications.

As of Kafka 3.5, ZooKeeper mode is deprecated and will be completely removed in Kafka 4.0.

Trade-offs of KRaft

While KRaft simplifies Kafka’s architecture, it comes with trade-offs:

• Compatibility: Requires a full upgrade; not backward-compatible with ZooKeeper-based deployments.
• Eventual Consistency: May introduce temporary inconsistencies during failovers.

Key Differences Between ZooKeeper and KRaft

Conclusion

The transition from ZooKeeper to KRaft marks a significant evolution in Kafka’s architecture. By removing the dependency on ZooKeeper, KRaft simplifies deployments, improves scalability, and enhances performance. Organizations using Kafka should evaluate their requirements and consider migrating to KRaft, especially as ZooKeeper’s deprecation approaches.

KRaft represents Kafka’s future, promising a more robust and streamlined experience for managing distributed data streams.

https://aws.amazon.com/blogs/big-data/introducing-support-for-apache-kafka-on-raft-mode-kraft-with-amazon-msk-clusters/

https://developer.confluent.io/learn/kraft