Exploring Kafka's Architecture: The Foundation for Real-Time Data Processing

In today's technology landscape, real-time data processing is a crucial requirement. Apache Kafka, a distributed data processing platform, has become a key tool for transmitting and processing large data streams. With its scalability, reliability, and high performance, Kafka helps businesses efficiently collect, manage, and analyze data.

Case Studies Using Kafka:

In the Financial Sector:

• Conditional Orders (Stream of market price signals)
• Notification Systems (Creating multiple stream channels for millions of notifications, emails, SMS)
• Synchronization between cache data sources and persistent data sources
• Asset Information Synchronization
• Corporate Bond Robo Trader
• Price Board
• And more...

In the E-commerce Sector:

• Inventory Management
• ETL for Data Warehouses
• And more...

Through various system architectures, from simple to complex, including high-load systems requiring high consistency, Kafka has proven to be highly reliable and an indispensable component in my architectural designs and deployments.

Throughout the implementation process, there are many interesting aspects that I will gradually share in upcoming articles, from usage tips to important configurations for Kafka clients and servers tailored to different use cases.

In this article, I will summarize Kafka's architecture to help you understand its key components:

Kafka Clients:

• Producer: Pushes messages to specific topics.
• Consumer Group: Subscribes to topics and consumes messages.

Kafka Servers:

Core Service and Storage:

• Broker: Holds multiple partitions. A partition holds a subset of messages for a topic.
• Storage:

- Data Storage: Messages are persisted in data storage in partitions.

- State Storage: Consumer states are managed by state storage.

- Metadata Storage: Configuration and properties of topics are persisted in metadata storage.

Coordination Service:

• Service Discovery: Identifies which brokers are alive.
• Leader Election: One of the brokers is selected as the active controller. There is only one active controller in the cluster. The active controller is responsible for assigning partitions.
• Apache Zookeeper or etcd are commonly used to elect a controller.