Kafka Producer: Secrets to Efficient Real-Time Data Processing

In modern architecture, there are many reasons why an application might need to write messages to Kafka: recording user activities for auditing or analysis, recording information from smart devices, asynchronous communication with other applications, and many more.

Each architecture requires diverse and different needs, such as: the importance of the message, or can we tolerate the loss of some messages? Can we accept accidentally duplicating messages? Are there strict latency or throughput requirements?

In financial transactions involving money or inventory management in e-commerce, there will be a requirement never to lose a single message and not to process duplicate messages. The requirement for low latency, allowing latency <= 500 ms, and the need for high throughput are also crucial. On the other hand, tasks like sending customer notifications can tolerate the loss of some messages. To meet the requirements, we first need to understand the architecture of Kafka. As I described in my previous post about the high-level design of Kafka, you can refer back to it for a comprehensive overview. Once we understand the architecture of Kafka, we can break down its components for detailed analysis and control. We can confidently tweak the parameters and make accurate assessments when applying Kafka to our architecture. In today's post, I will share about the “Producer,” a very important component that is often overlooked and under-analyzed. Regarding the “Producer,” I have two main topics:

• High-Level Design
• Configuration

These two topics will be covered in separate posts; today’s post will focus on the “High-Level Design” of the producer.

Let's dive straight into the producer's processing flow. You can follow the flow in the attached diagram. To start producing messages to Kafka, the first step is to create a ProducerRecord object. The required attributes are the topic and value information. Key and partition information are optional and may or may not be included. The producer then sends the ProducerRecord. In this step, the producer performs the following tasks:

• Serialize the key and value into byte[] so they can be sent over the network.
• Next, the data will be sent to the partitioner. If the partition information is already present in the ProducerRecord, the partitioner does nothing and returns the pre-configured partition information. If we have not specified the partition in the ProducerRecord, the partitioner processes it to select a partition based on the key information. Once the partition is selected, the producer determines which topic and specific partition the record will go to. Then, the producer adds the record to the batches being sent to the same topic and partition. A separate thread is responsible for sending the batch records to the corresponding Kafka broker.
• When the Kafka broker receives the message, it sends back a response:If the message is successfully written to Kafka, it returns RecordMetadata, including the topic, partition, and offset of the record in the partition.If the broker fails to write the message, it returns an error. The producer receives the error information, retries a few times before giving up, and throws an error.

By reading this, you should understand the producer's processing flow and identify potential bottlenecks. When high latency and throughput are required, the first thing to consider is the producer. Any issues at the source will impact subsequent flows. In the next post, I will share how to fine-tune and explain the configuration parameters of the producer to optimize for various requirements.