Kafka Idempotent Producer

When an application publishes events to a Kafka topic there is a risk that duplicate events can be written in failure scenarios, and consequently message ordering can be lost. This can be avoided by configuring the Kafka Producer to be idempotent.

Duplicate Messages

Duplicate messages can occur in the scenario where:

• A Producer attempts to write a message to a topic partition.
• The broker does not acknowledge the write due to some transient failure scenario.
• The Producer should retry as it does not know whether the write succeeded or not.
• If the Producer is not idempotent and the original write did succeed then the message would be duplicated.

By configuring the Producer to be idempotent, each Producer is assigned a unique Id (PID) and each message is given a monotonically increasing sequence number. The broker tracks the PID + sequence number combination for each partition, rejecting any duplicate write requests it receives.

The Kafka Producer configuration?enable.idempotence?determines whether the producer may write duplicates of a retried message to the topic partition when a retryable error is thrown. Examples of such transient errors include leader not being available and not enough replicas exceptions. It only applies if the retries configuration is greater than 0 (which it is by default).

To ensure the idempotent behaviour then the Producer configuration?acks?must also be set to?all. The leader will wait until at least the minimum required number of in-sync replica partitions have acknowledged receipt of the message before itself acknowledging the message. The minimum number is based on the configuration parameter?min.insync.replicas.