Kakfa Large messages

#apachekafka

Did you know that .... ?

Kafka configuration limits the size of messages that it's allowed to send.?By default, this limit is 1MB.

At a minimum, we need to configure these two properties :

1. Kafka producer Configuration: Property “max.request.size” needs to be updated to the expected size in Bytes.
2. Kafka Topic Configuration : Property?“max.message.bytes” needs to be updated to the expected size in Bytes.This holds the value of Kafka's largest record batch size?after compression (if compression is enabled).

More Optional Configurations:

3. Kafka broker Configuration

Property, “message.max.bytes“, can be used to allow ALL topics on a?broker. add this property in Kafka Broker's server.properties?config file.

Note : he maximum value among “message.max.bytes”?and “max.message.bytes” will be the effective value used.

For consuming large messages, to avoid performance impact on the consumer application :

4. Consumer Configuration

1. .max.partition.fetch.bytes:This property limits the number of bytes a consumer can fetch from a Topic's partition. It represents message fetch size from a single partition.
2. fetch.max.bytes: This property limits the number of bytes a consumer can fetch from the Kafka server itself. for?consumer listening on multiple partitions,?it?represents the message size that can be fetched from multiple partitions?

Note : the value of?"fetch.max.bytes"?should be higher than?"max.partition.fetch.bytes".

Generally avoid sending large messages using Kafka. The processing of large messages consumes more CPU and memory of our producer and consumer OR :

1. Compress messages
2. Store the large messages in a file and send location on kafka message
3. Split the large message into small messages of size 1KB then send all these messages to a single partition using the partition key to ensure the correct order.

Refrences :

[1] https://www.baeldung.com/java-kafka-send-large-message

[2] https://kafka.apache.org/documentation/#consumerconfigs_max.partition.fetch.bytes