MESSAGE BROKERS – WHAT ARE THEY, WHAT ARE THEY COMPOSED OF, AND PROS AND CONS: COMPARING APACHE KAFKA, REDIS AND RABBITMQ

When working on a project, developers, analysts, and testers have to participate in the architecture design process. To understand how different parts of the system or services can interact with each other, you need to understand message brokers since they are often used to build communication between system components.

In this article we will understand what message brokers are, what they consist of and when they are used together with their pros and cons. We will also look at the features of the three most popular brokers – Apache Kafka, RabbitMQ and Redis.

A message broker is an architectural pattern used in distributed systems. It is a software component that serves as an intermediary in communication between different parts of the system. It is implemented as part of the overall system architecture, or as a separate service.

WHAT DOES A MESSAGE BROKER?CONSIST OF?

Any message broker uses two main entities:

• Producer / Publisher — deals with sending messages to the broker;
• Consumer / Subscriber — receives and processes messages from the broker;

The message is sent directly from the sender to the recipient.

There is also a Topic – a logical unit that unites all messages according to some criterion.

Publish/subscribe scheme.

?

There are two main algorithms for sending messages to a queue:

1. There is a Producer who sends messages to the broker, and there is a Consumer who receives them
2. There is a Publisher that puts messages into the queue, and there is an n-th number of Subscribers that process them further

WHEN TO USE? BROKERS

There are many cases when brokers are used. Let's look at the main ones:

• Delayed execution of actions. The need to use brokers arises when it is necessary to perform heavy operations, but they do not necessarily need to be performed at the time of the request from the user. For example, to generate a report, we should not force the user to wait until time passes for the result to appear. Instead, we send the task to the queue and give the client a response that the process has started, and the response is being generated.
• Implementation of asynchronous exchange between services that ensures message buffering. In this case, the broker will act as a kind of buffer for accumulating messages. We can regulate the load and, in some cases, speed up data processing compared to the case if we did it single-threaded.
• Implementation of complex business logic which involves several components of the information system. This mainly applies to microservices. Often you need to make changes that need to happen in the right sequence. The broker will facilitate this process. Having received messages in the first service, the necessary changes are made, and the next task is launched for the second service.
• Smoothing of peak loads. Sending a task to a broker is faster than completing it. Therefore, we can accumulate certain actions and do them later.?
• Sending notifications (often similar to the one mentioned above). For example, if there is a mass registration for an event, we want to send everyone a registration confirmation. In order not to overload the main part of the system with sending notifications, we simply put these tasks in the broker, and then work with them in the background.
• When you need to aggregate and execute tasks on a schedule or collect telemetry data, logs and statistics. Writing to a file or transferring to some other system to aggregate this information is a rather labor-intensive process. By using queues we can reduce the load on our main service. As this is not a primary task for business logic, we can perform it in the background, and this will not affect the system performance for the user in any way.?

PROS AND CONS?OF USING BROKERS

Like any other technology, message brokers have their pros and cons:?

PROS

CONS

Fault tolerance and scalability. Helps create fault-tolerant and scalable systems thanks to asynchronous data processing and the ability to use broker clusters.

Complication of the system. The use of brokers will entail an increase in the complexity of the system by adding a new element to it. Debugging, maintenance and administration become more difficult.

Separation of layers. Allows the separation of application layers reducing the coupling between them, which makes it possible to make the application more modular, flexible and easy to develop. It will also be useful when organizing a microservice system.

Difficult to learn. Learning this software can take a long period of time due to the complexity of the architecture.

Message delivery guarantees. Provide guarantees for message delivery. If the recipient is not ready to process the received information at the moment, it will be stored in a queue, and the recipient can process it later.

Additional point of failure. This is the point of possible critical failure where most systems will not be able to function. However, many brokers provide high availability and reliable operation. This is a potential performance bottleneck. Therefore, modern brokers are designed to be highly scalable.

Productivity increase. Using brokers can improve performance by processing messages asynchronously.

Additional load on the network. The use of brokers may result in additional network load due to data transfer.

Safety. Provide encryption and authentication mechanisms to ensure secure data exchange.

Possibility of version conflicts: If system participants use different versions of protocols or different implementations of message brokers, compatibility problems may arise. This may result in incorrect processing or loss of messages.

POPULAR MESSAGE BROKERS

In fact, there are a huge number of message brokers on the market now. For example, Apache Kafka, RabbitMQ, Redis, Apache ActiveMQ, Amazon SQS, Apollo, Google Pub/Sub and others. This list can go on for a long time, but we will focus on the first three since they are most often used in modern development.

We compared brokers based on a number of criteria. For comparison, we’ll highlight three main criteria that will influence the choice:

1. Scale (масштаб брокера) ?—? количество сообщений, отправляемых в системе, в секунду.
2. Data Persistency (постоянное хранение данных, персистентность) ?—? возможность восстановления сообщений.
3. Клиентские возможности ?—? способность управлять клиентами в режиме one-to-one (один к одному) и/или one-to-many (один ко многим).

?Broker

Scale

Data Persistency

Client capabilities

Apache Kafka

Very fast, allows you to process more than 1 million messages per second

Yes

One-to-many

RabbitMQ

About 50 thousand messages per second (depending on configuration)

Supports permanent and temporary messages

One-to-one and one-to-many

Redis

Up to 1 million messages per second

Partial implementation

One-to-one and one-to-many

Features of Apache Kafka

One of the most popular brokers which is quite actively used and has a lot of features?

• Subscribers must “pick up” messages themselves;
• Permanent data storage. All messages are stored for a limited amount of time which is configured at the broker level;
• Allows you to re-read messages;
• Guarantees the order of messages within a topic. The subscriber will receive the messages in the same order as the publisher sent them.

Features of Rabbit MQ

It is also one of the popular brokers. It is less productive compared to Kafka, but it has its advantages.?

• Flexible routing. Here we can build systems that require data transfer to many parts or microservices. Thanks to it, you can make this task easier by using all the possibilities;
• Deletes the message after it is delivered to the recipient.?

There are difficulties when scaling horizontally in a cluster.

Features of Redis

• Backup for a specific point in time;
• Has limited functionality compared to other brokers.

IN WHAT CASES SHOULD EACH BROKER BE USED?

The choice of a message broker depends on the specific requirements and context of use.

Apache Kafka

RabbitMQ

Redis

when you need to process a large amount of data that is generated very quickly

no big data flow

processing of large volumes of data is required

when implementing transactional or pipeline systems

flexibility of message routing within the system is important

no persistence required

when building an event-driven architecture

the fact of message delivery is important

high speed of message delivery is required

when using a buffer for logs and metrics

?

?

CONCLUSION

Brokers are now very actively used in the construction of large information systems. You shouldn’t make a definite conclusion about which broker is better; we recommend working with several and choosing yourself because it all depends on the context in which it will be used. The choice of a specific solution must be made based on the task that is assigned to you at the current moment of work.?