All About Message Queues

Message queues are one of the most important components in software architecture.

A message queue is basically a queuing service used in micro services?to help different components of the system communicate asynchronously by passing messages between them.

Queues are absolutely great in scenarios where your application needs something done but doesn’t need the result right now as it's not a blocking call(sync call) and your system can wait for the result while parallelly processing other tasks. So,?Instead of calling a service using an API and waiting for a response, you can push the message to a queue and let the same business logic happen later.

They are a great way to make different components of the system self contained and more independent i.e decouple them.

?There are 3 components of a of a message queue - Producers, Messages and Consumers.

Producers push the messages into the queues and Consumers/Worker processes?read those messages and take actions.

A message is any information that is passed between the producer and the consumer. It contains all the information needed for the worker components to take actions. The format of the message in the queue becomes your data contract and any service that knows how to read that message format can be used to process the message.

Eg, if we have to update the user address, The worker component needs to know details like: user_id, house_no, city, country

The message the producer creates and puts into a message queues will look something like this:

{

"user_id": 12345,

"house_no": 567,

"city": "Tokyo",

"country": "Japan"

}

?Queues are a great means to increase fault tolerance and scalability in the system. Even if the producer component fails, the consumer would still keep pulling messages from the queue and continue the tasks. Similarly if the consumer or worker stops working, the producer would still keep pushing messages into the queue. Hence it helps in decoupling of both these components as they are not dependent on each other thus making the system fault tolerant.

Now since the components are independent of one another they can be scaled independently too. If the worker processes are doing all the heavy-weight work then powerful machines can be used to run Worker service. Similarly more tasks can be added if the number of users increase. With this architecture it becomes possible to scale the different components independently.