Rendezvous with RabbitMQ : Our unanticipated data pipeline

Following up on my previous article on

This article shall attempt to decipher our journey with ?? MQ.

Our services are fond of chattering! ?? Without sharing events, they cannot exist. ??

However, it is always challenging to reliably route messages in unreliable environments.

Having formerly used Apache Kafka in projects, we thought it was time to try RabbitMQ under certain design rationales.

Why RabbitMQ?!

? Everyone recognizes the true power and tenacity of Apache Kafka as a universal message broker. We have been using Kafka to drive our internal data pipelines.???

??However, the project was not exorbitantly demanding.Our team did not wish to over-privilege entities that can not/do not use resources.

??Our services were not extremely loquacious.They shared events in a moderate volume.

??Design Aspects??

??The producers pump events into our data pipeline.

??Our consumers feed on them.They are the ones who process them and maintain the chain of business operations.

??Our service topology is quite complex. Sadly,at times you just can not circumvent some complications that are driven by business logic. The same could also have been achieved by Apache Kafka,but we felt RabbitMQ exchange was a great way to deal with our complex topology.

??Our team designed the reliable retry mechanism for transient failure through RabbitMQ’s Dead letter exchange and? TTL(Time To Live) parameters.?

??We heavily utilized the RabbitMQ Delay feature to model our future events.We could even cut down some of our batch jobs in the process. ??

??The order of events is very important for our business events.RabbitMQ's Consistent Hash Exchange plugin helped us to tackle the competing consumer problem.

??We had many consumers feeding on our? queues.Given that the processing time was quite fast,we designed our RabbitMQ Consumers to allow them to load a moderate amount of unacknowledged messages in consumer memory for optimal performance.

There are many teams like us who are confused to choose between RabbitMQ and Apache Kafka ,given that we had experience on both. Nevertheless, the rationales that drive the engineering design are always business specific.

Feel free to share your views on how your team maneuvered through tough engineering design aspects.