MQTT Rate Limiting and throttling...

In a recent deployment, an MQTT broker was installed on the same server hosting critical applications. Unfortunately, rate limiting and throttling measures were not implemented. This oversight led to a significant incident, resulting in server overload and service disruptions.

As devices and sensors began connecting to the MQTT broker, they established an excessive number of connections without restraint. The server, already running multiple applications, struggled to cope with the sudden surge in traffic.

CPU utilisation spiked, memory resources became strained, and latency increased across the board and I kept wondering what could be the issue having applied rate limiting before designing REST APIs I kept wondering if it was also possible to do the same with MQTT connections… I wondered so much???????

The lack of rate limiting exacerbated the situation, leading to degraded performance and intermittent failures in our applications. Critical services experienced slowdowns, and in some cases, became unresponsive. I started an investigation to identify the root cause and mitigate the impact on operations.

Upon analysis, it became evident that the absence of rate limiting and throttling mechanisms allowed devices to overwhelm the MQTT broker with an unmanageable volume of connections and message traffic. This resulted in resource exhaustion and degraded performance on the server hosting both the broker and critical applications.

This incident underscores the criticality of implementing proper controls in MQTT deployments, such as rate limiting and throttling. Failure to do so can lead to severe consequences, including server overload and service disruptions. As software architects, we must prioritize implementing robust measures to safeguard the stability and reliability of our systems.

Lucky enough I found ways to perform rate-limiting clients subscribing to and publishing to an MQTT broker just the way you usually do with REST APIs. So I am leaving the following strategies that helped us handle the situation.

It is great to do these before your application comes to a halt or worse somebody uses the unlimited connection loophole to overload your server through? DDOS. Attackers could send large byte messages or just publish more messages than a broker can handle... but also remember ??????

Let's take a deep dive. These strategies are relevant to almost all MQTT brokers:

1/5: One approach to MQTT rate limiting is by configuring your MQTT broker. Many brokers offer built-in features to limit message rates per client, maximum message sizes, and connections. Check your broker's docs for details…Mosquitto, EMQX, and HiveMQ have good docs on the same

2/5: Quality of Service (QoS) levels in MQTT can indirectly control message rates. Higher QoS levels like 1 and 2 involve more overhead but provide stronger message delivery guarantees. Consider adjusting QoS to manage message rates.

3/5: Implementing client-side rate limiting gives you fine-grained control. Use timers, counters, or other logic in your MQTT client to throttle message sending based on predefined limits. Keep an eye on resource usage!

4/5: Topic-based rate limiting is another option. You can allow higher rates for certain topics while limiting others. Implement custom logic in your MQTT client to monitor message rates per topic and enforce limits accordingly.

5/5: Remember, whether you're using built-in broker features, client-side logic, or topic-based strategies, designing an effective MQTT rate-limiting mechanism is crucial for maintaining system performance and reliability.?

Happy messaging! I would love to hear your strategies too what worked for you...