Exploring Kafka and MQTT Integration for IoT
Kafka and MQTT are two complementary technologies. Together, they allow us to build IoT end-to-end integration from the edge to the data-center both in private and public clouds.
Motivation for Using Apache Kafka and MQTT Together
MQTT is a widely used ISO standard (ISO/IEC PRF 20922) publish-subscribe-based messaging protocol. MQTT has many implementations, such as Mosquitto or HiveMQ. For a good overview, see Wikipedia's comparison of MQTT brokers. MQTT is mainly used for Internet of Things scenarios (like connected cars or smart home devices). But, it is also used more and more in mobile devices due to its support for WebSockets.
However, MQTT is not built for high scalability, longer storage, or easy integration to legacy systems. Apache Kafka is a highly scalable distributed streaming platform. Kafka ingests, stores, processes, and forwards high volumes of data from thousands of IoT devices.
Therefore, MQTT and Apache Kafka are a perfect combination of end-to-end IoT integration from the edge to the data-center (and back, of course, i.e. bi-directional).
Let's take a look at two different alternatives for building this integration. Both solutions allow highly scalable and mission-critical integration of IoT scenarios. However, they use different concepts, including trade-offs.
MQTT Broker, Apache Kafka, and Kafka Connect MQTT Connector
This project focuses on the integration of MQTT sensor data into Kafka via MQTT Broker and Kafka Connect for further processing:
In this approach, you pull the data from the MQTT Broker via Kafka Connect to the Kafka Broker. You can leverage any features of Kafka Connect, such as built-in fault tolerance, load balancing, Converters, and Simple Message Transformations (SMT) for routing/filtering/etc., scaling different connectors in one Connect worker instance and other Kafka Connect related benefits.
Here is the GitHub project, including demo script, to try it out by yourself: Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data.
MQTT Proxy and Apache Kafka (no MQTT Broker)
As an alternative to using Kafka Connect, you can leverage Confluent MQTT Proxy. You will need to integrate IoT data from IoT devices directly without the need for an MQTT Broker:
In this approach, you push the data directly to the Kafka broker via the Confluent MQTT Proxy. You can scale MQTT Proxy easily with a Load Balancer (similar to Confluent REST Proxy). The huge advantage is that you do not need an additional MQTT Broker in the middle. This reduces efforts and costs significantly. It is a better approach for pushing data into Kafka.
Here is the Github project, including demo script, to try it out by yourself:
Both approaches have their trade-offs. The good thing is that you have the choice. Choose the right tool and architecture for your use case!
Conclusion: MQTT and Apache Kafka Are Complementary and Integrate Well
Both, MQTT and Apache Kafka have great benefits for their own use cases. But, none of them are the single allrounder for everything. The combination of both makes them very powerful and a great solution to build IoT end-to-end scenarios from the edge to data-center and back.