Containerizing Real-Time IoT Machine Learning Solutions with Docker, TML, Kafka, TMUX, and Python

The world of the Internet of Things (IoT) is growing rapidly especially as more devices and objects like bulbs, vacuum cleaners, cars, clothes, heavy equipment, routers, tires, engines, phones, pumps, HVAC, and many more devices get sensored up and produce real-time data. This real-time data, or data streams, contains valuable information on device characteristics, device health, telemetry, network information, security information, and other information.

Real-time IoT solutions are built to process this information to help humans or other machines make decisions from the outcomes of this processing and machine learning. The decisions made can be very useful to identify devices that are not functioning properly, or device that are about to fail. Using Transactional Machine Learning (TML) with Apache Kafka enables us to process large amounts of real-time data to provide predictive information for every device in real-time. Using TML and Kafka, each device can be individually analysed with its own Machine Learning model and with advanced processing algorithms; this means, even if there are 100 million IoT devices producing data, TML and Kafka can build 100 million machine learning model for each device in-memory.

Using TML and Kafka, each device can be individually analysed with its own Machine Learning model and with advanced processing algorithms. this means, even if there are 100 million IoT devices producing data, TML can build 100 million machine learning models for each device in-memory (no database required, only Kafka).

The ability to process large amounts of data from IoT devices, with advanced, in-memory, processing, we can not only build large scale IoT solutions, but also deploy them very quickly with Docker and Kubernetes in any operating system, for any Chip Architecture with Cloud or without Cloud.

So, let start.

An IoT Docker container has been built that provides the ENTIRE IoT solution. This container analyses IoT data streams using Apache Kafka, TML technologies, and provides a streaming visualization dashboard: https://hub.docker.com/r/maadsdocker/seneca-iot-tml-kafka-amd64

This container has the following technologies installed:

1. Python 3.9
2. Apache Kafka 2.13
3. Linux Ubuntu
4. Mariadb (MySQL)
5. Transactional Machine Learning (TML) Technologies: Viper, Viperviz, HPDE
6. Python scripts: This contains the scripts needed to run the IoT failure prediction solution
7. TMUX (terminal multiplexer): https://github.com/tmux/tmux/wiki - this multiplexer allows us to run ALL scripts and binaries needed for the IoT solution
8. IoT Data (we are simulating IoT data)

Anyone who has Docker engine installed can run the container very easily. Do the following:

1. docker pull maadsdocker/seneca-iot-tml-kafka-amd64
2. docker run -p 9005:9005 maadsdocker/seneca-iot-tml-kafka-amd64
3. ENTER this URL in their browser: https://localhost:9005/iot-failure-seneca.html?topic=iot-preprocess2,iot-preprocess&offset=-1&groupid=&rollbackoffset=500&topictype=prediction&append=0&secure=1
4. That’s it!?

If you are running Docker daemon and run into issues with Docker try the follow:

1. Name server issue: echo "nameserver 8.8.8.8">/etc/resolv.conf
2. If overlay2 issue try: rm -rf ~/Library/Containers/com.docker.docker
3. If overlay2 issue try: sudo rm -rf /var/lib/docker

Note the URL in 3 - we are exposing port 9005 in Docker run - which means we are mapping the host port 9005 to port 9005 in the container. Just paste this URL in your browser, and in a few moments after the container is running you will see a streaming dashboard:

These dashboards are streaming data from the running container. So, what have we accomplished? Here it is:

1. Well we installed an END-END solution in a Docker container.
2. The container is analysing IoT data in real-time.
3. The container is FREE - because it has all free or open source technologies installed. No cost is being incurred by the container.
4. The visualization is streamed to a browser - so no cost is being incurred to third-party visualization licenses.
5. Everything is self-contained and can be executed in two-steps: Docker Pull, Docker run.

Having the ability to run large IoT solutions using Containers, Kafka, TML and TMUX opens up tremendous opportunities to build similar solutions with relative ease and quickly deploy them in almost any environment with or without cloud.

Till next time...