Building a Simple Containerized IoT Platform with Docker, RabbitMQ, MQTT, WebSockets and ESP32.

January 20, 2024?| youtube.com/@codewithfrancis

A simplified Introduction to "Building a Dockerized IoT Platform":

This article is a simple introduction to building an IoT platform that can connect multiple devices (esp32 devices) via MQTT to a hosted IoT Platform. This can serve as guide to software engineers and hobbyists who would like to understand some of the intricacies of IoT and smart device connectivity. This article covers some basics in IoT, Docker and RabbitMq. This is meant to be a light reading for the Code With Francis Youtube Channel.

Project/Coding Resources:

• Full Video Tutorial: https://youtu.be/XiYJEA4LdIA
• ESP-32 Circuit & Code (micro-python): https://wokwi.com/projects/386829605804594177
• Platform API (Python): api-code-link
• Event-Hub (Node.js): event-hub-code-link
• Platform Dashboard (Node.js, Vue.js): dashboard-code-link

Introduction

With the continuous adoption of the Internet of Things (IoT) across multiple sectors, and its integration into our everyday life, the increasing demand for wireless control and valuable insights that can unlock new possibilities and innovations from data gathered by everyday devices in smart agriculture, cities, healthcare, manufacturing and homes has ignited pressing need to build robust and scalable IoT platforms for multiple (or combined) sectors of the economy/businesses. These platforms serve as the essential bridge between its users, data-generating devices and analytical tools. Efficiently managing data flow, security, control, and seamless communication.

From optimizing resource allocation in connected cities to predicting maintenance needs in smart factories, the diverse range of IoT applications demands an equally diverse range of platforms. Building these platforms requires careful consideration of real-time data processing, edge computing capabilities, device interoperability, and robust security measures. Investing in reliable and adaptable IoT platforms is not just a technological endeavor, but a strategic one. It paves the way for a future where data-driven intelligence reshapes industries, improves our lives, and builds a more connected and sustainable world.

This article elucidates how such an IoT platform can be built using a combination of multiple technologies such as Docker, RabbitMQ, Websockets, and the ESP32 microcontroller.

Why Containerize?

Containerization is a lightweight and portable solution for packaging, distributing, and running applications. With containerization, applications can be encapsulated along with all its necessary dependencies, libraries, and runtime environment into a single unit known as a container. Containers are usually isolated from the host system and other containers, ensuring consistency and reproducibility across different environments.

Key components of containerization include:

• Portable and Scalable: Containerized applications run across different environments, enabling easy deployment and scaling on any platform.
• Resource Efficiency: Containers share the OS kernel, making them lightweight and efficient, ideal for resource-constrained environments.
• Isolation and Reproducibility: Each container runs in its own isolated environment, enhancing security and ensuring consistent application behavior.

Simple IoT Architecture Diagram:

Architecture and Solution Components:

• Microcontroller (ESP32): The microcontroller (ESP32) will connects to actuators and sensors. It collects and transmits sensor data to the platform. It is also responsible for collecting actionable command data and triggering its actuators based on the command received.
• RabbitMQ: This is a message broker acts as a central hub for routing data between the ESP32 and the IoT Platform. For the implementation of this solution, the RabbitMQ MQTT plugin is enabled and is used by the ESP32 to to route messages to the RabbitMQ.
• Event Hub (Consumers): The event-hub is a data listening, ingestion and routing service. In this system, multiple rabbitMQ consumers are setup to retrieve and process messages from the devices telemetry queue, and then route these messages for storage to a database, or to a WebSocket service.
• WebSockets: This serves as a real-time communication service. Which allows the our web dashboard application to receive and display realtime data and updates instantly.
• Database (MongoDB): In the architecture above, we propose to store data in a document-oriented NoSQL database. This is due to the flexible nature of data storage which are similar to JSON objects with key-value pairs. In addition, the use of MongoDB provides a time series collection storage option, which is designed and optimized for the storage of time series data.

Platform API and Dashboard Setup:

Choosing the right cloud hosting provider can be overwhelming with so many options available. Some factors that may affect your decision, may include budget, application resource needs, traffic, scalability, budget, location, security and support. Some top hosting providers include Amazon Web Services (AWS), Digital Ocean, Google Cloud Platform (GCP), Microsoft Azure, Linode, Vultr, etc.

For the purpose of this article, we will use Digital Ocean as our cloud hosting provider.

• Register for an account, and create a server/droplet.
• For simplicity, you can create a server/droplet with Docker preinstalled.

If you prefer video instructions for setting up and hosting the Platform API and Dashboard App, you can use the links below:

• Digital Ocean Server Setup: https://youtu.be/XiYJEA4LdIA
• Linode Server Setup: https://youtu.be/IvFpORY-xco

(1) Create a Server and Install Docker: You will need a to create a server/droplet and then install Docker. It would be much easier to create a server/droplet with Docker preinstalled. Digital Ocean and Linode provide a market place where you obtain linux installation images with preinstalled docker. See the links above for a video tutorial.

If you already have a server and would prefer installing Docker on your own, here are some available options:

• Docker Engine Install: https://docs.docker.com/engine/install/
• Docker Desktop: https://www.docker.com/products/docker-desktop/
• Docker Install using Snap: https://snapcraft.io/docker

(2) Clone and Install the Platform Code: SSH into your server and clone (or copy) the code below into your server/droplet:

• Platform API (Python): api-code-link
• Event-Hub (Node.js): event-hub-code-link
• Platform Dashboard (Node.js, Vue.js): dashboard-code-link

(3) Build and Run the Docker Image: Go (cd) into the directory for each cloned folder and build the docker image.

To build the docker image run

docker-compose --project-name=iot_masterclass -f docker-compose.yml build        

To deploy the docker image run:

docker-compose --project-name=iot_masterclass -f docker-compose.yml up -d        

After running all three docker-compose scripts, all the docker images should have been uploaded.

• Using your browser, go to the IP address of your server/droplet. You should now have access to the IoT Connector Dashboard
• The default admin access for the IoT Connector Dashboard is provided below. Remember to change your password after first login.

default_email = [email protected]
default_password = admin        

• Access to rabbit MQ management should be available on port 15672:

• As seen below, the docker-compose.yml enables the rabbitmq_mqtt plugin: This ensures IoT devices can connect to the RabbitMQ service over MQTT.

# docker-compose.yml
version: '3.8'

services:

  mongo: 
    # Set the container name
    container_name: mongo_db_container

    # use mongodb version 7.0
    image: mongo:7.0

    # set environment variable
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${DB_USER_NAME}
      MONGO_INITDB_ROOT_PASSWORD: ${DB_PASSWORD}

    # Always restart the container if it fails
    restart: always

    networks:
      - custom_connector_network   # use created network bridge

    ports:
      - 127.0.0.1:27017:27017   # open mongo db port to localhost
    # persist data to a volume store to avoid loosing data on restart.
    volumes:
      - mongodb:/data/db

  rabbitmq:
    image: rabbitmq:management-alpine
    # Set the container name
    container_name: rabbitmq_container
    restart: unless-stopped

    # set environment variable
    environment:
      RABBITMQ_DEFAULT_USER: ${RABBIT_MQ_USER_NAME}
      RABBITMQ_DEFAULT_PASS: ${RABBIT_MQ_PASSWORD}
      RABBITMQ_CONFIG_FILE: /etc/rabbitmq/rabbitmq.conf
    
    networks:
      - custom_connector_network   # use created network bridge

    ports:
      - "5672:5672"   # open AMQP port: amqp port is defined in code
      - "15672:15672" # open management port
      - "1883:1883"
      - "15670:15670"

    command: "/bin/bash -c \"rabbitmq-plugins enable rabbitmq_mqtt; rabbitmq-server\""
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq   # Using local-persist driver for RabbitMQ data
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf



  # --------------------------
  # ALL API SERVICES
  all_api_service:
    # Set the container name
    container_name: all_api_container

    # buid the docker file in folder
    build:
      context: .
      dockerfile: main_all.Dockerfile

    # Always restart the container if it fails
    restart: always

    # Define environment variables
    environment:
      PORT: ${PORT}
      # rabbitmq access
      RABBIT_MQ_USER_NAME: ${RABBIT_MQ_USER_NAME}
      RABBIT_MQ_PASSWORD: ${RABBIT_MQ_PASSWORD}
      RABBIT_MQ_HOST: rabbitmq
      RABBIT_MQ_PORT: 15672
      # default admin user
      ADMIN_USER_EMAIL: ${ADMIN_USER_EMAIL}
      ADMIN_USER_PASSWORD: ${ADMIN_USER_PASSWORD}

      # API Keys
      API_KEY: ${API_KEY}
      WS_API_KEY: ${WS_API_KEY}

      # Database Ports
      DB_HOST: "mongodb://${DB_USER_NAME}:${DB_PASSWORD}@mongo_db_container"
      DB_PORT: 27017
      DB_Name: ${DB_Name}
    
    networks:
      - custom_connector_network   # use created network bridge
    ports:
      - "${DEFAULT_PORT}:7000"
    depends_on:
      - mongo
      - rabbitmq





# create volume for persistant storage
volumes:
  # mongodb volume
  mongodb:
    driver: local
  # rabbitmq volume
  rabbitmq_data:
    driver: local


# create network bridge access to resources
networks:
  custom_connector_network:
    driver: bridge
    # name: connector_network        

For a detailed description of the setup process, please watch the YouTube Video: https://youtu.be/XiYJEA4LdIA

IoT Device Setup:

(1) Create a Group ID: login into the IoT Connector Dashboard with "email = [email protected]" and "password = admin" or with your admin password if already changed. Create a new Group and copy the Group ID.

(2) Circuit Setup: Download the code and circuit schematics from https://wokwi.com/projects/386829605804594177.

edit env_settings.py

• Enter your server/droplet IP adress
• Enter the created Group ID
• If you are using a real ESP 32, enter your WiFi SSID and Password. Do not edit this if you are using the WOKWi simulator.
• If you are using a real ESP 32, load the code/files below into your ESP32.

ESP32 Files Required
--------------------
- main.py
- led_pwm.py
- helpers.py
- env_settings.py        

(3) Run the Circuit: Start and run the Code

If configured correctly, the ESP32 should provision the connected devices and start sending telemetry data to the RabbitMQ service via the MQTT plugin.

The Provisioned devices should be visible in the Group Details View

(4) View Telemetry Data: In the Group Details View, click on the "TELEMETRY" button to view Telemetry data.

• To View Telemetry Data, use the "Select Time Range" and "Profile States" dropdown menu to view telemetry data for specific device profiles. Telemetry Data should be available in the Time-Series Chart and Data Table.
• Control Commands Can be sent using the "Commands Panel". Control commands can be used to change device State values.
• To view Real-Time data, Connect to the WebSocket service. This ensures that the dashboard is subscribed to receive Real-Time data from the WebSocket service. When data is received, the Dashboard UI is updated.

Conclusion

Building a simple containerized IoT platform with Docker, RabbitMQ, Websockets, and ESP32 offers a powerful and flexible solution for data acquisition, routing, control, and visualization. The approach discussed is lightweight, scalable, and customizable, enabling you to leverage the power of the IoT within your own unique projects.

This basic platform, only shows a minute fraction of what can be achieved with the power of ioT and how you can gain valuable insights from the ever-growing world of connected devices. So, feel free to unleash your creativity into the into the fun world of the Internet of Things.