IOT+Kibana Visualization.

Have you ever thought of getting the daily temperature of your room and compare it with the official city temperature? This is a fun project you can do to compare the temperature and humidity while graphing it out in Kibana. This project requires the ELK stack to be setup.

IOT Arduino Components.

To get the room temperature, you will need the Arduino uno board, jumper cables and a DHT22 sensor.

1. Arduino uno board - Rs. 1250/-
2. Jumper Cables (male to female connector) - Rs.125/-
3. DHT22 temperature/humidity sensor - Rs.1300/-

First we need to get the Arduino connections to the DHT22 established with the jumper cables. The following diagram shows the jumper wire connections from the DHT22 to Arduino.

Once you have the jumper wire connections established, upload the following Arduino sketch to the arduino board using the arduino sketch editor. DHT22 is a very sensitive temperature & humidity sensor that works on Arduino & Raspberry Pi.

//Libraries
#include <DHT.h>

//Constants
#define DHTPIN 3    // what pin we're connected to
#define DHTTYPE DHT22   // DHT 22  (AM2302)
DHT dht(DHTPIN, DHTTYPE); //// Initialize DHT sensor for normal 16mhz Arduino

//Variables
int chk;
float hum;  //Stores humidity value
float temp; //Stores temperature value

void setup()
{
    Serial.begin(9600);
  dht.begin();
}

void loop()
{
    //Read data and store it to variables hum and temp
    hum = dht.readHumidity();
    temp= dht.readTemperature();
    //Print temp and humidity values to serial monitor
    Serial.print(temp);Serial.print(";");
    Serial.print(hum);
    //Serial.println();
    
    delay(30000); //Delay some seconds.
}

C# Serial Data Reader

The above sketch writes data to the Serial COM port, which I took advantage of. I used C# for my serial port data reader from COM port 5. You can get the port information from the arduino sketch editor. I also got the regional weather information from OpenWeatherMap.Org, which provides god weather information FREE of charge. Once you sign up with OpenWeatherMap, you will receive the api KEY through email, which must be added to the APPID parameter in the url. I used a C# Timer control(every 30s) to get periodic json data from OpenWeatherMap.

SerialPort port = null;
string path = "D:/Data/ELK/datafiles/dht22.csv";
public frmComCoder()
{
  InitializeComponent();
  port = new SerialPort("COM5", 9600, Parity.None, 8, StopBits.One);
  port.Open();
}

private void timer1_Tick(object sender, EventArgs e)
{
  string POT = port.ReadExisting(); //changing between 0-1023
  string strDate = DateTime.Now.ToString("dd'/'MM'/'yyyy HH:mm:ss");
  var json = "";
  var APIurl = "https://api.openweathermap.org/data/2.5";
  var theUrl = APIurl+"/weather?q=Colombo,lk&APPID=XXXXYYY";

  using (WebClient wc = new WebClient())
   {
     json = wc.DownloadString(theUrl);
   }

  JObject o = JObject.Parse(json);
  string temp = (string)o.SelectToken("main.temp");
  string humid = (string)o.SelectToken("main.humidity");
  float cTemp = float.Parse(temp);  // Convert Kelvin to Celsius
  string colTemp = (Math.Round((cTemp-273.15)*100f)/100f).ToString();
  using (StreamWriter writer = new StreamWriter(path, true))
  {
    writer.WriteLine(strDate + ";"+POT+";"+colTemp+";"+humid+";");
            
  }
    
}

The json returned from the OpenWeatherMap has the temperature in kelvin, which must be converted to Celsius before storing in ElasticSearch. The dht22.csv output would look like this.

01/01/2020 16:33:28;32.10;76.20;30;74;
01/01/2020 16:33:58;32.00;75.70;30;74;
01/01/2020 16:34:28;31.90;75.40;30;74;
01/01/2020 16:34:58;31.90;75.60;30;74;
01/01/2020 16:35:28;31.90;75.80;30;74;
01/01/2020 16:35:58;31.90;76.10;30;74;
01/01/2020 16:36:28;31.80;75.90;30;74;
01/01/2020 16:36:58;31.80;75.90;30;74;
01/01/2020 16:37:28;31.80;76.10;30;74;
01/01/2020 16:37:58;31.70;76.00;30;74;
01/01/2020 16:38:28;31.80;76.00;30;74;
01/01/2020 16:38:58;31.80;76.00;30;74;
01/01/2020 16:39:28;31.70;76.20;30;74;
01/01/2020 16:39:58;31.70;76.40;30;74;
01/01/2020 16:40:28;31.70;76.60;30;74;
01/01/2020 16:40:58;31.80;76.30;30;74;
01/01/2020 16:41:28;31.70;76.30;30;74;
01/01/2020 16:41:58;31.70;76.70;30;74;
01/01/2020 16:42:28;31.70;76.90;30;74;

Beats+LogStash Configurations

The dht22.csv data is read and transferred to LogStash via Filebeat. I then used a simple log transformation of the above lines through the LogStash filter{} sections to convert the above values to floats and strings to be stored in the ElasticSearch. As we need to match the logs via grok, we need to download the patterns files. The patterns can be downloaded from the https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns and stored in the config/patterns folder.

if ("DHT22" in [tags]){
   grok{
     patterns_dir => ["D:/Data/ELK/logstash-6.8.3/config/patterns"]
     match => {"message" => "%{DATESTAMP:date};%{DATA:temperature};%{DATA:humidity};%{DATA:KotteTemp};%{DATA:KotteHumid};"}
     overwrite => ["Message"]
   }
   date {
       match => [ "date", "dd/MM/YYYY HH:mm:ss" ]
       target => "@timestamp"
   }
   mutate{
       convert => {
          "temperature" => "float"
          "humidity" => "float"
          "KotteTemp" => "float"
          "KotteHumid" => "float"
       }
   }
  
}

The "DHT22" tags: name was set in the FileBeat -> filebeat.yml file, as I had other log files that needed reading and I didn't want to mess up my existing configurations.

The following temperature graph shows the Kotte temperature along with my room temperature. I have no doubt that my room is experiencing an above average temperature than the outside environment. Unfortunately, when the walls/windows heat up, the room stays warm all night long.

The humidity graph on the other hand shows a good co-relationship between the room and outside environments. The increased humidity in the room was caused by the Honeywell air cooler which I turned on in the afternoon. The avg humidity in Kotte was recorded at 77%

The Kotte temperature graph was plotted from the json object I got from the OpenWeatherMaps.org website. Using the Kibana dashboards, we can add all the related graphs into a single dashboard for easy reference. If you like to learn more about data-visualization or monitoring, feel free to contact me.