IBM Watson IoT, Python3 and JSON

To store, aggregate, and analyze building data is an enormous task and often this means the storage and compute functions are performed off the premises, or in other words the cloud. I wanted to learn more about the options for building automation cloud storage so I picked up a home automation project that has served me as a testing and learning platform. While the number of sensors I have in my house don’t compare to a commercial building, the principles are the same. I need to read the data, format it, send it to a storage device, aggregate and analyze it, then display it in a meaningful way. This article discusses what I learned about the IBM cloud platform and a few other interesting things along the way related to making improvements for better intelligence and responsiveness, improved error trapping, and a smarter strategy for managing building data.

My previous HMS version recorded data in locally stored SQL tables. While the SQL tables made it very easy to lookup, analyze, and graph the data, I never finished the code to offload the SQL tables, once storage got low. As a result, I overloaded the storage and corrupted it 'to boot' (pun intentional). So, HMS version 2.0 was dead, and even though I had backups, I decided to rewrite and create version 3.0.

For version 3.0 I planned to write sensor data directly to the cloud. Doing some research, I came across the mention of an IoT platform called Thingspeak. Searching for other IoT cloud providers I came across a list of many, including Microsoft Azure, Amazon Web Services, and IBM IoT Watson. I must admit, saying data from my home's sensors is stored and analyzed by IBM Watson sounds really cool - a testament to IBM platform thinking, so I decide to try IBM IoT Watson.

When working with Linux based operating systems you utilize the command line quite a bit. At first, I struggled with this, over time, however, I began to find the command structure intuitive and easier. In fact, when you search online for a resolution to a problem, the solutions often come in the form of a commend line statement, so it’s usually as easy as copying a command off a web page and pasting it into a terminal window, press enter.

Python uses libraries (modules), a fantastic feature for reducing the amount of code you need to write. Python libraries are just one more example of something you install/administer from the command line. There are two main tools for installing library packages: 'apt-get' which to me always seems bullet proof and 'PIP'. I've always had mixed results with PIP. While I have no actual data to back this up, I feel like apt-get works every time, whereas with PIP its maybe 4 out of 5 times. There's a whole host of reasons for this error that I won't go into here, but its very frustrating to get the 'Module Not Found' error message.

When I started experimenting with IBM IoT Watson, I was very excited to find a python module (library) that did most of the heavy lifting, only to be terribly frustrated when PIP install failed. After a few nights of troubleshooting, and with the help of the greatest online support community stackoverflow.com, I came to the conclusion that I could not use the IBM IoT python library with Python 2.7, it had to be Python 3.

Python 2.7 versus 3.x

The presence of Python 2.7 and Python 3 is a remarkable technological phenomenon unlike anything I've ever seen before. Python began in the late 1980's and matured to version 2.x until 2008 when Python 2.6 and Python 3.0 were released at the same time. Python 3 was developed to rectify some of the critical flaws in the language and as a result was not backward compatible with version 2.6. In 2009 Python 2.7 was released at the same time Python 3.1 was released, this was the final 'parallel' release and it was announced in that in 2020 support for Python 2.x would cease and users were encouraged to move to Python 3 as soon as possible. During this time period from 2009 till even now Python 2.7 is the more popular version, this is witness by popularity of 2.7 code sample on the internet.

My desire to experiment with IBM IoT Watson led me to switch to Python 3.x (current release now is 3.8). The code for my HMS software is currently only about 200 lines and it only took me about an hour the modify to be Python 3.8 compliant. There's even a python program for converting code from version 2 to version 3 and is appropriately named 2to3.

IBM Watson IoT

With the IBM Watson IoT python modules installed I was ready to dig in. It took a few hours to setup an account, get a little familiar with the platform, find my credentials and connect. I don't remember what I was expecting the IBM cloud to be, but I doubt it was what I discovered, and I don't mean that in a good or bad way.

First everything is broken up into modules (or resources). It’s not as simple as transmitting data to an address on the Internet. You provision an IoT hub or platform to connect, and in some cases, manage, your devices and gateways. You provision a storage container and route data from your IoT platform to your storage appliance. Analytics is another resource as is cloud applications you can write with a variety of languages. Some new things I tinkered with for the first time MQTT (IoT device transmission protocol) RESTful (API), IAM (authentication) and JSON table-less databases (more on that in a minute). In short, I was able to connect my computer (with the sensors attached to it) to the IBM IoT platform, but failed to connect the IoT platform to a storage device (IBM cloudant).

I was able to connect my computer (with the sensors attached to it) to the IBM cloudant storage device directly using the API with their IAM authentication and stored about 15,000 temperature readings over a 24-hour period. Finding my data was very easy, I struggled with analyzing and visualizing it. Like I mentioned everything in the IBM cloud offering is broken up into modules (resources), then you need to route from one module to another. IBM has many examples, they call them recipes, but many of them are old and as the web interfaces to the different modules change the recipes become harder or impossible to follow. Adding confusion there are so many modules (which can be a good thing), knowing what’s the best module to use is challenging. I equate it to the process of buying a house, sometimes you just need to talk to someone.

JSON and IBM Cloudant

For IBM's storage solution cloudant, it is a server-less, managed database as a service (DBaaS) for multi-cloud applications. Using simple HTTP GET commands it's easy to add documents. When I first read 'add documents' that somewhat threw me. My database experience is limited to relational databases like SQL, Microsoft Access, and Borland Paradox. Databases have tables with records that have fields, what are these documents you speak of? Instead of tables a 'document-oriented' or NoSQL database using small documents (ideally less than 1 megabyte each). Cloudant uses a document encoding scheme called JSON, Other encoding schemes include XML and YAML. IBM has an excellent video explaining the cloudant database and the JSON document structure.

{
    "FirstName": "Bob", 
    "Address": "5 Oak St.", 
    "Hobby": "sailing"
}

So what's the benefit of a document-oriented database like cloudant. Fields can vary from document to document. This flexibility allows the data model to evolve as the applications requirements change. In a relational database you must define the database structure (schema) before you may begin entering data, some of this structure cannot change once you start adding data. In contrast with a document-oriented database the structure can evolve without restrictions. Since the data is stored in individual documents (files) it's simple to distribute over a cluster of storage devices or servers. Document-oriented databases date back to the 1970's but began growing in popularity in the 2000's.

What's Next?

Since writing this I've moved on from IBM Cloud, although I may come back to it later. I want to see what else is available and how they differ. I have a list of 15 other IoT cloud solutions that I'm working my way through. So far, I've crossed off four. My next article will describe my experience with each.