Stream CBOR time series data into Microsoft Fabric

Why CBOR?

From "RFC 8949" published in 2020 (and you can find more details here):

The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation.

CBOR is based on the wildly successful JSON data model but stores data in binary format. Yes, you can send binary data over plain text JSON with Base64 encoding, but the result is we are making data processing more complicated with bulkier payload.

You may ask why does it matter? For example, a cellular IoT solution typically involves collecting sensor readings from remote microcontroller-based devices. These devices sometimes are very resource constrained and run on batteries. Also, if an IoT SIM card with data plan is used to transfer data, we may need to watch data usage too. CBOR's encoding in binary saves bulk and allows faster processing.

Architecture

In this article, we will use a Wi-Fi based development board (nRF7002DK from Nordic Semiconductor) to simulate sensor readings (with 3 threads running in Zephyr RTOS): Ingest CBOR data to Golioth cloud, use their Pipelines to transform CBOR to JSON and deliver to Azure Event Hub as CloudEvents. Then from Event Hub, data will be ingested into Microsoft Fabric Real-Time Intelligence (Real-Time Hub: Event Stream -> Event House -> Real Time Dashboard) for query and visualization.

Here's the diagram:

Here's the Real-time dashboard in Microsoft Fabric:

Prerequisites

1. Nordic nRF7002DK and local Wi-Fi available for internet connectivity. You can certainly leverage existing hardware with real sensors and cellular connectivity too. For example, I have a blog coming up for Thingy:91 X.
2. Install the nRF Connect SDK. Ideally, you should have also configured nRF Connect for VS Code. This will make it a lot easier to build, flash and debug. Well, there are several other dependencies you will need in order to debug, for example, SEGGER - The Embedded Experts - Downloads - J-Link / J-Trace.
3. Register your Golioth account. Follow the instructions in Getting Started | Golioth.
4. Install Golioth firmware for NCS. Follow this link for instructions: NCS with Golioth | Golioth.
5. Microsoft Fabric Trial (Free) license for 60 days. See reference.
6. Deploy Azure Event Hub.
7. Scale your Fabric capacity - Microsoft Fabric | Microsoft Learn.

Let's do it!

Developing and debugging in VS Code

Nordic Semiconductor's nRF Connect extension really made it easy to work on this solution in VS Code. I'm using this sample from Golioth to build the demo: created 2 additional threads and serialized the values into CBOR for upload.

After initializing the network and creating the Golioth client object, I have 3 threads (get_temperature(), et al.) in a while() loop to simulate sensor readings for temperature, pressure, moisture levels:

If you would like to dive deeper into the benefits of serializing your data using CBOR, I would recommend you read this excellent article regarding how it conserves bandwidth and power: CBOR Serialization in the Golioth Zephyr SDK: 20% Data Savings! - The Golioth Developer Blog

From lines 219-228 above, you can see how to serialize sensor values into the CBOR buffer. We will need to do some experiment and determine the right size for the buffer since if the buffer size is too small, the serialization of CBOR will fail. We can also inspect the CBOR data in memory and later send the buffer by calling golioth_stream_set_async() with content_type GOLIOTH_CONTENT_TYPE_CBOR:

Prepare Golioth Pipelines

Now we are streaming data from nRF7002DK to the Golioth Cloud, let's work on the pipelines in order to convert CBOR into JSON and stream into Azure Event Hub.

With Pipelines in the Golioth Console (https://console.golioth.io/), we can easily filter, transform and route data to many destinations. For example, I have 3 pipelines:

The LightDB Stream CBOR pipeline is great for debugging and as you can see, the pipeline definition is pretty self-explanatory (filter, steps, transform, destination). Without needing to read the documentation (although I do encourage you to read them!), you can easily construct sophisticated transformations and route data to your destinations (Examples | Golioth). For example, my LightDB Stream pipeline looks like:

Now, let's look at the pipeline for routing JSON data to Azure Event Hub:

Remember LightDB Stream mentioned above? It's great to able to inspect the payload being delivered to the destination:

Peek into CloudEvents in Azure Event Hub

Next, go to your Azure Portal (https://portal.azure.com/) to inspect whether data are arriving at Event Hub correctly (look into the Data Explorer from portal). If not, go back to your Golioth portal and make sure the Event Hub connection string is configured correctly in the Secrets.

Microsoft Fabric RTI

Microsoft Fabric is the all-encompassing solution from Microsoft that facilitates the discovery of insights from your time-sensitive data, allowing you to ingest, process, query, visualize, and act upon it in real-time:

It offers an end-to-end solution for event-driven scenarios, streaming data, and data logs. Whether dealing with gigabytes or petabytes, all organizational data in motion converges in the Real-Time Hub. It seamlessly connects time-based data from various sources using no-code connectors, enabling immediate visual insights, geospatial analysis, and trigger-based reactions that are all part of an organization-wide data catalog. Once you seamlessly connect any stream of data, the entire SaaS solution becomes accessible.

You can easily follow the instructions to connect your Event Hub as data source, ingest data into Event Stream, store data into the KQL database inside Event House and finally, visualize your IoT time series data with Real-Time Dashboard. My Event Stream and Event House look like:

Next, use KQL query to build the dashboard for visualization:

Sensors
| where tolong(parse_json(data).temp) < 50
| where ['time'] > ago(1h)
| project ['time'], tolong(parse_json(data).temp), tolong(parse_json(data).pres), tolong(parse_json(data).mois)
| render linechart        

Finally, you will be able to see your own dashboard as in the screenshot I have attached in the beginning of this article.

Conclusion

Cellular IoT solutions are one of the hottest topics in the embedded systems world. Being able to easily stream your time series data to those public cloud vendors from remote sensors will definitely speed up your development cycle and enable so many possibilities with data analytics and AI services in the cloud.

From the quick exercise above, with very minimal coding, we are able to stream data from an app running on a microcontroller to the cloud, filter, transform and route data to destination for real time analytics and visualization!

Thank you for reading this far. As a bonus, let me show you my fleet of toys!