Big Data: A case study from Brunata a/s

There's a lot of talk about Big Data, Internet of Things (IoT) and similar technological advances, but there are still many companies struggling with the challenges which follows from managing big volumes of data.

In this article we will touch on a Case Study from Brunata a/s about how to create a robust platform for receiving, processing, storing and sharing data in the Sub-Metering business.

First let's start with some facts to lay out the land of Brunata's business so to say: Every day we receive around 45 million data telegrams from our millions of meters installed across Europe. The meters are measuring consumption of water, heat, electricity, gas etc. and monitoring humidity and temperature.

To manage such a data load of course we need a platform which can comfortably cope with the volume, the diversity and the varying validity of data. Another motivation is the growth of our business, due in part to new European legislation requiring consumption measuring and in part because of more ecological awareness.

In response to this and to accommodate easier maintenance, scalability, reliability and much faster implementation of support for new meter types and models, we have been working on such a platform for the past 3+ years.

Big volumes of Data requires queuing & load balancing

A main task is to filter any unwanted telegrams from the data-flow, such as telegrams carrying no new information or “noise” from other wireless meters and devices on the assigned transmission bands (433 & 868 MHz).

The filtering engine was developed based on the reactive manifesto: it had to be responsive, resilient, elastic, and message driven. The message flow is not even or predictable, and it has many smaller or larger peaks. With the future growth in messages (meters) in mind, a horizontally scalable system architecture was chosen, starting with a minimum of two nodes to ensure fail-over capability. Nodes are deployed on virtual machines, which allow a relatively fast horizontal scalability.

The virtualization engine is Hyper-V and the nodes are running Windows Server. All nodes are configured equally, and as no node is assigned the master role, the collaboration between nodes is enabled by use of federated queues.

The reason for selecting a queue system is two-fold: to enable multiple nodes to collaborate on the received messages, and to allow a loose coupling between components in the filtering engine. RabbitMQ is selected as the queuing system, as it supports federation of queues, acknowledgement of transactions, and persistence of queues.

The engine is filtering messages to minimize the load on the Oracle relational database, contrary to existing legacy systems doing the filtering when writing to the database. The more advanced filters require feedback from the database and for that purpose a distributed cache, Couchbase, is applied.

Resilient, fault-tolerant and self-healing

Needless to say, a platform of this kind is truly mission critical for the sustainability of Brunata's business, which means we have high focus on performance and its operating time.

Components of the platform are all stateless and monitored by a controller, which will restart them, if they experience downtime. To monitor the system performance, probes are injected at important junctions to ensure that the engine runs as designed, and to signal when to scale with additional nodes.

An Agile approach using Scrum is basis for success

A small dedicated team of C# developers and an architect doing agile development (Scrum) provided the first version in approx. a year, which also included a major change in requirements and test management.

Builds are created using TeamCity Enterprise, which is connected to the Git version control system. All code is reviewed and committed to Git using Gerrit. Requirements are broken down into user stories, and acceptance testing is based on these performed by SpecFlow tests.

SpecFlow tests are written as part of delivering the user story. Unit testing  is also an essential part of the coding. By applying both unit test and SpecFlow tests most of the acceptance testing is performed in the release build, which of course saves time at release, but more importantly ensures built-in quality within the sprint.

Focus is shifting from input to output

Now with the input, processing and storage parts of the platform complete in their first evolution, we have turned our focus towards the use and availability of data.

There is increasing interest from customers and 3rd parties to be able to exchange data in various ways and to analyze the massive amounts of data being measured more intelligently, so using the data in new contexts and improving visualization are the next steps. To that end we are in the process of making a REST API available (services based on Katana).

------ || ------

We hope with this Case Study to inspire other businesses to work on their Data Platforms as that is the most essential foundation for Digital Transformation.

Feel free to comment and to share this!

Thanks to the Team of dedicated Developers and Architects at Brunata for help writing this piece.