New Big Data and IoT Trend: Analytics 'On The Edge'

An Internet of Things approach to business brings many benefits, primarily access to more data and better insights.

It also brings new challenges. The sheer size and speed of the data collected when every device involved in your business process is online, connected and communicating can strain the sturdiest network infrastructure. In situations where timing is critical, delays caused by bandwidth congestion or inefficiently routed data can cause serious problems.

To solve these issues, the concept of “edge analytics” is gaining popularity.  Sometimes known as distributed analytics, it basically means designing systems where analytics is performed at the point where (or very close to where) the data is collected. Often, this is where action based on the insights provided by the data is most needed. Rather than designing centralized systems where all the data is sent back to your data warehouse in a raw state, where it has to be cleaned and analyzed before being of any value, why not do everything at the “edge” of the system?

A simple example would be a massive scale CCTV security system, with perhaps thousands or tens of thousands of cameras covering a large area. It’s likely that 99.9% of the footage captured by the cameras will be of no use for the job it’s supposed to be doing – e.g. detecting intruders. Hours and hours of still footage is likely to be captured for every second of useful video. So what’s the point of all of that data being streamed in real-time across your network, generating expense as well as possible compliance burdens?

Wouldn’t it be better if the images themselves could be analyzed within the cameras at the moment it is captured, and anything found to be useless either discarded or marked as low priority, freeing up centralized resources to work on data of actual value?

It’s a model which is increasingly being rolled out – a recent IDC FutureScape for IoT report found that by 2018, 40% of IoT data will be stored, processed, analyzed and acted upon at the edge of the network where it is created.

Cisco is one big tech company keen to involve itself with this trend, having recently announced that it is launching a dedicated platform for building and running edge analytics-based systems.

Their VP and GM of data and analytics, Mike Flannagan, told me “For me the idea of analytics at the edge is about the notion of doing the right amount of processing of data at the right place.

“If you’re generating tons of data in your data center, then analyze it in your data center.

“If you’re generating tons of data at the edge, and you have unlimited bandwidth, then you can still send it all back to the data center – its fine.

“But there are environments where that is not possible, and the two main things that come together to make it not possible are the volume of data and the perishable nature of data.”

It is worth noting that Flannagan does not in any way see edge-based analytics replacing the centralized data center model. Rather it is an approach which can be used to supplement or augment analytics capabilities in certain situations, such as when insight needs to be acted upon very quickly.

An interesting example comes from the world of powerboat racing. The 200mph twin engine boats form part of a data system which includes a land crew constantly receiving and analyzing data on the boats’ performance as they race, often many miles offshore.

However one particular algorithm generates data which needs to be fed back to the pilot within a split second. The powerful engines must operate at the highest output levels possible without burning out, to stand a chance at breaking world records and winning races. SilverHook Powerboats implemented a system which gives the pilots instant feedback, allowing them to ease off on the throttle at the right point, based on real-time analytics carried out within the boats. This potentially shaves seconds off the time it would take if the data was sent to the remote team first, for processing, and prevents expensive, race-losing engine failure.

Other uses for analytics on the edge include:

Large retailers could analyse point of sales data as it is captured, and enable cross selling or up-selling on-the-fly, while reducing bandwidth overheads of sending all sales data to a centralized analytics server in real time.

Emergency repair work and equipment down-time can be reduced when manufacturers build edge-based analytical systems into machinery and vehicles, allowing them to decide for themselves when it is time to reduce power output or send an alert that a part may be due for replacement. 

Smart City architects can build edge analytics into systems such as traffic signals, allowing intelligent monitoring and management of traffic. Pollution levels caused by traffic could be monitored in real time and regulated by reducing traffic flow when air quality falls below a certain level.

Autonomous and driverless vehicles will heavily rely on edge analytics systems for functions that require immediate response, such as hazard avoidance. At the same time they will rely on centralized analytics for fleet management and optimization of pathfinding. They will also rely on a middle ground, sometimes known as “the fog”. This involving analytics carried out between a network of vehicles which are close together, for the purpose of managing local traffic flow. 

Of course edge analytics is not suitable for every IOT implementation. Flannagan says “Clickstream analytics – information about web traffic – all that data is centralized so there is no edge component. It would be ridiculous to suggest that any sort of distributed analytics would be necessary.

“What I am really evangelizing is the need to carefully plan where you process data to make sure you’re processing it at the most efficient place, otherwise you are unnecessarily spending money to move data around, that you don’t have to spend.

“It’s just being sensible about where you process your data. And in those places where processing your data locally can save you a bunch of time and money or give you insights, that’s where edge analytics becomes very powerful.”

In short, the principle that makes edge analytics such an enticing prospect is that it means bringing the analytics to the data, rather than the other way round. As data sets grow ever larger, and IoT-enabled devices grow ever smarter, it is likely that it will become an increasingly important strategy for those looking to implement the most efficient analytics architectures.

As always, I would love to hear your thoughts on this topic. 

Thank you for reading my post. Here at LinkedIn and at Forbes I regularly write about management, technology and Big Data. If you would like to read my future posts then please click 'Follow' and feel free to also connect via Twitter, Facebook, Slideshare, and The Advanced Performance Institute.

You might also be interested in my brand new and free ebook on Big Data in Practice, which includes 3 Amazing use cases from NASA, Dominos Pizza and the NFL. You can download the ebook from here: Big Data in Practice eBook

Image: Shutterstock