The decade of understanding and action

As the year 2019 closed and we roll in to a new decade, I was thinking what is it in our industry that will mark the next ten years. However, before looking forward we need to look back and understand what the focus was for the decade we have just left.

Without a doubt the biggest thing I had experienced in the 2010s was the relentless pursuit of storage technologies to tackle the onslaught of data collection we experienced. The drive for digital transformation and the desire for insights, resulted in a focus on capturing data on everything... every transaction, every log, every document. The storage industry was awash with technologies and advancements to tackle the challenge of storing and retaining more data for longer periods. Advancements such as higher density flash drives drove speed of ingestion and data recall, compression and deduplication technologies meant we could store more and drive more cost efficiencies in our platforms, finally high end management technologies filtered down in to mid range storage systems empowering administrators to do more with their systems while reducing the complexity in operations.

It's without a doubt that the advancement in analytics and the ability to put tools in to the hands of all users from data scientists to business users was a large driver for this push in the storage world to store more from more sources, be it on premise or in the cloud. As the tools became simpler yet more powerful, organizations hungered for more data to generate more insight. This is the state we have come to as closed out the decade, we stored data to provide analysis and insight.

As we look in to the next decade, there are numerous other technological advancements that will have a profound impact on what we saw in the last decade. I would propose that the coming years will see a reversal in what we saw before, we will understand and action data to determine what we store. These are some of the factors why.

The Explosion of Edge Computing

Miniaturization and cost reduction in manufacturing is enabling compute power and systems to be developed smaller and cheaper than ever before. One of the technologies I am enjoying experimenting with right now are Raspberry Pi devices.

By Michael Henzler / Wikimedia Commons, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=80140656

These mighty little devices can do pretty much what our desktop computers can do today, they have a desktop operating system, supported applications for office work, they have built in developer tools to build and deploy applications and more.

Putting all that aside, what is really interesting is the large number of sensors and hardware that can be easily attached to these small computers at low costs. You need a camera attached to the device? Depending on resolution you can get them from $5 to $30 at Amazon. You need temperature and humidity sensors, you can get those for around $10 at Amazon. You need a touch screen display, you can pick up one of those for less than $50. All of this flexibility in configuration at low prices means organizations can start to build their own custom sensor based systems to be run at the edge.

Beyond Raspberry Pi, even the sensors themselves are becoming smarter, video cameras no longer just capture images, but can even have their own operating environments that can host applications that can be executed directly on the device. They can capture video streams with high resolutions, they have night vision built in to them, all of this adds up to rich data streams from which we can gather data from.

With these sensor based systems we can start generating lots of interesting data that can be used in a variety of use cases. The thing we need to take in to account though when we do deploy these systems is that organizations will not be deploying one, two or five, they will most likely be deploying one hundred, or even a thousand of such devices blanketing the organizations with the ability to capture massive amounts of data. Herein lies the problem, the question to ask is not only can we store all this data, but can we ingest it all across our networks without impacting the rest of the business? Maybe simple text based data might be ok, but what about live video streams in 4K or even 8K?

The way forward has to be more intelligent processing of what is going on at the edge. At our recent customer and partner event last year in San Francisco, a partner of ours was showcasing exactly how they were tackling that particular challenge. They themselves were receiving live video streams of alleyways in a city to ensure the smooth flow of the city's utility vehicles. What they wanted to be able to do in real time was detect objects that might constrict the flow of these vehicles, but pushing a live video stream from multiple cameras across hundreds of city alleyways was just impossible. The solution, leverage the power of the edge to recognize the objects in the video stream, and transmit the identified object tag and its geo-co-ordinates as a text based data stream to a monitoring application that could recreate a digital twin and all the activity going on within it in real-time.

This example leads me to my second point...

Machine Learning and AI will thrive at the Edge

Machine learning and other relevant AI based technologies, while being around for a number of years, really saw an accelerated state of adoption in the last year. Due to the increasing sets of frameworks and tools being made available from software and cloud vendors, organizations have been embracing machine learning for a variety of different use cases from object recognition to preventative maintenance.

Even just recently Microsoft has released their AI Builder toolset for Power Apps, putting the ability to build AI models for things like object detection, prediction or text classification, in the hands of everyday users. While this is great for applications hosted and executed in the cloud, these sorts of tools and frameworks will have a much greater benefit at the edge themselves. As with our previous example, the ability to ingest and understand video from a live stream at the edge and transform this into data that can be better handled and transmitted centrally is essential to ensuring that edge computing can provide value in an enterprise context.

This means that AI and Machine Learning has to be successful in executing at the edge. For example a company like Cooch.ai has released object recognition frameworks at the edge, recognizing the need for this to happen closer to where the action is taking place, and transmitting only the data which is necessary for the organization to act on. With this capability we can start to manage the data with which we work with in a distributed architecture fashion. If we have the ability to recognize objects at the edge and understand what they are, then it stands to reason with the computing power we have with these edge devices, that we can also take action at the edge. Leading me to my final point...

Intelligent Actions will take place at the edge, not centrally

When dealing with real-time systems we need to ensure we have real-time actions. Regardless of network speeds, sending data to a central location to be processed and a resulting instruction being sent back wont deliver real-time. There will be a need to have intelligent systems that can orchestrate the feeds of all these sensors and data flows at the edge, applying rules and best practices to make a determination to issue some sort of action to take place.

This requires application architects to have a physical distributed view of the applications they are building today, understanding what intelligence and actions need to be distributed at the edge, and what can be executed centrally. One of the things I am proud about what we do at Vantiq is the ability to help our solution developers build and deploy aspects of their solutions in a distributed environment, some at the edge and some centrally.

As an example a customer we are working with dealing with employee safety, and they want to ensure that people do not accidentally wander in to dangerous areas which might cause themselves harm. The data we capture at the edge might be GPS co-ordinates of a user walking in to a logical geo-fenced zone recorded every second, it might be a video stream of what was going on etc., this data would flood the network and could quite possible delay an action being taken resulting in a catastrophic outcome. What we really need is for the edge to determine the action to take, sound an alarm, alert nearby security, and after for the pertinent data to be tracked centrally, which could be in the form of a record of the event.

This final point brings me back to my original point...

The Future is Understand, Action then Store

As we embrace real-time systems many organizations are going to come to the reality that we will be reversing the flow, we will understand data in real-time, take some sort of action based on that understanding, then store the result of that action. Edge computing and AI technologies while resulting in amazing new application experiences will exponentially create new and massive sources of streaming data. More of this understanding and action will take place at the edge, and this means organizations need rethink their approach to capturing and storing data. Is it necessary to capture every single measurement from every single sensor from every single edge device, or is it more important to store the conditions of an event when a something has happened?