Enterprise AI Trilogy: Data Strategy for Real-Time Analytics

In this article we try to explore two main questions -

a.??????Why does your business need real-time analytics?

b.??????What does it involve to build your enterprise data strategy to move from “data at rest” to “data in motion”?

We delve into it as a trilogy – real time analytics, data in motion and event driven architecture to understand the journey that enterprise technology leadership needs to consider as a part of their roadmap.

Business Relevance of Real-time Data

Business needs everything real-time or near-real-time.

However, building technology to handle real-time is not always easy – be it handling real-time data, real-time analytics and/or machine learning.

?Traditionally we have been working with “data at rest” (data loaded and stored with batch processes), however as systems evolve, we are witness a growing popularity with streaming data and “data in motion” is becoming popular.

As the technology for developing becomes easier and cheaper, enterprise leaders are working on roadmap moving in the direction – from batch processing to stream processing, from request-driven to event-driven architecture.

Why real-time data?

“Real-time data is data that is available as soon as it’s created and acquired”

We see an emerging trend in the business world with use cases using real time data.

Look at examples like dynamic pricing used by uber and airbnb, or real-time-bidding(RTB) ad-space in advertising, credit scoring, fraud detection, estimations for driving and delivery, recommendations – these and similar use cases need predictions to happen within milliseconds.

Not to say that stored data-at-rest does not have its importance; but from a business perspective, use cases based on real-time analytics holds tremendous value and can harness decision-making and become a key differentiator to your business.

Understanding data-in-motion and its significance

“Data in motion also referred to as data in transit, is digital information transferred between locations either within or between computer systems”

Data in motion (stream data) is closely related with stream processing?

“Stream processing continuously analyzes data as it flows into the organization, and then triggers an action based on the information flow.”

Increasingly, enterprise applications are being built as distributed systems - with integration at its core to deliver business outcomes. Data pours in the organization from all directions, be it transactional, operational, sensors, mobile, web, supply chain partners and the so on. The need for integration of systems / building distributed systems is a solid driver for messaging and in turn real-time data communication.

To achieve this, we see companies building their data strategy for moving from batch processing to stream processing or adopting a hybrid approach.

?The need of event-driven-architecture

?The usage of APIs has become ubiquitous and it has been the go-to technology for integration architecture. Rest APIs operate as a set of ongoing conversations over the HTTP protocol, however streaming sends updates to the client when the events occur. Event-based approach beats request-response based architectures by providing recoverability, fault tolerance, scalability, agility amongst others. With a lot of lucrative benefits offered with streaming processing, we have system architects debating between REST vs Streaming APIs ?to make the right choice for their integration technology.

With the promise they provide; it is fair to say that stream processing pipelines have a very important place for modern application integration.

? REST APIs can be thought of like an ongoing conversation.

Streaming APIs, on the other hand, send updates to the client when events occur.

Machine Learning Models and Online Inferences

Machine Learning models can be seen at two levels

• Level 1 is online predictions: ML systems that make predictions in real-time
• Level 2 is continual learning: ML systems that incorporate new data and update in real-time

(source: Machine learning is going real-time: Here's why and how)

Companies are moving towards online inference. However mainstream adoption of continual learning is still a few years away.

A recommended approach towards online predictions is: Stage 1- batch predications, Stage 2 is online predictions with batch features extracted from historic data and Stage 3 – is online prediction with complex streaming and batch features.

In summary, moving to ML models for streaming data is a staged approach built over time.

?

The Value Proposition – TCO / ROI

Does your business team see value with working real-time analytics?

?How do you measure the business value of investing in building real-time analytics?

It is very apparent that the journey to real-time analytics is complex and expensive. So, is it worthwhile? The three key drivers to your strategic objectives can be increase revenue, decrease cost and mitigate risk. An exercise to build the TCO / ROI Business Case with the real-time analytics use case definition to the estimation for design-build-operate is something worth-while for company.

Building the TCO/ROI Business Case for real time analytics is a good starting point for the journey to get your organization from “data-at-rest” to “data-in-motion”

?

?

?

?