The Rise of DataOps

"Decision makers are used to making judgments. Any CEO understands statistics at a gut level, because that's what they do every day. They may not know the math behind it, but the idea of collecting evidence, iterating on it and basing decisions on this is intuitive for executives." - Paco Nathan

Ever made a decision that you knew was right but couldn’t explain why? All you could do was shrug, and say, “It’s my gut instinct”. Not too long ago (and even up to the present time, in some cases), most decisions were based on gut instinct. This doesn’t necessarily mean that there was a lack of information; more that the decision wasn’t entirely based on the data, in part because enough of it just wasn’t available. Instead, people filled the gaps with ‘experience’. Nowadays, however, experience is no substitute for statistics.

We’re quickly moving away from data analytics untilised as a discreet action to gain a particular insight or solution to one problem, and moving into an age of continuous data flow and analysis. Data will not only become the primary driving force behind decision making, but the access to it and scope of it will allow us to meaningfully articulate how we came to the answer.

Make no mistake; the new decade will be about DataOps, and the continuous delivery of information.

What is DataOps?

DataOps is about enabling a continuous and unobstructed flow of access to and insight from data. It focuses on improving the speed and accuracy of analytics, which covers automation, integration, quality control, data access, and deployment and management. It expands and quickens the data analytics pipeline.

In the last decade, the quality of and access to data has improved out of sight.

1. Volume We have seen an exponential growth in the amount of data generated and captured from devices and software.
2. Variety From structured to unstructured, geographic to emotional, we now have access to many different types of data.
3. Velocity The frequency at which data is generated and processed is higher than ever.
4. Veracity The trustworthiness of the data collected has increased.

However, the four V’s will all be for nought if we cannot see the value of data.

The Value of DDDM

Data driven decision making (DDDM) is a process that involves collecting data based on measurable goals or key performance indicators (KPIs). DDDM is all about working towards key business goals by prioritising the analysis of data over intangible and often unexplainable notions found under the umbrella term, ‘gut instinct’.

DDDM hasn’t always been easy. A couple of years ago, an Experian Data Quality survey found that 53% of chief data officers cited a lack of data access as the key barrier to driving better decision making. So what’s changed?

DataOps. DDDM is not new, but DataOps has changed its scope and ability. The best way to demonstrate how the two work hand-in-hand is through an example.

Homing in on the trouble spots

Imagine a fictitious organisation called Global Corp, which has two million employees scattered across thousands of stores worldwide, as well as offices in all major cities. Now imagine trying to get metrics on all of these branches. You can be overwhelmed, or you can use DataOps to get a birds-eye view of the data in a meaningful way.

The next part helps if you think about those spy movies where the protagonist pulls out a pair of tiny binoculars that can zoom in on a business card from three miles away. The data gathered can be visualised in such a way that you can pinpoint individual stores that are underperforming. But it doesn’t stop there; we can then drill down into the score and view historical metrics, compare that store with other branches and gain actionable insights into what decisions will best support the underperforming store and help it improve.

By having the infrastructure in place that allows a continuous flow of data through an organisation, DataOps takes DDDM to an entirely new level. This isn’t so much about replacing ‘gut instinct’ – experience will always be highly valued – but enabling decision-makers to better articulate the reasons behind the decisions they know are right.

In the case of Global Corp’s underperforming store, DataOps allows it to come to the attention of upper management before it is too late. The problem may be something local like an inept store manager, or something that store has no control over. In the dim, dark past, this underperforming store would have slipped through the cracks. Today, however, DataOps ensures decision makers are aware of the store’s issues before it is too late, and can even uncover opportunities that are only obvious when you connect previously siloed data.