How Not to Be Surprised by Big Data Misses

Big Data…

You know the term.

The knowledgeable amongst us reference Big Data at least once a week…and the really savvy at least once a day…and the incredibly shrewd at least once a meeting…while the most serious invoke Big Data no matter the topic of discussion.

But what is Big Data…really?

big da·ta: noun Computing

Extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions:

“much IT investment is going towards managing and maintaining big data”

As is much investment in general…much of which is directed by the “most serious” who invoke the term….

I also recommend that you search the term “Big Data” and browse some of the links to help you get a full view from a variety of companies and sources so that you, too, can in the words of one of the big players:

“Learn about big data and how ‘we’ can help you use big data to achieve big results.”

And it’s those big results we are all chasing. Big game-changing results, big world-shaking results, big profit-making results….

Needless to say I spent hours searching for the best examples to share…the ones that would really nail the notion of mega data sets that used human behavior and interaction to solve complex and pressing problems.

Here is what I found…Macy’s, Tesco, American Express and Wal-Mart stores seem to be amongst the most successful users of Big Data–driven solutions and, according to the write-ups (to be fair, not from them), the results generated are worth billions of dollars…HMMM…I also looked in vain in each of their earnings reports to see additional billions of dollars of profit reported due to Big Data…to no avail…

But put my cynicism aside for a moment…trust me, I will come back to it…

Municipal services is an area where I do believe Big Data provides insights that can lead to solutions such as traffic control, crime prevention and even snow cleaning during big storms.

However, it is healthcare where the world is already benefiting from Big Data game-changing initiatives and where it really is Big Data at work…more on that latter…

McKinsey & Company compiled a report for the Center for US Health System Reform which identified four main sources of big data in the healthcare industry.

They are:

• Activity (claims) and cost data
  • These are the basic figures showing the amount of care which has been supplied by providers in the system, and the cost of paying for that care. Analysis of this tells us about the spread of diseases, and the priority that should be given to dealing with specific health threats.
• Clinical data
  • These include patient medical records and images gathered during examinations or procedures, as well as doctors’ notes.
• Pharmaceutical R&D data
  • Over the last few years a large number of partnerships have sprung up between pharmaceutical companies. In the US major firms such as Pfizer and Novartis pool their data from trials into the clinicaltrials.gov website. And in the UK GlaxoSmithKline recently unveiled its partnership with the SAS Institute which aims to increase collaboration based on data from clinical trials.
• Patient behaviour and sentiment data
  • This is data from over-the-counter drug sales combined with the latest “wearables” which monitor your activity and heart rates, patient experience and customer satisfaction surveys as well as the vast amount of unstructured information about our lifestyles broadcast every day over social media.

And if you have been following the Ebola story, Big Data is at play here as well – although sadly the human factor constantly changes the info…

All good so far…and let me just say my cynicism is not related to data per se – I think many of these companies are doing amazing and great things with real data – I just question whether or not it is really BIG DATA in the true meaning of the concept…“extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions.”

All of which leads me to the headlines in the UK late last week:

“Cameron’s Conservatives win in surprise UK election”

“Unexpected triumph means Cameron can govern UK on his own”

And on and on…

Come on!!!

Tell me it ain’t so!!!

We can add billions of dollars to the sales of a company, we can find the root cause of diseases and we can’t predict an election?

And by the way – the UK is not alone – another recent high-profile election garnered the same headlines:

“Israeli election results: How did Benjamin Netanyahu pull off surprise victory?”

And if you search, you will find way more…

So what’s up here?

Do we not have enough data to make it BIG DATA?

Is voting not a human behavior or interaction??? Some might say so….just saying…

Or is our view of this whole thing so skewed by the investment opportunities that we are missing the trees for the BIG DATA?

Returning to the election results…

Let’s be clear – data has been used to predict elections for a long time – with varying results.

In an online article on UK-based IT Pro, the following quote was recorded:

“People are wonderfully unpredictable,” says Martin Lee, cybercrime manager at Alert Logic. “You can create mathematical models that describe everything that is likely to happen to the finest detail, and then people go and do something you didn’t expect.

And there you have it…

HUMAN SERENDIPITY

The one factor we cannot account for and the elusive grail in the data picture, whether we talk about elections, crime or disease…

We are HUMANS…and while you can take good better-than-educated guesses at what I might like, or buy, or do…or how I might vote…at least a good 50% of the time – and I am being conservative – I will fool you and do just the opposite…

Another great story…

The Wall Street Journal reports…

Amazon Mines Its Data Trove To Bet On TV’s Next Hit….The Internet retail giant, known for efficiency and top-notch logistics, is venturing into the unpredictable game of searching for TV hits. The company is betting it can improve on the traditional TV development process by collecting viewer feedback in unprecedented ways and using it to make less risky bets on which shows to produce….

And 6 months later in The New York Times…

It’s not like you can come in on Tuesday and the computer says: ‘Doot, doot, doot. Here are the shows you are going to do…’ You have to use judgment as well. – Roy Price, Head Of Amazon Studios, August 2014

And a year after that…

This is the first time Prime Instant Video has given a creator such a commitment without first vetting the show with the public. Typically, Amazon orders one episode of a show and then invites people to vote on whether it should become a series… –The Wall Street Journal

And there you have it…

In the end…they say tigers are unpredictable and so are we….

Perhaps the best summation I can share is from someone who is known for trying to predict the future…listen:

All human plans [are] subject to ruthless revision by Nature, or Fate, or whatever one preferred to call the powers behind the Universe. Arthur C. Clarke, 2010: Odyssey Two

Or perhaps this thought from a famous icon of the “future”:

We humans are full of unpredictable emotions that logic alone cannot solve. James T. Kirk

At the end of the day the issue isn’t BIG DATA and its efficacy – the use of data is not new – read your Bible….it’s just the digibabble associated with it that leads to surprises.

Embrace the use of data; believe it can change the world – because it can.

But most importantly remember the words in the dictionary definition…relating to human behavior and interactions…

And that is the key that will unlock it all for you and for the world….

Turn that key….DOOT…DOOT…DOOT…