Customer Analytics: Too Much Data?

In his book Messy, best-selling author and economist Tim Harford offers a brilliant discussion of how legendary jazz pianist Keith Jarrett was forced to innovate his playing style when he found a sub-par piano that was out of tune and had insufficient volume to fill the Cologne Opera House, where Jarrett was about to play. After considering cancelling the concert, Jarrett felt sorry for the 17-year old concert promoter who had pulled the necessary strings to bring him to Cologne and decided to play despite the lack of a decent piano. Faced with a new situation and with clearly less-than-perfect tools, Jarrett could not anchor on his standard play. He had to innovate and improvise. What happened was pure magic. 

(consider playing Keith Jarrett's K?ln Concert now :-))

In Harfords own words: “Jarrett was avoiding those upper registers, he was sticking to the middle tones of the keyboard, which gave the piece a soothing, ambient quality. But also, because the piano was so quiet, he had to set up these rumbling, repetitive riffs in the bass. And he stood up twisting, pounding down on the keys, desperately trying to create enough volume to reach the people in the back row.” With sales of more than 3.5 million records, The K?ln Concert is the best-selling solo album in jazz history, and the all-time best-selling piano album. But that was only possible because Jarrett knew he had to rely on his intuition and gut feeling, to innovate and improvise his playing style...

Big Data: Three V’s 

Big Data is usually characterized by at least three “Vs”: it has a very high volume (petabytes…), high velocity (often in real-time) and high variety (includes numbers but also text, images, video, etc). Some authors refer to other “Vs” such as “veracity” (reliability of the data) and “value” (see e.g. Wedel and Kannan Journal of Marketing 2016). With such exploding datasets in terms of volume, but also velocity and variety, it may be tempting to feel that we can now know the customer preferences inside out all the time. Evidence-based decisions and marketing analytics should make it easier to develop the right products and offer them to the right customer at the right time.

Indeed, Big Data has many unquestionable benefits. There are dozens of stories and anecdotes showing the potential of Big Data and predictive analytics to revolutionize how firms market to customers (Who has not yet heard "How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did"?). Yet, as the K?ln Concert anecdote shows, there may also exist a dark side of having so much data at our hands. How would have Jarrett played if, on that cold night in January 24, 1975, he would have played with a super smart piano that would know the preferences of all the 1,400 people filling in the Cologne Opera House? Would such a smart piano, with so voluminous data help Keith Jarrett perform even better?

Let me explore three risks I see with Big Data, especially with the tendency to focus on the first "V" of Big Data (volume) at the expense of two potentially more important "Vs" (velocity and variety). I call these risks the three “Is” of Big Data: (1) Incremental Mindset, (2) Illusion of Control and (3) Insensitivity to Costs.

Risk #1: Incremental Mindset

The first risk is that Big Data may lead us to incrementalism. Despite its volume, variety and even velocity, Big Data tends to focus on the present or past behavior of our customers. Hence, unless we complement such data with some foresight (which typically requires us to stretch beyond what the data is telling us), we may run the risk of anchoring too much on the customers’ current and past needs and behaviors. Moreover, it is rational to give more weight to higher than to lower volumes of data. Hence, Big Data may make us very inflexible when considering deviations from our current practices. Had Keith Jarrett had the perfect piano, he would not have felt the need to deviate from his standard way of playing the piano, and possibly a masterpiece would have been missed.

Risk #2: Illusion of Control 

A related risk has to do with Big Data induced “illusion of control”. First demonstrated by psychologist Ellen Langer, “illusion of control” is a generalized human tendency to overestimate our ability to control events even when such control is hard or impossible. One of the risks of having a lot of data is, precisely, that we may believe that we are fully in control of dynamics in customer needs and other market changes. However, many trends are very hard to capture with standard datasets, even if such datasets have high volume, velocity and variety. For instance, sometimes not even customers are aware of their latent needs and changing preferences. It may take some experimentation and gut feeling to design and test new value propositions. If Big Data leads marketers to overestimate their degree of control over changing customer needs, its value may be lower than one would expect.

Risk #3: Insensitivity to Costs and to the Cost-Benefit Trade-Off

The third risk I see with Big Data is insensitivity to costs and to the cost-benefit trade-off of analyzing gigantic datasets. In a February 2015 interview, Wharton Professor Eric Bradlow said “the first thing when somebody sends me a big data set, the first thing I say, and, excuse my French, is ‘oh shit.’” He then explains the statistical concepts of data sufficiency and smart data compression which point in one direction: For most decisions one may not actually need gigantic datasets. This means that the benefits of the first “V” of Big Data (“Volume”) may be often overestimated leading to unnecessary costs (with hiring, computational requirements, etc). Typically, the other two “Vs” are (velocity and variety) are more promising than volume per se. 

In sum, finding creative ways to analyze reasonably sized datasets with new sources of data (e.g., eye-tracking, movement data, text data, images, etc) and doing so in real-time or near real-time seems to be where the bang for the buck lies, not in volume per se. Exploiting these new, often unstructured, types of data, however, will require firms to carefully balance (1) analytics knowledge (e.g., machine learning and computer science techniques possibly complemented with econometric and statistical techniques for causal inferences) with (2) business insights (the standard “MBA” types; "data translators"). Balancing analytics and business insight is where the sweet spot is.