The Dark Side of Big Data

Originally published on Forbes, click here to see.

Because of many million more sensors, faster computers and better resolution satellite imagery producing vast swathes of data, we now know much more about Hurricanes than ever before. In fact typically we see them around a week earlier than just ten years ago. Yet due to the vast complexity of weather interactions and their near perfect chaos, we’ve made little progress on predicting the path. So we now have a week longer to be anxious.

There is no better metaphor for the current reality of big data in Marketing. We can celebrate the process or be embarrassed by the meaninglessness.

We’ve become enamored with the technology, we love the blue lights on big server farms, a real time dashboard makes us feel in control. Yet I’m aware we’re celebrating inputs not outputs.

We celebrate the SIZE of big data when we should be excited only by the profoundness of decisions we can make with it.

We overestimate the importance of what we know, rather than focus on what this data makes clear we don’t actually know. The more you know, the more you know you don’t know.

Above all I’m concerned we believe that big data is used as a cure all, we’ve somehow assumed that it will solve all our problems and I think that the reality doesn’t meet the hype and maybe it won’t for the medium term future.

We’re using Big Data wrong.

Big Data shows supermarkets that at times of higher humidity that sales of bagged salads go up. That’s a pretty useful way for stores to predict demand, but it doesn’t show the whole picture. We need to establish the causal link and work with that.

Big Data tells us that people who buy toasters, often look at other toasters, so Amazon takes great pride in showing us endless toasters that people like me also considered. Rather than showing me toast racks or egg slicers or matching kettles.

Big Data thinks I like certain movies, when in fact I like certain movies at certain times or in certain situations. The crap I watch after a beer on a plane ruins any algorithms chances.

Big Data establishes that I’m interested in a hotel in Chicago next week, and then endlessly retargets me with the same hotel I’ve booked rather than offering me boat tours or car rental.

Big Data tells my American Express Platinum app to send me a real time offer to eat Lunch in Dunkin Donuts at 4pm in the Bronx as I drive through, I think not.

Big Data doesn’t get how weird we are. Big data can’t explain how I can be a Guardian reading, Whole Foods loving, Golf playing guy that owns an old lowered plastidipped BMW with spinning chrome wheels.  Well, I know I can’t.  People are irrational, the do things for strange reasons that even they don’t understand. They may explain it, but they will post rationalize to seem more logical.

We need to get the balance right.

We’ve in a world where we’re over emphasizing the hardness and science of data, the objectivity of maths and the precision of software. We need this, but we have to unleash the powerful levers of data to provide value. We need to celebrate those who ignore data, those who challenge it, those who used data to back up insights from their gut, not that used a spreadsheet to find one. 

For centuries marketing has been a dark art, we’ve made decisions with feelings in our stomachs. We’ve had to empathize with our customers and to feel our way around, dial in instincts. We didn’t know much, but we didn’t expect to. Nobody for centuries complained about a lack of data, they just got on with finding rich ways to connect their brand with people.  And it worked.

In 2016 in any nation or company around the world it’s very hard to be fired for making what turns out to be a terrible but rational decision. Any choice, supported by enough data, is defendable.  Yet if you choose the opposite, even a moderately poor decision makes people feel vulnerable. 

Today, an irrational decision that works wonders could be thought to be wreckless and lucky, or risky but that worked out. An irrational decision that doesn’t work is almost certain unemployment.

Yet most and the greatest leaps in business happen from the remarkable not the iterative. Small iterative predictable but unremarkable success  comes from data and trend lines, and linear uncreative projection. The shifts come from something data can’t see because it’s not happened yet. 

In 2005 I worked alongside Nokia advanced Nseries unit and we trialled many touchscreen phones with people before the iPhone launched. The customer feedback was terrible. The screens people felt would get smudgy, the phones battery life would be intolerable and above all else people felt like the screens would break. What we needed was confidence to overwrite data.

Data forces us to do the opposite, it forces us to make things tidy, to get closer to an iterative solution. Data makes Sony Discman’s ever cheaper and lighter while the world wakes up to MP3’s. Data forces us to make sensible decisions. But above all else a culture of data makes risk aversion the goal.

Above all else data only knows the past, Data can't show things that have never happened, no trend line predicted Pokemon Go, even when we do project data into the past we do so as a linear interpolation, when few trends in life are ever such. The only way we can hope to predict future events is with data and empathy, imagination and joining dots, and for that you need gut.