Data Science vs the C Suite. A solution

‘Vs’? That’s how it feels to many data scientists and to many C Suite Execs I’ve spoken to. Even the best, most brilliant, most successful data scientists I know from around the world feel that their work is often not understood, appreciated or appropriately used by business leaders. And most business leaders I’ve spoken to struggle with reconciling the gap between expectations for how data science should transform their organisations and the reality of achieving results. I have a solution. 

Magic is possible 

I’m recently back from chairing a day of talks and panels at Strata, the world's best data science conference in NYC. It was an amazing experience. The speakers and panellists were all brilliant. It’s remarkable what’s possible with data science these days. Many of the examples they shared felt like they were sufficiently advanced to almost feel like magic. But all is not well in data science paradise. 

Criteria for success

Magic is possible when three things combine: the right person, the right data and the right business question. These sound obvious! But in my experience these rarely combine, leaving the real world experiences of many businesses far short of the ‘magic’ that is rightly expected. Particularly when we’re talking about data science for anything that’s vaguely strategic. The further you move away from basic ‘operational optimisation’ problems like recommendation engines the more this problem manifests itself. 

The first problem is that most great data scientists don’t sufficiently understand business and most great business leaders don’t sufficiently understand data science. The best data scientist is often not well versed in the business considerations and questions required to do data science that’s truly impactful. It takes many years of exec meetings and grinding out results to learn what makes a business tick and to know how best to apply data science to it. This is generally incompatible with the many years of grinding out data science work that’s required to be the best in that field. 

This sometimes results in great data science … but that doesn’t quite answer the important questions. I see a lot of talks like this at data science conferences. Amazing science. Magic. But the business person in me often feels uncomfortable that it just wasn’t addressing a very useful business problem. 

The flip is also true. Most in the C Suite are not well versed in data science and it’s uses. This generation of leaders grew up in something approximating a ‘pre-data’ era and are doing their best despite not having the many years of experience in data projects required to know its uses and limitations. They are hearing about the magic that can be created and want some for themselves but they don’t have enough understanding to know how to apply data science. 

The final problem is with getting ‘the right data’. Amongst the many caveats that are critical to success but rarely talked about are that there be enough examples, not many outliers and not too many missing values. One or more of these constraints have stopped about three quarters of the data science projects I’ve seen attempted in real world businesses. Literally stopped. The projects, in the end, weren’t achievable. They were aborted. This leads to frustration and wasted time / money for both the data scientists and the companies. Data scientists, vendors and those setting expectations amongst C Suites should be much clearer up front about what conditions are necessary for data science to work. This would help it to find the right business problems much more efficiently!

The CEO of a previous multinational, multi-billion dollar company I worked for once stood up at a leadership team conference and told people how he’d heard a talk from a Spotify data scientist. He said that they had the best mathematicians in the world and that’s why they are successful. And that we need the best mathematicians in the world if we have any hope of succeeding in the new world. I grabbed him over coffee later in the day to explain that I’ve had some of the best data scientists in the world look at our business and our data and that we just don’t have anything that they can work on. Not enough data and not the right business problems for their skills. 

Bridging the gap 

So, how do we bridge the gap between the magic that’s possible and the impact that’s being had in the real world? How do we help data scientists to work with the C Suite? We need a translation layer sufficiently experienced in both business and data science. 

We need a type of person that can take C Suite enthusiasm for data science and, using business experience, translate it into a roadmap of specific, achievable projects. This person will also hugely increase the productivity of data scientists by reducing how often they are pulled into projects that ultimately go nowhere. 

The trick is that the person needs to be sufficiently expert in both business and data science. I’ve seen horrible results when the person is too much of a data scientist and also when the person is too much of a business person. 

Finding the right person

Two good interview questions for someone in a role like this would be their view on the landscape of data science tools and techniques that can be applied to business problems and their view on the landscape of business questions that data science might reasonably be applied to. Without a sufficiently technical understanding of the complete landscape on both sides, the person will be unable to lay out a roadmap of opportunities that is likely to lead to success. Business opportunities will be missed and the toolkit available will be too narrow to be successful. Most consultants or managers I’ve seen struggle to apply data science in businesses would have been identified by shortcomings on one of both of these questions. 

Data science should have an impact that feels magical. All businesses should aspire to that. But there is lots of careful consideration required in between that expectation and delivery. To get started, your first hire should be a brilliant translation layer, not a brilliant data scientist.