Data Philosophy – The missing piece of your data practice

For years now, we have seen that data science has had an extremely high failure rate. Typically reports say that data science has a failure rate of about 80-90%, meaning that at best, 20% of data science projects actually achieve any kind of positive impact for the business. That doesn’t even cover the cost aspect of data science which is going up in cost every year. The issue is that data science has an extremely large blind spot to what could have made it a success. Most practitioners of data science have little to no knowledge of philosophy and its importance to understand the data of which data scientist are using.

Welcome to the dark ages of knowledge when it comes to data. Frankly there is no other way to describe it. We have self-proclaimed data scientist running around acting like doctors during the dark ages. Business executives turn to their data science team expecting them to have answers, only to find the answers they get are akin to the advice a doctor gave in the dark ages. Your algorithm didn’t produce what you wanted? More automation of human tasks is needed, is often an answer given. That’s like the dark ages practice of cannibalism for headaches. It doesn’t work in a lot of cases. Or more tools, which tend to be expensive to solve a timing issue. It sounds a lot like the idea that a woman’s womb wandered around her body, which was a common medical belief. Like then as now, the beliefs are wrong. And like then as now, the answers are actually very simple.

Philosophy has been with humans probably since the dawn of civilization and it still holds an important place in our world today as it did in the past. Data philosophy is a concept that gets thrown out of many data science teams because frankly, it’s not taught to anyone. Believe me, I looked. There are a few bloggers which you can count on your fingers who mention the idea. But there isn’t a real practice around data philosophy. Which is a real shame because once you start to explore it, you start to see the real advantages of having it be a part of your data practice. 

We have data science, data engineering, data strategy, data governance, all in an effort to make sense of our data. Yet much like the idea of living in a cave an only seeing the shadows, most data teams are still in that cave. I’m not trying to be rude; I was there myself and to some extent, I probably still am. I am just aware that philosophy has been with us a long time and has a lot of the answers we are looking for to improve the success rate of our data science teams. 

Data philosophy at its core solves a really big problem we are all dealing with today. That issue is, the inability to think deeply anymore. We were sold the idea that technology would give us more time by solving problems for us, it didn’t. It made many work 24 hours a day as they were always connected. The cost of that is often our ability to think deeply about anything. One of the top skills going into 2020 is the ability to focus. In other words, the ability to think deeply again. When you have a problem, the temptation to go online and look for an answer is very easy. What that costs you is the ability to think deeply about an issue. To really analyze it. We have systems and processes in place that show this lack of deep thought. Product management 10+ years ago was in many companies the CEO of a product. Today we call them product owners and they are little more than glorified backlog groomers. When I was a product manager at GE, you needed 10+ years of experience before they would talk to you. Now, as a product owner, you can get a high school kid to do that job. The lack of deep thought created that. It is reactionary in the worst way.

How would philosophy solve this problem? Three years ago I started looking into data and philosophy and realized, many of the answers we are looking for, can be explained in philosophical concepts, already created! You don’t need some new multi-million-dollar tool, you don’t need the cloud and you don’t need an army of data scientists. What you often need is an understanding of the concepts that created the conditions you are trying to solve. That’s the advantage philosophy brings to the table. How do you understand and engage with influencers?  That’s the human network effect. Why is experience such an advantage on your team?  That’s expert intuition. Why do people go into Target to buy one thing and walk out with ten? That’s the Diderot Effect. Why is Hong Kong protesting? They are in an atemporal period. Why is mass marketing less effective today? Society is fragmented into Heterotopias. Philosophy gets you back to thinking and applying logic and using well developed concepts that allow you to unplug and really get to the hard of the issue. At the end of the day, why have a data practice? You want to remove uncertainty and you want to solve problems. Does a formulaic practice with reactionary decision-making sound like a good idea? Or is it better to truly apply deep thought and come up with a unique impactful solution? You liked option one, you are saying your business isn’t unique and good luck to you. 

The concepts are already there, you just have to become aware of them and the answers to your data problems become very simple to solve. It took me years to assemble a list of philosophy concepts that matter to data. I really wish someone was teach this because once you start to pair these concepts to the world of data, not only does solving data problems become easier, they are also more cost effective. Mainly because it brings us back to deep thinking and away from reactionary thinking which is what drive the data science space today. Unlike a number of people, I didn’t get into data science because it was the hottest field, I got in before that crazy. I spent months finding a solution to a problem and created a solution, I practiced deep thinking. Then as people saw my success and others, they jumped in. Back in 2017, we saw this same thing going on with deep learning, just another crazy that few thought about. Is that how you really want to run your business? Reacting instead of really leading? Philosophy is going to help you get out of that loss column and into the winner’s column when it comes to your data practice.