The Failure of Data Science and the Role of Philosophy

In recent years, data science has suffered from a remarkably high failure rate, with reports indicating that approximately 80-90% of data science projects fall short of achieving any meaningful impact for businesses. This issue is further compounded by the escalating costs associated with data science endeavors. The underlying problem lies in the significant blind spot that data science possesses, preventing it from realizing its full potential for success. The majority of data science practitioners lack a fundamental understanding of philosophy and its crucial role in comprehending the data they work with.

We find ourselves in a knowledge void, reminiscent of the dark ages, where data is concerned. It is akin to witnessing self-proclaimed data scientists who resemble doctors from that bygone era. Business executives turn to their data science teams in search of answers, only to receive advice comparable to the dubious medical counsel of the dark ages. When algorithms fail to produce the desired outcomes, the suggested solution often revolves around increased automation of human tasks—an approach akin to the misguided practice of cannibalism as a headache remedy. Such remedies seldom prove effective. Alternatively, organizations may opt to invest in more tools, often at great expense, to address timing issues. This approach bears a striking resemblance to the belief that a woman's womb wandered within her body, a widely held medical fallacy of the time. Just as then, the prevailing beliefs in data science are misguided, and the solutions are, in fact, quite simple.

Philosophy has been an integral part of human civilization since its inception and continues to hold significant relevance in our world today. Unfortunately, the concept of data philosophy is frequently overlooked within data science teams, as it remains largely absent from educational curricula. Although a handful of bloggers touch upon the idea, a genuine practice of data philosophy has yet to take root. This oversight is regrettable since delving into this discipline reveals its remarkable advantages in enhancing the success rate of data science teams.

In our pursuit of understanding data, we have established various disciplines such as data science, data engineering, data strategy, and data governance. Yet, akin to dwelling in a cave and perceiving only shadows, most data teams remain confined within that metaphorical cave. My intention is not to be dismissive; I have personally been in that position, and to some extent, I may still be. However, I am acutely aware that philosophy has accompanied humanity for an extensive period, holding many of the answers we seek to improve the success rate of our data science endeavors.

At its core, data philosophy addresses a critical problem we face today—the loss of deep thinking. Technology was marketed as a means to save us time by solving problems on our behalf, but it failed to deliver. Instead, many find themselves working around the clock, constantly connected. The price we pay for this perpetual connectivity is often our capacity for deep thought. The ability to focus became one of the most sought-after skills leading into 2020, emphasizing the need to regain our capacity for deep thinking. When faced with a problem, the temptation to seek immediate online solutions is pervasive, but it comes at the cost of genuine contemplation and profound analysis. The absence of deep thought is evident in the systems and processes we employ. Product management, for instance, once held the role of a product's CEO in many companies. Today, product owners are often reduced to mere backlog groomers. Previously, at GE, one needed over a decade of experience to become a product manager. Now, a high school student can handle the responsibilities of a product owner. This decline in deep thought has been the catalyst for such reactive practices.

How can philosophy address this issue? Three years ago, I embarked on a journey to explore the intersection of data and philosophy, and I discovered that many of the answers we seek are encapsulated within existing philosophical concepts. We do not require new multimillion-dollar tools, cloud infrastructure, or armies of data scientists. What we truly need is an understanding of the concepts that underpin the challenges we aim to solve. This is where philosophy excels. How can we comprehend and engage with influencers? The answer lies in the human network effect. Why does experience offer a significant advantage to your team? It stems from expert intuition. Why do shoppers enter Target with the intention of purchasing one item but leave with ten? This phenomenon is known as the Diderot Effect. Why are the people of Hong Kong protesting? They find themselves in an atemporal period. Why is traditional mass marketing less effective nowadays? Society has fragmented into Heterotopias. Philosophy guides us back to logical thinking, enabling us to apply well-established concepts and detach ourselves from the reactionary thinking prevalent in the field of data science today.

Ultimately, why do we establish a data practice? We seek to eliminate uncertainty and solve problems. Does adopting a formulaic approach accompanied by reactionary decision-making sound like a viable strategy? Or would it be wiser to engage in deep thought and craft unique and impactful solutions? If you prefer the former, you are essentially asserting that your business is not unique—a viewpoint that may prove detrimental.

The concepts we require already exist; we need only familiarize ourselves with them to find simple solutions to our data-related challenges. It took me years to compile a list of philosophical concepts relevant to the realm of data. I earnestly wish that someone had taught me these concepts, as connecting them to the world of data not only facilitates easier problem-solving but also proves more cost-effective. This approach brings us back to the realm of deep thinking, liberating us from the constraints of reactionary thought processes that dominate the field of data science. Reactivity should not supersede genuine leadership. Philosophy can guide you out of the realm of failures and into the realm of triumph within your data practice.