Data: Striking the Right Balance

(The following represents my personal views, based on years of actual experience, and does not represent the views of my employer or any groups I'm associated with).

If you’re working for a company managed by sentient individuals there’s little doubt that some people are either pushing forward with ideas about how to apply data science or already actively engaged with certain projects. There may even be a strong push to go hire “data scientists.” Much of this reminds me of the early internet boom, circa year 2000, when company executives wanted us to start working on web development even without articulating any actual strategy or clear business outcome. I can even recall some very savvy managers just telling us to start writing code and then we’ll figure it out later.

Well, if you are turning the data scientists loose on your organization without any good guidance or some articulation of actual business outcomes you’re probably headed for some frustration and another lesson learned. 

In the first place, the data scientists are probably going to have a degree of frustration in actually gathering together the data they need to undertake the analysis. It probably isn’t organized in a fashion that encourages analytical processing. Most likely, it was organized in a way to be efficient with transaction processing or fulfillment. This doesn’t mean the data are “bad”; rather that you’ve organized the data in a way different than the current needs to analyze it.

You may also be finding that once you locate the data the ability to “join” it for meaningful analysis becomes challenging. That usually happens when there isn’t a sound “data model” in place across the organization. For example, a product may not have the same name or code across systems. In some areas the ways a client may be referred to could be different depending upon the system or type of transaction. 

So, should we just despair and cancel the efforts? Probably not! What I’m just trying to highlight is that just can’t start cooking a meal before you properly source and prep the ingredients. Making progress with data science requires a commitment to do the “boring stuff” related to proper data management. 

You’ll find that by helping put the scientists in close working relationship with the people who handle the day-to-day tasks of processing data you should begin to understand the limitations of using your transactional data for analytical processes. The progress forward may not come quickly but it should for the basis for less frustration, misunderstanding and endless meetings about “bad data.” 

It somewhat reminds of watching an episode of “The Dog Whisperer”, where the great Cesar Milan reminds us that there are no “bad dogs” just some owners who need training.  It’s not about “bad data” and more likely about improper data management.