Small (data) is beautiful

Companies are investing in big data projects with the expectation that aggregating data into one big central data repository will automatically allow them to find solutions to all of their business problems. They merge exiting easy-to-use data sets, maybe add new ones, and develop an entire taxonomy with limited access rights around this central data repository.

And now what? What do you do with this data? Will your data scientists dissect the data, but won’t know how relevant it is since they are disconnected from business operations? Or will your business analysts be trained on how to access the data first, but then will only scratch the surface since they may have confirmation bias focusing on data that they know and agree with and ignoring everything else?

Maybe it would be better to start small. Before embracing big investments in big data, it would be worthwhile to find out if your organization can handle small data. Do you have the analytical talent to examine and interrogate a small data set with let’s say half a million records, which can be even managed in a spreadsheet? Can your analysts find relationships across different dimensions and across time that may impact your company’s revenues and profits? Are they able to distinguish correlations from causations? Can they recognize patterns in the data? Are they unbiased enough to identify the unknown unknowns and create hypotheses for further testing? Are they skilled to apply basic statistics and advanced regression, simulation, clustering, and other techniques to test these hypotheses? And are they willing to reiterate all the steps if the analysis and modeling results don’t confirm their initial assumptions?

If you don’t have the talent to do all this, maybe big data is not for you … at least not until your analytics teams can fully understand (small) data and its implications on business results. Otherwise your big data project may be like teenage sex as written by The Register: “everyone's talking about it, only a few know how to do it, they all think everyone else is at it and so pretend they are too”.