Small data
Chandramouli Gopalakrishnan, Ph.D
Cooking something interesting. Ex : Product Leader @ Microsoft | Leadership at the intersection of People, Product, and Engineering | Enterprise, Startups
We keep hearing about Big data everywhere - sometimes in places where we do not even expect to here it. I was listening to the latest Trailblazers podcast from Walter Isaacson, and he just casually dropped this nugget.
Small data is the capturing of the small, subtle nuances of a customer. A lot of times, these small seemingly insignificant subtle pieces of information lead to huge product insights.
While extracting big data, and distilling data out of it is important - it is still generalisation. It is amortised data. It is a collective. It is very important for PMs to observe customers at close quarters on a regular basis.
In my three years of observation, being a product leader, I have seen that insights distilled from big data, can only result in incremental improvements.
To get 10X improvements, we need to observe and incorporate these behavioural, often times visual, subtle insights. These are few in number - viz small data.
Update: Textbook definition of small data (according to wikipedia) is data that can be handled by people (as opposed to big data for machines).
Software Developer
6 年Thanks for the post. It is true that the world is riddled with Big Data but how much of that is actionable begs question. Statistics as a subject had started in the 1920s in India looking at data in small patches of agricultural land to enhance cultivation yield. In todays business world also we are trying to enhance customer yields. So some of the problems that is there in today's data which makes it big than yesterday's small data is not the amount of data but the number of parameters that one could be tying to estimate together. It was clearly proven in the 1960s and 1970s that we could get better estimation in multiparameter estimation problem than some of the techniques referred in the Frequestist Framework of Fisher, LeCam , Rao, Roy and others. So the summary is there are gems in small data (volume) which are part of a larger big data (volume) , from which multiparameter inference (a problem of big data) can yield very useful insight, but the extraction of such data sets is sometimes a major statistical challenge. We do not do that job very well.