What is an insight?

This is my perspective and I make no claim that this is comprehensive, I am happy to hear yours and include them in any future articles I write.

A few weeks ago I and a couple of young folks were looking at results of a research exercise. The results were clear, consumers appeared to be have changed their life priorities. Financial security as an element appeared to be trending downwards and my instant question was why? The answer was swift, the economy was doing better, inflation was down, people were feeling good, why should they worry?

It was a reasonable explanation. But I was stubborn, I asked - Can you demonstrate this linkage using data? The answer was No, while it is true that there was strong correlation, previous instances of jumps in confidence did not lead to similar trends.

We went back to the data, one of the hypothesis was that there appeared to be more affluent respondents in the research. So we looked at the data and started working backwards, eventually we picked up the following trends -

Over the same period of time, it appears the number of multiple earners had gone up in an household, improving financial security significantly

• This is because while India's overall female work participation was trending downwards, Urban India was picking up slowly (also leading to marriage at a later age for both men and women), this was an ongoing secular trend
• More married folks were staying with their parents than ever before given significant increase in rental prices while their parent's home was available at a much lower 'cost/convenience', this was counter to the previous nuclear family trend (notice the joint family adverts of patanjali)
• More female work participation plus the gradual change in societal attitudes meant more people were single than ever before for a longer time leading to more people staying with their parents for a longer time
• All of this was happening at the time of a population bulge (also called demographic dividend) which means that the growth of these segments was picking up at a faster pace than before.

Essentially, previously secular trends of female empowerment, late marriages and seeking independence was now coinciding with higher rental prices pushing more of the research participants to stay with their parents than ever before. We dug deeper and found more trends but really that is not the aim of the larger point I am making.

What appeared to be a simplistic insight of change in government leading to low inflation leading to better confidence may not be so simplistic at all. There may be a variety of factors (as they often are) that help determine why consumer/voter trends move in a certain fashion.

Simplistic observations often come from lazy assumptions

The correlation causality problems donot start with a high correlation coefficient. They start with previously designed rules in our head (if there is an observation from the research, the laziest rule is check for demographics and geographies, in politics, it is caste and religion). Given time pressures, peddling these rules is an easier job than meticulously finding the 'truth'.

One of the best ways to get to insights is to generate 'hypotheses'. In simpler words, possible reasons for an observation in the research. In the previous example, the researchers could have laid out a huge list of reasons why financial security may have become lesser of an issue (also by the way visible in higher mutual fund consumption) - More wealth, more confidence, change in environment, exposure to financial advisory and so on. A minimum of 7-10 hypotheses would do. You then bear upon all possible data sources to understand the linkage. You establish the linkage by utilising a variety of statistical tools in your hand. And then you come up with an insight (a shallow example here)

Higher rental prices plus slowing entry level salary growth is leading to more Indians (Married or unmarried) staying back with their parents at a faster pace then ever before leading to reduced sense of financial risk amongst Indians

The client knows two things now - One, my product portfolio will have to tie in with an increased ability to take risk and two, I have to be wary of changing property market trends and income trends, any reversal could push more of their customers to buy property exposing them to debt and reduce their willingness to take risk (possibly). Of course, the research has to provide insights on which product portfolio and why and insights around that.

So what is an insight?

a. It gives us a sense of completion and the confidence to take decisions

b. It is not an observation but a detailed path of finding root causes of root causes and Often it has been arrived at after eliminating multiple hypotheses

c. It is often supported by more than one data sources

d. It provides joy (emotional high) to those who find it, often it is new (but not always)

The mindset of anyone working on data is no different from a murder investigator. There is an observation, and there could be multiple reasons for that observation. Your job is to consider all possibilities and meticulously arrive at the reasons for the observation. So, if X has been murdered, you could probably say Y killed X (similar to saying India is not doing well because UP is not doing well) but then that is not insight. You will have to then say how and why Y killed X with evidence. (It is like saying UP is not doing well due to poor investments in healthcare leading to higher fertility rates leading to lower education standards and higher distribution of farm income lead to lower growth (and) lower per capita)

Often the conflict is between observation and insight and sooner one understands the difference the better off one is. Finding an insight is an emotional journey, you can say, a smaller version of the sea farers of the 15th century. You want to reach India (insight in our case) by smartly and meticulously plotting your way and reaching your destination. You will have to be determined against all odds (often laziness and/or time pressure) and temptations (simplistic answers), at the end only the persistent ones will win, remember -

Insights are a lead indicator of revenues