Inference vs. Insight // Inference and Insight

As a species, our intelligence is the superpower that makes us resilient, progressive, and intuitive. The rise of data and analytics has fed into our Intelligence dimension and at the same time it has questioned our intuitiveness. Given the pace at which the world is moving, the data is becoming overwhelming, it is but natural for us to augment it with analytics.

As I reflected on the future of analytics, a core chord kept playing in my mind - inference vs. insight. Both do co-exist and are needed. I believe “insight” is more natural ( there and yet not there)? while inference is man-made, needed to make sense of the world around us on a day-to-day basis. Let me explain this a bit, let’s start with my definitions.

Here is how I define them:

Inference: Is drawing logical conclusions from data based on observations. This is essentially the bread and butter of all analytics functions across the world. This is what we do when we share a report, when we talk about a trend, build out business intelligence dashboards, and send out data-led notifications. Inference is very important dimension, and is needed in a data hungry world - to run businesses efficiently and effectively.

Another Definition:

Inferences are steps in reasoning, moving from premises to logical consequences;?

Ref: https://en.wikipedia.org/wiki/Inference

Insight: Is deriving deeper understanding from data that can drive significant actionable change. This requires an interpretation that goes beyond the inference in front of you. For example, I can draw a 2X2 matrix of price band by sales band and place all brands on top to see if there is a white space opportunity here (inference), however only with some external data - maybe from industry reports or market research data or open-ended responses, I can answer the Why? And decide on action.

Another Definition:

A piece of information which is both obvious, but unknown. Self-evidently true, but never spoken or thought of before. - by Alex M H Smith

Ref: https://basicarts.org/a-definition-of-insight-thats-actually-useful/

To me, the difference between an inference and an insight can be looked at from 4 perspectives:

The Science: Inference relies heavily on core statistics, possibly machine learning, while Insight incorporates domain expertise, human intuition, and qualitative unstructured data inputs.

The takeaway: Inference deals with What is happening, while Insights deals with why and what needs to be done.

The Outcome: Inference identifies patterns, helps in operational decisions, and keeps the lights on, while Insights drive strategy & innovation.?

The Timeline: Inference has a short life span, its action is needed now. Insights have a long life span, these usually don’t change that frequently.

How have businesses adapted the two:

Although businesses have been aware of the need for both inference and insights, it is fascinating how businesses have embraced this adoption and its impact on the analytics industry.?

Two/three decades back marketing research was the kingpin of anything insight. There was hardly any relevant internal data. Inference was drawn directly from survey data and insights from agency frameworks.?

From the early 2000s, the focus shifted towards modern data warehousing (DWing had its evolution before that), working on the business value chain and ensuring we had the systems to capture relevant data at all points. Business operations started moving towards inference-led decisions as opposed to human dependency. This decade saw a rise of statisticians and econometricians.

From 2010 onwards analytics came full circle and as systems matured, all aspects of analytics from descriptive to predictive and prescriptive became core. Given the data glut, inference became the norm as there was too much to do for running the business. This was the rise of the data engineer and the data scientist. The art of insight took a back seat, to be honest. It still needed the domain expertise and the sharpness of an individual to connect the dots across inference, experience, unstructured data, and business environment.

Now and Future:

And then came the 2020s, here and now, where the data load for any business is so high that without AI it is impossible to navigate this mesh. Inference is still the core as businesses need to keep their operations optimized, however, I see them becoming fairly automated. Possibly the death of dashboards as we know them, which only means another evolution is coming!

What happens to insights?

I believe there are multiple pathways that are emerging -

1/ LLMs have broken ground with a true unstructured data engine. Possibly the first time we can design systems that can get the context right.

2/ Organizations will need to build/embrace/latch onto platforms and products to ensure that they are able to navigate the data flow and get both inferences and insights. Newer consumption options are just around the corner.

3/ Democratising inference and linking the dots will be at the core. - Fosfor

4/ Rapid on-demand input for external validation will become the norm. - ADNA (Audience DNA)

5/Generation engines will emerge that will create content learning from both structured and unstructured data of an organization. - Yarnit

5/ Deep linked inference systems will emerge, as for the first time an organizational knowledge framework, external knowledge base, and internal data will be talking to each other seamlessly.

This last play is important and I look forward to its evolution.

There is no doubt that we will end up in a world of agents. A lot of business actions will not wait for inference but have agents with context, cognition, and decision-making ability. Agents will communicate with each other - through inference, and create networks.

Will these evolve and transform to create the ultimate Agent - The Insight?