Causality: The Missing Ingredient

Today’s #AI capabilities are very impressive. But just wait until one of the most important MISSING INGREDIENTS gets added into the mix. This ingredient is what will make AI a truly indispensable partner in business.

?The missing ingredient is CAUSALITY.

Causality will enable businesses to do more than just create predictions, generate content, identify patterns, and isolate anomalies. They'll be able to also so play out countless scenarios to understand the outcomes of various actions, explain causal business drivers and collaboratively problem solve. They’ll know WHAT to do, HOW to do it and WHY certain actions are better than others – collectively being able to prescriptively shape future outcomes. ??

#CausalAI is the next frontier on the journey to AI. On the 2024 Gartner Emerging Technologies Hype curve, its time-to-impact has been shorted to 2-3 years. It also tops the list of new technologies that businesses plan to adopt. While only 1 in 10 are using causal inferencing today, another 5 in 10 plan. Overall, the causal AI marketplace is projected to grow at a whopping 41% CAGR through 2030, nearly 2x traditional AI.?

So, why is this such a big deal? ?Well, Apostolos Lymperis, the chief strategy officer at causaLens, framed it in an intriguing manner:

This analogy to the human brain is right on!? Simply put, humans are causal by nature, so AI also needs to become causal by nature.

And, in turn, create a more collaborative partnership between human and machine intelligence.?As such, this article on Causal AI will explore:

• The Basics
• The Ingredients
• The Use Cases

Finally, I’ll outline some CALLS TO ACTION for those interested in getting more involved, including how to get hands-on experience and a certification.

Read on and let me know what you think!

The Basics

The methods which AI reasons is a function of its inferencing design – that is, how it progresses from a premise to logical consequences to judgements that are considered true based on other judgements known to be true. In essence, it must determine WHAT will happen, HOW it will happen and WHY it knows certain actions are better than others. This level of progressive intelligence can only occur by fusing AI’s correlative powers with causality.

Let’s dive deeper, while building on the analogy to the human brain:

• The WHAT – Today’s AI, including LLMs / GenAI, derive their powers from correlating variables across datasets, telling us how much one changes when others change. They rely on observations learned from past data and lack any algorithmic understanding of causality, beyond causal relationships explicitly captured within those observations. Simply put, LLMs are trained to identify patterns, anomalies and observations as their method for predicting or creating outcomes. LLMs are like the limbic brain which drives instinctive actions from memory, making them good at automating tasks and creating content.
• The HOW – Beyond predicting or generating an outcome (the “what”), businesses also need to understand and explain how the outcome was produced. This requires advanced transformations that integrate correlative patterns, influential factors, neural paths, and causal relationships to de-code the “how”. ?Think of this as playing the role of the cerebral cortex that encodes explicit memories (the ‘what”) into skills and tacit know-how. This is key to recommending prescriptive action paths that are trusted, transparent and explainable.
• The WHY – For AI to truly reason and problem solve, it must understand the dynamics of why things happen, what can be done to change things, and the consequences of interventions. This requires a shift from today's forward-progressive inferencing to bi-directional inferencing as a mechanism for exploring the outcomes of various “what-if” propositions. These capabilities mimic the neocortex which drives higher order reasoning such as decision-making and planning and perception. For AI to truly help problem solve, these capabilities are a must.

While there may be different degrees of implementation, causal AI is realized when it can deliver all the capabilities above in an integrated manner? - knowing the what, the how and the why of business-critical problems.?

The ingredients

No matter how sophisticated a predictive model is, it still only establishes a correlation between a behavior or event with an outcome. But that is different than saying that the outcome happened because of the behavior or event.? There can be correlation but not causation, and causation but not correlation. To make good decisions you need to understanding root causes, as equating correlation with causation creates an incubator for hallucinations and bias.

Casual AI can identify precise cause & effect relationships, and thus root causes. This is critical to problem solving, as knowing what caused something is the key to knowing what could be done differently to improve an outcome. Also, the pathways through today’s neural networks are black boxes. They don't tell you how variables interact, nor their values, nor how they influenced an outcome. So why should you trust it?? How do you explain it??

Causal AI excels at uncovering the causal pathways to an outcome, and can do it across multi-dimensional, randomized spaces with billions of parameters. It can infer the relationship between ALL the variables in a dataset, which variables influence each other and the outcome, and the extend of those influences. This allows data-driven discovery of complex causal relationships that are ranked in terms of influence. In turn, the ability to truly reason.

Casual AI will deliver an array of new “ingredients” (or tools) that will truly make AI an indispensable partner in business, including:

• Interventions: model the impact of alternate actions, strategies or conditions on an outcomes (“what-if” scenarios).
• Counterfactuals: create hypothetical alternatives to the current or past factual state and/or underlying assumptions to see impact on outcomes.
• Confounders: identify irrelevant or misleading considerations while also uncovering previously hidden causal influences on an outcome.
• Prescriptions: create recommendations composed of interrelated actions (pathways) that will result in an optimal outcome.
• Explanations: gain trust and audit-ability by summarizing how and why certain actions were taken or are the best ones, with full transparency.
• Interrogations: infuse expert human knowledge, policy, and/or constraints into a model to reflect tacit or practical conditions.
• Scoring: optimize impact of action planning on metrics (KPIs) by quantitatively ranking influential factors across scenarios.

By adding these new “ingredients” into the mix of today’s generative AI models, businesses will gain the ability to play out countless scenarios, understand how actions impact the business, and confidently pursue the actions to optimal outcomes. Furthermore, they break open the “black box” of today’s AI models to gain fully explainability and transparency which is critical to establishing trust, eliminating bias, and managing compliance mandates.

In the end, the seven key features of causal AI will finally enable humans and machine to collaboratively reason to solve complex challenges, progressing from descriptive --> to predictive --> to prescriptive decision-making.

The Use Cases

The use cases for business are truly limitless, as “cause & effect” is the foundation of decision making, process automation and learning. Understanding why things happen and why future things are likely to happen given different paths forward is an invaluable addition to today’s descriptive and predictive AI capabilities.

To illustrate, here are some real-world examples of causal questions that AI can now help answer, and that today's #GenAI models would struggle with:

• Marketing: what is the best mix of tactics, spend, target cohorts, offers and lead scoring criteria to improve lead generation KPIs?
• Customer Churn: why did we start losing customers and what can we qualitatively and quantitatively do to reverse this trend?
• Pricing: how do we optimize the balance of profit, revenue, and customer acquisition KPIs with an elastic pricing strategy?
• Supply chain: what options do we have to minimize delays in our supply chain for each season of the year, and by how much?
• Manufacturing: is there a missing link between product failures and our inventory sourcing and management policies?
• Legal : which trial strategy would will be most legally sound given different mixes of cited case law, and why?
• Financial: what are the downstream effects of a 1% fed rate cut, to economic KPIs to S&P 500 retail stocks?
• HR Talent: what would have happened to revenue if I staffed 23 salespeople instead of 13 marketing and 10 support?
• Regulatory: can you produce a detailed, data-driven compliance report that explains in why we invested in this retirement fund?

And, to bring it all together, an example of solving a challenge in retail:

CALLS TO ACTION

?

Contact me via LinkedIn or at [email protected].

?I can help you:

(1)??? Navigate the world of causal AI

(2)??? Create a use case strategy

(3)??? Gain hands-on experience

(4)??? Earn a certification in causal AI

(5)??? Connect with industry experts

(6) Source AI talent and tools

Thank you for reading - Scott Hebner

Sources:

Causal AI: The Next Step in Effective Business for AI

The Case for Causal AI (Stanford University)

Causal AI Marketplace (Markets & Markets)

The Plan Truth about AI (Databricks)

Hype Curve for Emerging Technologies (Gartner Group)