Part 3: Enterprise LLMs need to distinguish correlation from causation.

In the previous article, we saw how out-of-the-box LLM models can hallucinate. Lack of context can lead to false positives and impair professional credibility.? A key role of a semantic layer is to provide this context.

Let's pick up where we left off. Back to the drawing board, Amy, our rockstar Product Marketer, spends a few hours updating and testing her LLM prompts, to ignore terms like "security review" and "Infosec review". She now finds that the "security and compliance" use case was legitimately brought up by reps in 2 of 5 deals (40%, down from her earlier 100% estimate.)? Excited, she reports back to management that the use case has still materially influenced Acme Co's pipeline. She recommends demand generation prioritize this use case for a campaign.??

Unfortunately, Acme's RevOps leader asked the dreaded correlation vs. causation question. Just because the use case was brought up by Reps, was it really a priority for the customer, compared to the 20+ use cases in Acme's playbook that Reps were also trained to bring up???

Ouch! Back to the drawing board again.?This time, Amy needs to train the LLM model for Acme's 20+ use cases!? Not just that, she needs to develop her own model to predict the importance (causality) of each use case, deal by deal.?

The RevOps leader suggested the following causality features for the model:

(a) did the buyer talk about the use case, or just the rep,

(b) did the use case come up in the early CRM sales stages, or repeatedly in the mid-to-late stages of the deal as well, and

(c) how in depth was the use case discussion.?

Can't just write a prompt for all that!?This is going to take resources.?Luckily, the Board is pushing for A.I. initiatives, so Amy gets a business analyst, LLM engineer, and data warehouse specialist.???

Couple months later, we have it!??Her model is able to stack rank use cases per deal. She re-ran her analysis and identified deals where "Security and Compliance" was a top-3 use case.?Amy's original estimate of 40% pipeline influenced, based on "mentions" alone, came down to just 5%, after inferring causality.?

Dang it.?This isn't such a hot use case anymore.??

Takeaway #2:?Enterprise workflows are all about outcomes. To inform critical decisions, LLM models need to interpret correlation versus causation, keeping outcomes in mind. That's another key reason why we need a semantic later.

Questions to ask yourself:

1. What causality will executives probe for in your analyses???
2. How would you design, implement and test causality features??What data will be required and how will you access it? What will be the time/resource cost?
3. Would the causality be transparent to users??Can they inspect it?
4. Will the same model work for different situations - e.g., causality by Industry, segment, geography, competitor, deal size, new logo vs expansion
5. Can ordinary users access and update these models, e.g., add or retire use cases, change CRM stage thresholds, etc. Will changes be retroactive?

Next up:?Now that 'Security and Compliance' wasn't the best pick for a revenue-maximizing campaign, let's explore how a semantic layer can help test hypotheses, and make actionable recommendations. For that, my next post outlines the key building blocks that need to be in place.

Follow me and? Tribyl, Inc. ?for more insights on leveraging LLMs to unlock revenue growth in a reliable manner.? Look forward to learning from your experiences and feedback as well!