Don't Settle For AI Which Only Does Half The Job

Would you hire a talented executive who can only do half the job they were hired to do? Certainly not. Consider the implications if you did...

In an office somewhere

CEO: Welcome to Big Business Corp. We are so excited to have you on board as our new executive in charge of Data Analysis - we can’t wait for you to get access to all of our data and to start making recommendations for how we can make measurable improvements in our business.

New Exec: Great to be here. When can I get started? I would love to read all of the company documents and then start making recommendations for operational improvements.?

CEO: That’s great! We will give you immediate access to all the documents, and our head of BI will be in touch to get you started on our data. We have invested a ton in our data infrastructure - it’s a goldmine!

New Exec: Well actually no need to introduce me to the head of BI - I am not very good at analyzing large and complex structured data. Just give me the documents and any reports your analysts write, I have a photographic memory, I’ll know everything very soon. Can’t wait to start driving changes in the business.

CEO: What do you mean? How do you expect to make useful changes which drive results without using our data to form a deep, up-to-date, understanding of how our business operates? If you want to drive measurable changes in our metrics you will need to be very familiar with what is currently driving our metrics.?

I think there is some kind of mistake. I don’t think this is going to work out.??

New Exec: I spent a fortune on a fancy education so I can handle an unlimited number of documents and rapidly synthesize them. Why not just try me out for a bit? You’d be amazed at how well I do at document summarization and creative writing.

CEO: ?......

/End

It is not reasonable to hire an executive responsible for data analysis?if they only have an expertise in textual data but aren't an expert in structured data analysis. If you want to make measurable improvements in your business metrics, a partial view of the business will result in ill-informed changes to the business and lackluster performance. The gems are hidden in the structured data - documents will only give a partial view and one which is biased based on the authors assumptions about which analyses are relevant.

This is well put by Edwards Deming in his famous quip:

“In God we trust. All others must bring data”.

If this is so obvious why did I write this? In this post I will look at the incredible potential for GenAI to help organizations drive real operational improvements and outline how LLMs are currently like the senior employee with only a partial ability to get the job done - something which prevents massive GenAI adoption for operational use-cases. Finally I'll cover SparkBeyond's approach to solving this.

The Promise & Limitations of GenAI

An LLM is an exceptional technology. Its superpower is to ingest and train on effectively unlimited text documents, synthesize them and then to have the ability to flexibly use the knowledge contained in them.?

Unfortunately despite LLM’s incredible capabilities with documents, when it comes to structured data, it is not able to ingest large amounts of structured data and to systematically extract the insights contained therein.?

See this article written by our Co-founders Sergey and Ron which reviews different ways to assist GenAI in overcoming these limitations and the relative merits of our Database Knowledge Mining Agent approach.

As they point out, it is not that LLMs are incapable of making some use of the knowledge contained in structured data. They can be fed reports (using RAG) describing the results of the analysis of data or even make hypotheses about what patterns there might be in the data and write code to query the data - a bit like a data scientist. However these approaches at best provide a partial view of a business, one that is only as good as the hypotheses formed and the corresponding answers generated by the LLM or BI team.

Which important insights would you be OK with your AI not knowing?

If the GenAI + structured data approach chosen by an organization only provides GenAI a partial view of the business (i.e. some subset of the knowledge contained in the structured data) this leads to questions which must be considered before using it to power operational decisions:

• How would you decide if the subset of knowledge the LLM has available is enough to base decisions on?
• How do you decide which important aspects of your business you would be OK if your LLM only discovers some of the knowledge in the data?
• And possibly most importantly - what if your competitors took a different more systematic approach?

What if your key competitor has built an AI stack which has a more complete view of the drivers of their key metrics (as opposed to the partial view your AI stack has)?

How will this impact your ability to compete with them?

For example, you might be using an LLM which is giving you generic answers as to how you might typically improve the profitability of your retail stores or how to understand the general drivers of engine failure for a manufacturer. What if your competitor has an LLM with detailed knowledge of the drivers of THEIR stores’ profitability (at the customer and product level) or a metrics-driven understanding of the top 10 ways to prevent failure of THEIR machinery... I know whose operation I would bet will likely improve faster in the coming years!

To make this more concrete let's review some example LLM chats which shows how an LLM with generic knowledge will find it difficult to compete with the value provided by an LLM with deep operational knowledge of a specific organization:

Now let's look at what happens if we empower the LLM with knowledge about what causes failure of a specific manufacturers engines:

The generic LLM understands the general problems experienced by engines but has nothing to say about the specific engine in question. The LLM which has access to the data about the performance of a manufacturer's actual engines can provide a far more useful answer.?

This can then be translated into actionable Root Cause Analysis grounded in data e.g.

The business potential of AI can be unlocked if we give an LLM FULL access to the knowledge locked up in both enterprise documents AND in operational data. Only with this understanding of the drivers of your business will GenAI be truly able to maximize its potential to drive measurable improvements in your business.?

We bring our partners the freedom not to compromise?

At SparkBeyond we are focused on unlocking the full knowledge in structured data for easy use by LLMs. No need to make the impossible decision as to which important knowledge your GenAI can manage without or to compromise on GenAI which doesn’t understand enough about your business to be given important tasks.?

Our Database Knowledge Mining Agent provides the missing link between the structured data and LLMs. It ensures that your LLM is constantly updated with the most important drivers of your key operational metrics, information which is encoded in the patterns of the precious structured data generated by your business.??

The result is a GenAI which deeply understands the drivers of success and failure in your business. As LLMs grow in their abilities, so do the number of situations in which you can apply their AI to help you solve business problems. Examples include:

• Cyber security e.g. Which alerts are likely to be attacks? Which transactions are likely to be fraudulent? What is the evidence for this? And what should I do to mitigate them?
• IOT e.g. Which devices are likely to fail and what can I do to prevent that happening? How can we improve energy efficiency of facilities or fleets? How can I use an LLM undertake effective RCA when problems occur?
• CRM e.g. What messages should I send and who should I send them to if I want to drive increased sales / reduce churn?
• BI dashboards e.g. What are the drivers of the trends I see in a chart? And what can I do to leverage/mitigate them?

We are actively partnering with Systems Integrators and Solutions Builders to build these GenAI + structured data solutions to drive improvements in their clients operational metrics. Feel free to reach out if you want to discuss how to bring this to your organization.