One of the best ideas for AI application - Case Study

Which use cases are good for AI?

That's a question that echoes in many business leaders' meetings.

There is no silver bullet, but the highest chance for yours being one of them if it's:

- an internal use case

- coming from a problem statement

- benchmarked

- from a battle-tested area of AI

So that's exactly what we've been working on lately, and we decided to wrap it up with a nice case study of an AI project's early phase.

The business

The client we've partnered with reached out to address a pain point. They're a major Swiss producer of specialized high-precision chemical analysis instruments.

As a quality-first organization, they offer years-long technical support and maintenance of their devices. And here comes a challenge.

The knowledge base used by field technicians is quite overwhelming. They have many information sources, such as:

- manuals

- instructions

- historical work orders

Thousands of pieces for hundreds of devices.

Accessing a distributed database of such size to find relevant information about a specific troubleshooting procedure for a specific device model was difficult. Adding to it a worldwide maintenance service network, as all the fixes has to be done on-site, it raised to a rank of a problem.

So, they researched the case and found a possible solution: an AI assistant that would access a knowledge base and present information in a user-friendly manner.

We thought it was a perfect case for a RAG system. We pitched our idea of how such a system might work to the client and soon started working on the project's first phase.

UX and working out the case

The goal of the first phase was to give the client enough data to decide whether the project should be scaled and brought to production or if they should look for another solution.

We've decided to run a series of workshops to learn as much as we can about the business background, the users, and the requirements.

We've divided them into:

• in-depth expert interviews to understand the business and user perspective for technical problem formulation and requirements mapping
• technical interviews to get an overview of the client's IT infrastructure
• data interviews to learn more about data types, data sources and corresponding end-user processes

In the end, we delivered a set of documents:

• project scope focused on delivering core functionalities
• user path and data pipelines
• system's architecture overview, including integrations with the client's other systems
• a detailed system development roadmap

All of these were complimented with an extended PoC of the solution so the stakeholders could see how the system would work.

AI assistant

Here, I'm about to talk about the project and give a few insights into how building an AI-based system works.

Let's start with the technique of choice.

Fine-tuning vs RAG - thought process

These two approaches are used alternately, which is incorrect. They should be used according to how we want to develop the application, each with pros and cons.

Each will also mean that different frameworks are used and are related to another area of an AI-based system.

The best visualization is the following:

Prompt engineering is the first and the most basic step. You can do surprisingly lot with just prompts (think of all your interactions with ChatGPT), but in 99% of the cases, a production-ready system will be much more complex to rely just on prompts.

Another thing is that the prompt-only approach is hard to scale due to the context window, and using huge prompts may be costly.

However, it's great for the early phases, such as PoCs/pilots. Interacting with such a system will give us many cues on what to do next.

Now, it's good to use a simple visualization to understand the optimization flow.

Think of learning for an exam.

The prompting phase describes the model that there will be an exam it has to prepare for. It details the subject matter overview, present with a task, approach, and the way to think about questions and formulate answers.

Fine-tuning the model is the process of learning the methodology and approaching various problems it will encounter during the test.

RAG is a book it can use.

The best results are achieved by combining all three approaches, but it's costly and pretty complex. Also it's not always necessary.

Our flow

Here's how we did the groundwork for the PoC of our solution:

1. We knew there would be many domain-specific documents the model would have to process, so we put our bets on RAG. We also knew we'd have to develop and nuance this approach in the next phases, so it was a grateful choice for early testing.

2. We've also focused on two major challenges. First, the database of documents would have to be dynamic - some documents will be added, some others removed. Second, as the model hallucination is a significant issue, embedding documents seemed the safest approach.

3. We were still in the early phase, so we decided to use only system prompts to control model behavior, such as output format, quotes, and linking relevant sources.

We relied on a tried and tested R&D approach (and that's a good way to look at AI projects) - test, evaluate, improve, test, evaluate, improve...

In the end, we achieved very satisfying results. Our development team has thoroughly tested the demo solution, looking for clues and signals of where the model's performance could be improved from a technical standpoint. The client tested the system from a user perspective, focusing on the relevance and usability of the outputs.

How the system works

There are three important stages in using AI Assistant:

1. Uploading document data and converting them to text (PDF documents are converted to images and described by the LLM)

2. Embedding text and storing it in a database along with metadata

3. Interaction with the chatbot

Here's how an interaction with a chatbot looks like:

1. The user sends a query (e.g. 'I have an issue with <device, model number>. What I see is <issue description> and need troubleshooting assistance')

2. The model interprets the query and reaches the embedding database to identify the most relevant documents

3. The model:

• chooses the best documents,
• compares them,
• narrows the list down to the three most relevant,
• synthesizes an answer
• outputs an instruction and links corresponding documents/pages from the knowledge base

Prompt engineering

System prompts are used mainly to describe the problem context to the LLM - what's the model role and how the problem should be approached, which questions should be answered and which not. We've also emphasized that the model should not generalize and should be true to the retrieved knowledge base. If it cannot find relevant information, it should say so and ask for more details.

What works well in the PoC phase

1. PDF conversion, despite being multi-stage (PDF -> image -> text -> embed) is done very well.
2. The model is also good with tables.
3. It describes images accurately, recognizing objects, their purpose, and placement.
4. It works well even with longer descriptions on 2-3 pages of a document.

Identified problems

1. The model has some issues with complex diagrams as they're generally hard to describe in plain English.
2. Basic chunking is insufficient for instructions that cover an additional, small piece of another page.
3. There are some troubles with very long instructions - prompt engineering works in about 50% of tested cases.
4. There are optimization issues. We'll definitely introduce prompt caching so the model doesn't have to run the whole process each time.

Summary

First, the client was very happy with the first phase and the demo that was delivered. We're scaling the system now.

Now, the discovery phase (or an early phase / introduction phase / testing phase / you name it) is crucial in AI. It's one of these unique areas where a project might be doomed from the start, and the goal is to determine it as soon as possible.

A the same time, if a project seems very doable and the results of the discovery are promising, the deep analysis resulting in a set of documents benefits the business by:

• backing up the case with strong evidence (so it can be win internally)
• gathering stakeholders and convincing them to join (as there's a high chance of scaling)
• mitigating risks by pointing them out early
• showing a path to success (that is a production-ready system)

If you have a similar project in mind and want to test it, we've formalized the discovery phase in the AI Navigator approach -> https://ai.sparkbit.pl/ai-navigator. You get all the benefits with no risk. It's just you, the documents, and the decision.

Author: Kornel Kania , AI Delivery Consultant at Sparkbit