An introduction to the components of an AI chat application Part 2

This is the second part of a two part series where I provide an introduction to the components of an AI chat application / agent.? The first article covered topics including zero-shot / single-shot prompts, prompt template / moderation and Retrieval Augmented Generation (RAG).? This article will introduce the concept of tools that can be used to provide additional information to chatbots.

Tools

How is a Large Language Model (LLM) able to provide the current weather around the world if it takes 1-6 months to train a model?? The answer is through the use of integrations commonly called Tools.? Similar to RAG, tools are a way for an LLM to access external information and then incorporate information into the response.

When an LLM receives a prompt that it is not able to respond to such as the current weather, it reaches out to the tool to try and get the needed information.? Through APIs, the LLM would connect to a service like OpenWeatherMap that returns the current weather and forecast for cities around the world?

?

Multiple Tools

By using tools, the LLM is able to access real time information that was not included in the original training data.? Of course this concept can be extended to include multiple tools.? Some common tools include financial information like current stock prices, Wikipedia, calculator and web searches as shown below:

Tool Selection

With so many tools available, how does the LLM determine which one to use? Initially, the LLM attempts to answer the question by itself. If it cannot provide an answer, it will review the description of each tool. For instance, the weather app might be described as ‘Use this tool to find the current weather,’ while the calculator app could be described as ‘Use this tool to perform complex mathematics that you cannot manage on your own.’??

One thing to note about tool selection is that LLM is as a decision maker rather than as a content generator.? This decision will be based on the actual description of each tool as well as how the model interprets the description.? Different models may be better at the decision making process and complex chatbots could use a mix of models, some that are optimized for decision making while others are more suited for providing responses based on the information included along with prompts from the tools.

In some cases there may be overlap between what the tools can do.? For example both Web Search and Wikipedia could provide information on the start date of the 2024 Olympics in France.? If you are only concerned with getting an answer, you can just leave it up to the LLM to decide.? However, there may be reasons that you prefer one tool over another because of cost, response time, or data confidence.? In that case, you can add instructions to the LLM as part of your prompt to look at specific tools first.? Of course, if you really want to control variables, you should reduce the use of overlapping tools, especially tools like web search.

Memory

While tools are good at fetching generic information, how do we make a chatbot remember our specific conversation?? LLMs are simply functions that take in an output and provide an output.? At the moment the model itself does not change when it provides an answer unlike humans that are able to recall their response to a question at a later date.? There are several ways for a chatbot to get around this problem.

The first solution is brute force where a chatbot would simply record the conversation and then attach the growing conversation with each prompt.? This solution could get expensive and slow as more tokens are required to process the conversation history before a response is provided.

A second solution would be to use RAG.? The entire conversation would be saved, chunked and converted to a vector file and stored in a database.? The database would then search for relevant content each time you make a new prompt.

A third solution is for the chatbot to summarize the conversation and then pass the summary along with each prompt.? For example, if you tell a customer support chatbot that your iPhone 14 is not turning off, the chatbot may summarize the following information:

Name: John

Product: iPhone 14

Problem: Won’t Turn Off

Recommendation: Update Software

This data would then be stored in a database and could be fed into the LLM along with your future prompts.? The response may look like this next time you log into support:

Welcome back John.? Last time we chatted, you mentioned you had a problem with your iPhone 14 not turning off.? I recommended that you update the software.? Did this help?

An extreme solution would be fine tuning a model for a particular user.? This would involve updating the model after each conversation.? While impractical at the time, as models become smaller and processing power increases we may see applications such as customer support where the model is updated each night to reflect the questions of the day.

Testing / Optimization

Setting up a chatbot for the first time is relatively simple, however making sure it is effective for the particular use case takes extensive testing and iterating.? Reminder, a chatbot would have things that you can control such as tool selection, prompt templates, and model types.? However there are also things that you can not control such as how the model will interpret the inputs. Changing the temperature of the model can introduce even more variability.? The goal is to get the system close enough and then fail gracefully using moderation either asking the user for more clarification or having the model say, “I don’t know”, if it can not solve a particular problem.? One technique for testing is to create a set of training questions with acceptable answers then feed them to the system automatically and use an additional LLM to score the accuracy of the results based on the expected answers.

Implementation

Now that you have an understanding of the different elements that make up a chatbot, how can these be combined into a working solution?? You can always program a solution by hand but one of the best tools to use now is an open source library called LangChain that uses Python.? LangChain allows you to easily build a chatbot / agent that allows you to integrate prompt templates, RAG, tools and multiple external LLMs into a single solution.? If you are not a fan of writing code, there are multiple graphical tools built on top of LangChain such as Flowise that allow you to experiment with different tools and workflows before going into production.? These tools also allow you to quickly make changes after you have experience running the system and monitoring feedback.

Conclusion

By integrating with different tools and modifying prompts, chatbots are able to answer a wide variety of questions beyond what the LLM is trained on.? This capability will continue to improve as access to tools is standardized and specialized LLMs emerge that are focused on the different tasks that create a seamless experience for the user.