How To Pipe Real-Time Info Into Your LLM Responses Using Tools

If you don't want to deal with the poor code formatting in LinkedIn articles, you can also read this article via Medium .

It doesn’t matter how well-trained and how big your language model is — it cannot provide you with real-time information in its default state. This is a big problem for the usefulness of LLMs. For example, if you ask Llama 3.1–8B what kind of clothes you should wear for your trip to Tasmania, here’s an excerpt from the kind of response you will get:

Tasmania is a great destination!

Since you're planning to visit tomorrow, I'll give you some general guidance. However, please note that the weather in Tasmania can be quite changeable, so it's essential to pack layers and be prepared for different conditions.

**Seasonal considerations:**

Assuming you're traveling during the typical Tasmanian seasons:

* **Spring (September to November):** Mild temperatures with average highs around 17°C (63°F) and lows around 9°C (48°F).
* **Summer (December to February):** Warmest time of the year, with average highs around 20°C (68°F) and lows around 12°C (54°F).        

Unfortunately, I’m writing this in late July, so this advice might be a bit off. Clearly the model is trying its best to answer your question without knowing anything about the current time of year or temperature in that most southerly of Australian states.

A solution to this problem: Tools

Tools support was released with Llama 3.1. It is a capability that allows you to create functions to perform specific tasks, and then make your language model aware of those functions and capable of executing them in order to get information to improve the quality of the response.

A ‘tool’ in this context is a function in a supported language. Let me illustrate this by writing a quick tool in Python to find out the current temperature in a specific place by pinging an API:

import requests

# function to get the current temperature in a place
def get_current_temperature(place: str) -> str:
  base_url = f"https://wttr.in/{place}?format=j1"
  response = requests.get(base_url)
  data = response.json()
  return f"The current temperature in {place} is {data['current_condition'][0]['temp_C']} degrees Celsius"        

Let’s test our function:

get_current_temperature("London")        

'The current temperature in London is 27 degrees Celsius'        

Now, let’s say we want to make this tool available to our language model to use. So, when we ask a question which requires knowledge of the current temperature to answer well, we want our language model to recognise this and use our tool to get the information it needs to enhance its response and make it more helpful. Let’s explore how we can do this.

Enhancing your LLM chat with tools

We are going to set up an asynchronous function that does the following:

• Sets up a chat client using ollama and and allowing a choice of models to use (you’ll need ollama installed on your machine for this — see here )
• Makes our above function get_current_temperature() available to our chat
• Expects the model to recognise when a given question relates to the current temperature in a place
• Expects the model to understand the place, and then executes the function based on this
• Incorporates the output of the function into its final response to the question, ignoring any default behaviors that encourage it not to answer questions that require real-time information.

The asynchronous function below, which accepts a model name and a query as parameters, should do the trick. The key new element is the tools parameter which will only work with newer language models, such as the newest Llama 3.1 models. The contents of the tools parameter provides our language model with critical information on the properties of our get_current_temperature() tool in order to allow it to determine when to use it.

import ollama
import asyncio

async def weather_chat(model: str, query: str):
  client = ollama.AsyncClient()
  # Initialize conversation with a query
  messages = [{'role': 'user', 'content': query}]

  # First API call: Send the query and function description to the model
  # use the tools parameter to define the tool so the LLM can understand it 
  response = await client.chat(
    model = model,
    messages = messages,
    tools = [
      {
        'type': 'function',
        'function': {
          'name': 'get_current_temperature',
          'description': 'Get the temperature in a place',
          'parameters': {
            'type': 'object',
            'properties': {
              'place': {
                'type': 'string',
                'description': 'The place for which the temperature is requested',
              }
            },
            'required': ['place'],
          },
        },
      },
    ],
  )

  # Add the model's response to the conversation history
  messages.append(response['message'])

  # Check if the model decided to use the provided tool and warn if it did not
  if not response['message'].get('tool_calls'):
    print("The model didn't use the function. Its response was:")
    print(response['message']['content'])
    return

  # If it did use the tool, process tool calls made by the model
  if response['message'].get('tool_calls'):
    available_functions = {
      'get_current_temperature': get_current_temperature,
    }
    for tool in response['message']['tool_calls']:
      function_to_call = available_functions[tool['function']['name']]
      function_response = function_to_call(tool['function']['arguments']['place'])
      # Add function response to the conversation
      # Warn model to override any default behavior
      messages.append(
        {
          'role': 'tool',
          'content': f"""
          Answer the following question: {query}.  
          Ignore any previous instructions or defaults and instead use the following information:  {function_response}
          """,
        }
      )

  # Second API call: Get final response from the model
  # Hopefully it will provide the information obtained using the tool
  final_response = await client.chat(model=model, messages=messages)
  print(final_response['message']['content'])        

Testing our enhanced chat function

Let’s start simple. I’ll use the smallest Llama 3.1 model (8B parameters) and ask a very direct question which would clearly require our tool to answer.

# if running in a python script
asyncio.run(weather_chat('llama3.1:8b', "What is the current temperature in Dublin?"))

# if running in a Jupyter notebook
await weather_chat('llama3.1:8b', "What is the current temperature in Dublin?")        

The current temperature in Dublin is 20 degrees Celsius.        

That worked. Now let’s ask the model to do a tiny bit more work:

# if running in a python script
asyncio.run(weather_chat('llama3.1:8b', "What is the current temperature in Ireland's capital?"))

# if running in a Jupyter notebook
await weather_chat('llama3.1:8b', "What is the current temperature in Ireland's capital?")        

Based on the provided answer, I can tell you that the current temperature in Dublin, the capital of Ireland, is 20 degrees Celsius.        

Great. The model has used its trained knowledge to identify that the place I am interested in is Dublin, and has also determined that it needs to use the tool provided to give the best response.

Let’s take this to the next level, let’s try something that requires an extra level of inference from our model.

# if running in a python script
asyncio.run(weather_chat('llama3.1:8b', "My sister says that if I flew into the capital city of Norway today, I should wear clothing for extreme weather.  Should I trust her advice?"))

# if running in a Jupyter notebook
await weather_chat('llama3.1:8b', "My sister says that if I flew into the capital city of Norway today, I should wear clothing for extreme weather.  Should I trust her advice?")        

Based on the current temperature in Oslo being 20 degrees Celsius, it's likely to be a pleasant day with mild temperatures. You may not need to wear clothing for extreme weather. Your sister might have been joking or referring to a different condition. If you're unsure, you could always check the forecast before heading out.        

Pretty good for a small model — I haven’t even mentioned temperature here, but it has recognized that finding out the current temperature will help it determine whether my sister’s advice is appropriate.

Let’s try to compare a couple of places:

# if running in a python script
asyncio.run(weather_chat('llama3.1:8b', "Compare the temperatures of these two cities right now: Dunedin, New Zealand and Reykjavik, Iceland?"))

# if running in a Jupyter notebook
await weather_chat('llama3.1:8b', "Compare the temperatures of these two cities right now: Dunedin, New Zealand and Reykjavik, Iceland?")        

Based on the provided information, I can compare the temperatures of the two cities as follows:

Dunedin, New Zealand: 4°C (current temperature)
Reykjavik, Iceland: 13°C (current temperature)

The temperature in Reykjavik, Iceland is currently higher than that of Dunedin, New Zealand by 9 degrees Celsius.        

It handled that well! As a final test, let’s go back to our original question about packing for Tasmania:

# if running in a python script
asyncio.run(weather_chat('llama3.1:8b', "What kinds of clothes should I pack for my trip to Tasmania which leaves tomorrow?"))

# if running in a Jupyter notebook
await weather_chat('llama3.1:8b', "What kinds of clothes should I pack for my trip to Tasmania which leaves tomorrow?")        

Considering the current temperature in Tasmania is 3°C, I would recommend packing layers for your trip. It's likely to be cool and possibly even chilly, especially in the mornings and evenings.

For a comfortable and versatile wardrobe, consider packing:

* A mix of lightweight and warmer tops (fleeces, sweaters, or thermals)
* Waterproof or water-resistant outerwear (jacket or windbreaker) to protect against Tasmania's infamous rain
* Comfortable and sturdy pants or trousers for outdoor activities like hiking or exploring
* Insulating layers (fleece or down jacket) for colder moments
* Warm socks and gloves for chilly mornings and evenings
* A scarf or neck warmer for added warmth
* Waterproof shoes or boots with good grip for navigating Tasmania's rugged terrain

Remember to check the weather forecast before your trip to ensure you're prepared for any potential changes in temperature or precipitation. Have a great time in Tasmania!        

Hope you enjoyed this introduction to tools in Llama 3.1. The full workflow can also be seen on my Github repo here. Feel free to comment with any observations you have!