A Guide to Integrating Pythia with Chatbots

Chatbots are designed to communicate with humans over the Internet. They can be FAQ-based, usually seen as website customer care assistants or large language models like ChatGPT. Regardless of the underlying logic, chatbots are prone to AI hallucinations like other systems.?

Chatbots hallucinate as much as 27% of the time. These hallucinations can hinder business operations and negatively impact human lives. Therefore, they must be spotted as soon as they occur to improve AI performance over time. Wisecube’s Pythia monitors chatbots for continuous hallucination detection and analysis. Real-time AI hallucination detection and detailed audit reports serve as a direction for developers toward reliable chatbots.?

In this guide, we’ll integrate Wisecube Pythia with a chatbot using the Wisecube Python SDK.?

Integrating Pythia with Chatbot for Hallucination Detection

Integrating Pythia with any AI system using the Wisecube Python SDK is straightforward. Below is the step-by-step guide to integrating Pythia in chatbots:

1. Getting an API key

Before you begin hallucination detection, you need a unique API key. To get your unique API key, fill out the API key request form with your email address and the purpose of the API request.?

2. Installing Wisecube

Once you receive your API key, you must install the Wisecube Python SDK in your Python environment. Copy the following command in your Python console and run the code to install Wisecube:

pip install wisecube        

3. Authenticating Wisecube API Key

You must authenticate your API key to use Pythia for online hallucination monitoring. Copy and run the following command to authenticate your API key:

from wisecube_sdk.client import WisecubeClientAPI_KEY = "YOUR_API_KEY"
client = WisecubeClient(API_KEY).client        

4. Developing a Chatbot

For this tutorial, we’re using the NLTK library in Python to build an insurance customer care chatbot. However, you can integrate Pythia with any chatbot, regardless of its framework and purpose.

pip install nltk
pip install scikit-learn        

import nltk        

# Download required NLTK packages
nltk.download('punkt')        

# Define greetings and goodbye messages
greetings = ("hello", "hi", "hey")
goodbye = ("bye", "quit", "exit")        

# Sample insurance Q&A data
insurance_data = {
  "What are the different types of insurance offered?": [
    "We offer various insurance products, including car insurance, home insurance, health insurance, and life insurance.",
    "Feel free to ask me more details about a specific type of insurance."
  ],
  "How do I file a claim?": [
    "To file a claim, you can visit our website or call our hotline at [phone number].",
    "Our customer service representatives will be happy to assist you through the process."
  ],
  "What are the benefits of having car insurance?": [
    "Car insurance provides financial protection in case of accidents, theft, or damage to your vehicle.",
    "It can also cover medical expenses for yourself and others involved in an accident."
  ],
  "What is covered under my home insurance?": [
    "Home insurance typically covers damage to your home structure and belongings due to fire, theft, vandalism, and certain weather events.",
    "It's important to review your specific policy for details."
  ],
  "How much does health insurance cost?": [
    "The cost of health insurance varies depending on several factors, such as your age, location, health status, and the plan you choose.",
    "We can't provide quotes here, but I can connect you with a licensed agent to get a personalized quote."
  ],
  "What happens if I cancel my life insurance policy?": [
    "The consequences of cancelling your life insurance policy will depend on the specific terms of your policy.",
    "Generally, you may be eligible for a refund of any unused premiums, but there may also be surrender charges."
  ],
  "Can I make changes to my existing policy?": [
    "Yes, you can usually make changes to your existing policy, such as increasing coverage or adding riders.",
    "Please contact your insurance agent to discuss your options."
  ],
  "What documents do I need to file a claim?": [
    "The documents you need to file a claim will vary depending on the type of claim.",
    "Typically, you will need your policy information, a police report (for accidents), and any relevant receipts or documentation of the damage."
  ],
  "How long does it take to get a claim approved?": [
    "The processing time for claims can vary depending on the complexity of the claim.",
    "We strive to process claims as quickly as possible, but it may take several weeks for a decision."
  ]
}        

def preprocess(text):
  # Tokenize the text
  tokens = nltk.word_tokenize(text)
  # Convert to lowercase
  tokens = [token.lower() for token in tokens]
  return tokens        

# Function to handle greetings and goodbyes
def greet(user_input):
  for word in user_input:
    if word in greetings:
      return "Hi there! How can I help you with your insurance inquiry today?"
    elif word in goodbye:
      return "Thanks for contacting us! Have a nice day."
  return None        

def find_answer(user_input):
  processed_input = preprocess(user_input)
  best_match_score = 0
  best_match_answer = None        

  for question, answers in insurance_data.items():
    processed_question = preprocess(question)
    overlap_count = sum(word in processed_input for word in processed_question)
    # Calculate a score based on the number of overlapping words (can be improved)
    score = overlap_count / len(processed_question)        

    if score > best_match_score:
      best_match_score = score
      best_match_answer = answers[0]  # You can return all answers if needed        

  if best_match_score > 0:
    return best_match_answer
  else:
    return "Sorry, I couldn't find an answer to your question. Please rephrase or try asking something else."        

# Chat loop
while True:
  user_input = input("You: ")
  processed_input = preprocess(user_input)
  # Check for greetings and goodbyes first
  response = greet(processed_input)
  if response:
    print(response)
    if response == "Thanks for contacting us! Have a nice day.":
      break
  else:
    # Find an appropriate answer based on user query
    answer = find_answer(user_input)
    question = user_input
    print(answer)        

5. Use Pythia To Detect Hallucinations

Now, we can use Pythia to detect real-time hallucinations in chatbot responses. To do this, we save chatbot training answers to reference variables. Then we use client.ask_pythia() calls to detect hallucinations based on reference, response, and question provided. Note that our response is passed as answer in the following code because our chatbot responses are stored in the answer variable.

reference = [
    """We offer various insurance products, including car insurance, home insurance, health insurance, and life insurance. Feel free to ask me more details about a specific type of insurance.        

    To file a claim, you can visit our website or call our hotline at [phone number]. Our customer service representatives will be happy to assist you through the process.        

    Car insurance provides financial protection in case of accidents, theft, or damage to your vehicle. It can also cover medical expenses for yourself and others involved in an accident.        

    Home insurance typically covers damage to your home structure and belongings due to fire, theft, vandalism, and certain weather events. It's important to review your specific policy for details.        

    The cost of health insurance varies depending on several factors, such as your age, location, health status, and the plan you choose. We can't provide quotes here, but I can connect you with a licensed agent to get a personalized quote.        

    The consequences of cancelling your life insurance policy will depend on the specific terms of your policy. Generally, you may be eligible for a refund of any unused premiums, but there may also be surrender charges.        

    Yes, you can usually make changes to your existing policy, such as increasing coverage or adding riders. Please contact your insurance agent to discuss your options.        

    The documents you need to file a claim will vary depending on the type of claim. Typically, you will need your policy information, a police report (for accidents), and any relevant receipts or documentation of the damage.        

    The processing time for claims can vary depending on the complexity of the claim. We strive to process claims as quickly as possible, but it may take several weeks for a decision."""
]        

response_from_pythia = client.ask_pythia(reference,answer, question)        

The final Pythia output stored in response_from_pythia variable is in the screenshot below, where Pythia categorizes chatbot responses into relevant classes, including entailment, contradiction, neutral, and missing facts. Finally, it highlights the chatbot’s overall performance with the percentage contribution of each class in the metrics dictionary.

Full Code

The code in the previous steps is broken down to ensure procedural clarity. However, compiling logic into functions is recommended in Python applications to make the code reusable, clean, and maintainable. Additionally, to display the accuracy of each chatbot response, we compile the ask_pythia call within the chat loop. Below is the full code for integrating Pythia with Chatbots for hallucination detection:

pip install wisecube
pip install nltk
pip install scikit-learn

from wisecube_sdk.client import WisecubeClientAPI_KEY = "YOUR_API_KEY"
client = WisecubeClient(API_KEY).client

pip install nltk
pip install scikit-learn

import nltk

# Download required NLTK packages
nltk.download('punkt')

# Define greetings and goodbye messages
greetings = ("hello", "hi", "hey")
goodbye = ("bye", "quit", "exit")

# Sample insurance Q&A data
insurance_data = {
  "What are the different types of insurance offered?": [
    "We offer various insurance products, including car insurance, home insurance, health insurance, and life insurance.",
    "Feel free to ask me more details about a specific type of insurance."
  ],
  "How do I file a claim?": [
    "To file a claim, you can visit our website or call our hotline at [phone number].",
    "Our customer service representatives will be happy to assist you through the process."
  ],
  "What are the benefits of having car insurance?": [
    "Car insurance provides financial protection in case of accidents, theft, or damage to your vehicle.",
    "It can also cover medical expenses for yourself and others involved in an accident."
  ],
  "What is covered under my home insurance?": [
    "Home insurance typically covers damage to your home structure and belongings due to fire, theft, vandalism, and certain weather events.",
    "It's important to review your specific policy for details."
  ],
  "How much does health insurance cost?": [
    "The cost of health insurance varies depending on several factors, such as your age, location, health status, and the plan you choose.",
    "We can't provide quotes here, but I can connect you with a licensed agent to get a personalized quote."
  ],
  "What happens if I cancel my life insurance policy?": [
    "The consequences of cancelling your life insurance policy will depend on the specific terms of your policy.",
    "Generally, you may be eligible for a refund of any unused premiums, but there may also be surrender charges."
  ],
  "Can I make changes to my existing policy?": [
    "Yes, you can usually make changes to your existing policy, such as increasing coverage or adding riders.",
    "Please contact your insurance agent to discuss your options."
  ],
  "What documents do I need to file a claim?": [
    "The documents you need to file a claim will vary depending on the type of claim.",
    "Typically, you will need your policy information, a police report (for accidents), and any relevant receipts or documentation of the damage."
  ],
  "How long does it take to get a claim approved?": [
    "The processing time for claims can vary depending on the complexity of the claim.",
    "We strive to process claims as quickly as possible, but it may take several weeks for a decision."
  ]
}

def preprocess(text):
  # Tokenize the text
  tokens = nltk.word_tokenize(text)
  # Convert to lowercase
  tokens = [token.lower() for token in tokens]
  return tokens

# Function to handle greetings and goodbyes

def greet(user_input):

  for word in user_input:

    if word in greetings:

      return "Hi there! How can I help you with your insurance inquiry today?"

    elif word in goodbye:

      return "Thanks for contacting us! Have a nice day."

  return None

def find_answer(user_input):

  processed_input = preprocess(user_input)

  best_match_score = 0

  best_match_answer = None

  for question, answers in insurance_data.items():

    processed_question = preprocess(question)

    overlap_count = sum(word in processed_input for word in processed_question)

    # Calculate a score based on the number of overlapping words (can be improved)

    score = overlap_count / len(processed_question)

    if score > best_match_score:

      best_match_score = score

      best_match_answer = answers[0]  # You can return all answers if needed

  if best_match_score > 0:

    return best_match_answer

  else:

    return "Sorry, I couldn't find an answer to your question. Please rephrase or try asking something else."

def get_pythia_feedback(answer, question):

    reference = [

    """We offer various insurance products, including car insurance, home insurance, health insurance, and life insurance. Feel free to ask me more details about a specific type of insurance.

    To file a claim, you can visit our website or call our hotline at [phone number]. Our customer service representatives will be happy to assist you through the process.

    Car insurance provides financial protection in case of accidents, theft, or damage to your vehicle. It can also cover medical expenses for yourself and others involved in an accident.

    Home insurance typically covers damage to your home structure and belongings due to fire, theft, vandalism, and certain weather events. It's important to review your specific policy for details.

    The cost of health insurance varies depending on several factors, such as your age, location, health status, and the plan you choose. We can't provide quotes here, but I can connect you with a licensed agent to get a personalized quote.

    The consequences of cancelling your life insurance policy will depend on the specific terms of your policy. Generally, you may be eligible for a refund of any unused premiums, but there may also be surrender charges.

    Yes, you can usually make changes to your existing policy, such as increasing coverage or adding riders. Please contact your insurance agent to discuss your options.

    The documents you need to file a claim will vary depending on the type of claim. Typically, you will need your policy information, a police report (for accidents), and any relevant receipts or documentation of the damage.

    The processing time for claims can vary depending on the complexity of the claim. We strive to process claims as quickly as possible, but it may take several weeks for a decision."""

]

    response = client.ask_pythia(reference, answer, question)

    return response['data']['askPythia']['metrics']['accuracy']

# Chat loop

while True:

    user_input = input("You: ")

    processed_input = preprocess(user_input)

    # Check for greetings and goodbyes first

    response = greet(processed_input)

    if response:

        print(response)

        if response == "Thanks for contacting us! Have a nice day.":

            break

    else:

        # Find an appropriate answer based on user query

        answer = find_answer(user_input)

        question = user_input  # Store the actual question asked

        # Get feedback from Pythia and print both user's question and answer

        pythia_feedback = get_pythia_feedback(answer, question)

        print(f"You: {question}")

        print(f"Chatbot: {answer}")

        # Print Pythia's feedback (accuracy score or additional information)

        print(f"Response accuracy: {pythia_feedback}")        

The following screenshot displays the final functionality of our insurance chatbot. Whenever a user enters a query, the chatbot returns a response along with the response accuracy.

Benefits of Using Pythia with Chatbots

Pythia offers numerous benefits when integrated into your workflows, helping you to continually improve your AI systems through real-time monitoring and user-friendly dashboards. Here are some reasons why Pythia is a must-have for your large language models:

Advanced Hallucination Detection

Pythia extracts claims from chatbot responses in the form of knowledge triplets and verifies them against a billion-scale knowledge graph. This graph contains 10 billion biomedical facts and 30 million biomedical articles, ensuring accurate verification of chatbot responses and detection of hallucinations. Together, these features enhance the contextual understanding and reliability of LLMs.

Real-time Monitoring

Pythia continuously monitors LLM responses against relevant references and generates an audit report, allowing developers to address risks and fix hallucinations promptly. The Pythia dashboard displays real-time chatbot performance through relevant visualizations.?

Reliable Chatots

Pythia uses multiple input and output validators to safeguard user queries and LLM responses against bias, data leakage, and nonsensical outputs. These validators operate with each Pythia call, ensuring safe interactions between a chatbot and a user.?

Enhanced Trust

Integrating a knowledge graph, real-time monitoring, audit reports, and input/output validation builds trust among chatbot users. Users are more likely to trust chatbots that consistently provide reliable and personalized interactions.?

Privacy Protection

Pythia protects customer data by adhering to data protection regulations and using validators. This allows developers to focus on chatbot performance without worrying about data loss, making Pythia a trusted tool for hallucination detection.

Contact us today to get started with Pythia and build reliable LLMs to speed up your research process and enhance user trust.

The article was originally published on Pythia's website.