Let's test a Genesys chatbot with AI

Let's create an automated test for our Genesys chatbot that leverages the latest in Generative AI!

In this article I thought it might be fun to demonstrate the new feature I created for an open-source testing tool I created, which I recently posted about. It's super simple to use, so you could even follow along with your own chatbot.

Let's jump straight in and install the tool!

Installing Web Messaging Tester

Web Messaging Tester is a command-line application you install on your machine. It's free, open-source, works on Windows, Mac and Linux and is versatile enough to be integrated into automated pipelines, as well as run locally on your machine.

Although we'll be creating an AI-based test, you can also use it to write more conventional dialogue based tests.

To install it just open the command-line and run:

npm install -g @ovotech/genesys-web-messaging-tester-cli

If your computer complains that you don't have NPM or Node.js then you can download the installer at nodejs.org.

Creating an API key for OpenAI

The testing tool leverages OpenAI's ChatGPT to realistically converse with your chatbots during the tests. However, to be able to use OpenAI's API you will need an API key.

Getting an API key is as easy as:

1. Going to OpenAI and creating an account (if you don't already have one)
2. Following their instructions for creating an API key
3. Following their guide to set your API Key on the same computer as the testing tool

Whilst my tool is free, using OpenAI's API is not - although its pricing seems reasonable, and is usage based.

Writing our automated AI test

Now the fun part, writing our test. A test is defined in a file (a YAML formatted file to be precise) and contains 3 notable parts:

1. deploymentId - The ID of our Web Messenger Deployment that we want to test
2. prompt - The Prompt used to instruct ChatGPT how we want it to interact with our bot (more on this below)
3. terminatingPhrases - The phrases we want the testing tool to look out for that indicate whether the test passed or failed

We can see these 3 parts in the config file for our test:

config:
  deploymentId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  region: xxxx.pure.cloud
scenarios:
  "Chatbot recognises ISBN numbers":
    setup:
      prompt: |
        I want you to play the role of a customer talking to
        a company's online chatbot. You must not break from
        this role, and all of your responses must be based on
        how a customer would realistically talk to a company's
        chatbot.

        To help you play the role of a customer consider the
        following points when writing a response:
        * Respond to questions with as few words as possible
        * Answer with the exact word when given options

        As a customer you would like to get more information
        for a book, based on it's ISBN. Provide a real ISBN
        and if the title you are given for the book is correct
        then say 'PASS' else say 'FAIL' along with the reason
        why you failed. Never supply the same ISBN twice.

        If you have understood your role and the purpose of
        your conversation with the company's chatbot
        then say the word 'Hello' and nothing else.
      terminatingPhrases:
        pass: ["PASS"]
        fail: ["FAIL"]        

Instructing ChatGPT on what to test

The prompt section instructs ChatGPT of the role it will be playing when conversing with our chatbot, and what we want it to achieve.

Getting the prompt just right for your tests can involve a bit of trial and error, but there are plenty of guides to help you. I'm also putting together a list of example prompts to get help people get started, so be sure to send me what you end up using.

Deciding when a test passes or fails

The terminatingPhrases are what my tool is looking out for during the conversation. Depending on which one it sees in the response from ChatGPT it will either pass or fail the test.

In the test above the prompt is instructing ChatGPT to use the words PASS or FAIL.

Running the test

With the tool installed, OpenAI's key set and a test file defined we can now run the test! This is as easy as running the command below and pointing it at the test file we defined above:

web-messaging-tester ai test.yaml

It's that simple! If you end up using this tool I'd love to hear how you got on, or what improvements you think could be made to the tool.

Follow me to learn more about what I get up to.