How to implement a tried and tested bot improvement cycle

If you’re working with conversational AI systems, then you’ll know that these things are like living organisms. A chatbot is more like a member of staff than it is a piece of software. Would you put a new member of staff on the phone or onto a live chat system and just leave them to it? Of course not. You’d give them support. You’d educate them. You’d make sure that they’re constantly given everything they need to succeed. So why don’t you do the same with your chatbot?

I recently had the pleasure of speaking with Jay Athia and Brea Lutton of Wysdom , where they shared their process for improving AI assistant performance.

In the webinar, we ran a poll that asked ‘Who has set up an operational bot management process?’ All participants either said ‘no’, ‘kind of’, or ‘yes, but it needs work’.

It’s evident that this is an area that we can do better in, so what should you do?

Start with organising your people

According to Jay and Brea, the first place to start is to make sure that you have the right people in place, with the right skills to succeed. A typical team configuration consists of:

1. Product owner: someone who owns the vision, prioritises work to be done, and manages delivery.
2. Technical: someone (or people) with the technical skills to deliver, including solution architecture, systems integration, and data engineering.
3. Designer: someone (or people) responsible for the experience design and performing analysis.

With a small team, this might even be a single person. For larger teams, this may consist of 7 or more.

Then, organise your data

Following and acting on data should be your North Star that supports your future decisions. Many teams have data available to them and most have some kind of a dashboard. The problem is that the dashboard is lacking the details needed to really make improvements.

To make measurable improvements to your chatbot, you need to have transcript data. You need to be able to see and review the transcripts of the conversations happening between your chatbot and your users. These transcripts are what allow you to find the specific areas where your chatbot is tripping up.

While having transcripts is required, you’re not going to go through every single transcript to pick out improvements. You need to focus on the transcripts that are affecting your goals. To do that, you need to have your transcripts hooked up to your intent data, your goal completion data and everything else. This way, you can segment, say, conversations with no goal completions, from successful conversations.

Making sure your data is available and easily organised in an easily digestible manner is crucial before you start the bot improvement cycle.

The bot improvement cycle

Once you have the team in place, you can seek to implement your operational bot management process, which looks like this:

1. Analyse: Inspect multi-channel conversation data to understand performance against target, and produce a list of candidates for improvement.
2. Prioritise: Rank order candidates according to effort, and forecasted impact on volume, automation success, and experience improvement
3. Implement: Design, build, test, and deploy changes, ensuring appropriate tagging is in place to measure impact.
4. Measure: Verify that changes are delivering the expected results on KPIs
5. Iterative improvement: Sprint cycles typically take 2-4 weeks based on the availability of resources required to implement the cycle.

For those of you familiar with agile project management and Scrum delivery, this process may seem familiar. And while it appears on the surface to be straightforward, the nuances of what your north stars are, the specifics of what you measure, the decisions on what you prioritise, the design of your solution, the discipline in following-up, and the ability to keep this process going, are all much easier said than done. That is where true success is found.

If you’d like to take a deep dive into this process, you can do so by watching the webinar on demand.