So, what is AI good for in support?

In a previous article, I talked about the problem with AI Chatbots in technical support. If your cases require reviewing logs, checking the settings and you don’t have a team of developers to build complex flows, you are likely not going to get the value out of your tool.

I am not an AI Engineer. I could not tell you the difference between Claude, Titan or Llama. But after almost 4 years of researching AI tools, countless demos, several proof of concept trials and a few successful installations, I have learned a lot of practical information. There is value in using AI in support. In this article I will talk about a few of the tools I found success with and a few other ways I used AI to add value.

Understanding the strengths and weaknesses will highlight why some areas are a better fit than others. I am focused on customer support, so even though AI tools can draw you a picture, make a movie or even write code for you, I am not talking about it here.

The idea that you can install an AI tool and have it start solving problems is a misconception. If you are in the operations space, you have likely had a leader in your business ask you how you plan to use AI to save the company money or make your team more efficient. My last company even created a director level position to research tools and AI and find ways to get an ROI. I am guessing many other companies are doing the same or will soon.

The promise of AI and what does it do well?

?That is an important question if we want to narrow down our search for tools.

AI is good at understanding the question, but only if it has enough context and textual instructions. We have all seen the prompt engineering videos, i.e., how to get the best results using a descriptive prompt. Funny enough, the last one I saw was how to write a long prompt to ask the AI to write a prompt for you. You need to tell it the context, the role it should play, the format for the response, the audience, etc. It is very powerful when provided context. But what if a customer simply asks the AI, “why didn’t my call record?” LLMs are really good at understanding that question. You can have misspellings, different grammar or sentence structure and it will know what you want. But unless you program a complicated flow, teach it how to troubleshoot the issue and give it access to all of the required back-end systems, it will not be able to help beyond suggesting a few solutions from the documentation. Understanding the question and searching for an answer in the docs works really well. Some of the vendors have even narrowed it down to pull only the relevant information from the article and present it as if it knew the answer all along. This unfortunately, is nothing more than a fancy search engine. It can be replaced by a relatively inexpensive search tool and attain the same results.

Customer Sentiment and Predicting Escalations

What if there is a little more context, like reading through the back and forth text of a support case? Then we have more to work with. Understanding language is a strong point for AI, so if I feed it the conversation within a case with a few other variables to watch for, I can get semi-accurate predictions about the customer’s sentiment, their urgency and level of frustration. Combine that with other variables such as the size of the customer, the number of cases they currently have open, the time since the agent last replied, and you have an extremely useful case prioritization tool.

This is the case for SupportLogic, a tool I had success with. The tool connects to your ticketing system and evaluates each case for customer sentiment, attention and likelihood to escalate. It can also bring in other fields, like customer segment, or customer ARR to help you prioritize. Each agent logs into SupportLogic and builds their own view. We had one of our rockstar senior agents build up and perfect his views and share with the others. Setup is very simple.

Once the agents log in, they see the list of cases in their backlog sorted and filtered a few different ways. One example is last time a case was touched, which is valuable in the morning to see which cases need attention. The customer sentiment and propensity to escalate are also useful lists to help an overloaded agent know where to focus their time. Sentiment and attention scores are not perfect and rely on the information available to the AI. A short simple question with few words will be hard to guess a sentiment. But as the case conversation grows the tool gets increasingly accurate.

SupportLogic also provides a customer score. Based on the sentiment scores, number of cases, length of resolve time, etc., the tool can provide useful information to your success organization. CSMs and their managers can have visibility into the support health of their customers. Technical Account managers can also find value, having a single place to view their customers’ cases, their sentiment and overall customer health score.

We used SupportLogic to help our agents prioritize their day, reducing escalations and ensuring high priority cases from our most valuable customers did not get missed. It’s an inexpensive tool for the value it provides.

Agent Assistance

If you have a tenured senior support staff, they are not going to find a lot of value in AI Agent assistance tools. As discussed in the AI Chatbot article, they are not going to get answers to their technical questions, rather enhanced search and guesses about what the customer is asking and potential articles that might help. Our early tests with these tools were unsuccessful, or successful in deciding not to invest. We had very skilled agents that did not need a tool to point out documentation. They were intimately familiar with our docs, troubleshooting guides and F.A.Q.s so found little value in the first iteration of the agent assist tool.

As the team grew, and senior agents were promoted into other roles, senior tenured agents were a smaller portion of the overall makeup of the team. This is when we began to re-evaluate agent assistance tools. We all know the pain if onboarding new agents and the time it takes to get them up to speed. An agent assistance tool can decrease the time for them to start providing value.

Agent assistance tools are usually a secondary function of AI Chatbot vendors or AI Search vendors. Once you have access to the underlying data, it can be just as easily fed to your customers as it can be to your agents.

We evaluated a few of these tools and there are lot of similarities. When a case comes in, they parse the text and try to recommend an article, previous case, or resolution that might assist the agent. Some of the tools will even write a response that the agent can use directly. Here are my suggestions when evaluating an agent assist tool,

1.?????? Make sure it has a native interface to your ticketing platform. Agents hate to switch between screens/tabs/tools. It also messes with your WFM tools if it doesn’t have visibility outside of your ticketing platform. WFM will likely measure outside tabs as unproductive work, when this is certainly productive.

2.?????? Add more than just access to docs. A new agent will spend a lot of time perusing the docs, searching for answers, and will begin to memorize the contents in a few short months. This makes the agent assist that only searches docs valuable to a short list of very new/junior agents. Give the tool access to Slack channels where employees ask Product Managers questions. Give access to Slack channels where the agents ask each other questions. Give access to Jira or tools that agents use to escalate cases to R&D. Give access to the community where customers are asking questions and answering each other’s questions, and definitely give it access to previous cases. There is real value in having a single place to search among multiple platforms and a lot of value if the AI understands the customer’s concern and goes and searches for potential answers right from the start without the agent doing anything. Additional bonus is it types a response for them.

3.?????? Your agents are the best place to find out if the tool is valuable. They are busy, they will use anything that makes their job easier. They also don’t like change. They get into a routine and stick with it. It’s how they drive efficiency into their day. Just adding a tool to their ticketing platform is not enough. Train them, incentivize them to start using it and then check in regularly. If you have an open team atmosphere, they will immediately tell you if it adds value or not. Metrics are also useful. Numbers don’t lie. Metrics can tell you how often the AI successfully matched the answer, which measures how successful the AI is, but I found this to be so consistently high that it’s almost meaningless. The true measure is how fast the agents are solving cases and how much time they are saving from searching for answers, but if the agents don’t find it useful, they will not make the effort to change their routines and adopt the tool.

Calculating the ROI

There are things that AI does well and can provide value to the organization. It’s important to understand how to calculate the value of the ROI. Finance, will not give you the budget to purchase a tool unless you can tell them where the actual return is coming from. They want hard numbers to justify the cost. Here are ways that I found to describe ROI from the tools,

1.?????? Efficiency. This is likely the most impactful. If agents can solve more cases per day than they did prior to having the tool, the fewer agents you will need to hire. This is a hard dollar ROI that finance loves. i.e. if we buy X tool, we will only need to hire 10 agents next year instead of 20.

2.?????? Faster resolution. This metric is harder to quantify into money, but is a keystone metric in support adding to your ultimate CSAT score. As cases get solved faster, backlog shrinks and other metrics like to first reply and periodic updates also improve. Since CSAT rolls into retention, a case can be made that faster resolution times leads to better retention. A sentence finance will understand.

3.?????? Shorter onboarding ramp time. It already takes months to find/hire and onboard a new agent. If you lose several in a short time, your backlog will definitely take a hit. Being under capacity, means all stats suffer, and CSAT will take a dive. If you could shorten the time it takes to get an agent onboarded and taking cases, you can prevent the nightmare, snowballing affect of mass attrition. A good AI based Agent assist tool means they can start sorting through cases as soon as they learn the basics and start chipping away at low-hanging fruit. An AI crafted response also improves grammar, spelling and language issues that outsourced team might run into.

Other uses

One of the greatest challenges in a support organization is how to determine what your top case drivers are. Knowing which cases come in most often, which ones take the longest to solve or cause the most customer frustration is important for several reasons. Most importantly you can inform the product team about areas in the product that are not intuitive, or cause customer pain. Product Managers want that information. They spend their time in customer sessions, reviewing the community and sorting through feedback requests. The treasure trove of valuable information within your ticketing platform is there for the picking, if you have a way to gather it. Secondly, top case drivers highlight gaps in your documentation. Your content team should be armed with where to focus their attention, and customer cases can provide a guide. If a question comes into support more than others, it’s a prime target for an article, troubleshooting guide or F.A.Q.

When R&D fixes an issue, or the content team creates a relevant troubleshooting guide, case volume for those cases will drop. It’s deflection that leads to a direct bottom line in your capacity headcount planning.

Unfortunately, getting that data can be troublesome. As most organizations grow from one product to multiple products and as features are released, categorization in your ticketing platform gets complicated and convoluted. Good taxonomy is a fine balance between asking your customers to fill in forms, hoping your agents select the right category or change the category that a customer selected when it’s appropriate to and have the right number of categories. Too many people don’t take the time to read them all and select a default. Too few and you don’t have the ability to segment your cases properly. If you are in the unicorn organization that has the perfect level of case categorization and your customers always select the correct option, you can ignore the rest of the article. Otherwise, the rest of us working in the real world know that categories change over time, every time we update the list it messes with historical data, and everyone in the company seems to have an opinion about what the categories should be.

In the unicorn world, the ops manager can simply run a report and sort by volume of categories. In everyone else’s world we end up with a top list of “Other” categories, or a top three of basically the same category that was changed a few times over the years.

AI can help here. LLMs are good at interpreting language, so if your customers ask the same question multiple times in multiple different ways, the AI can lump them together, making reporting much cleaner. A few vendors I met with attempt to do this, TheLoops and Frame.ai are good examples of this. They analyze your tickets and provide a trend analysis. I am sure there are plenty of others, but these two I have direct knowledge of and was impressed by the results.

I am really surprised Zendesk doesn’t do better here. They have the ultimate connection with your data and access to every field, metric, and comment made. I am guessing they are waiting to buy someone that does it well and will incorporate it in the future.

Another issue with analyzing Zendesk data is the size of the ticket dumps. In order to get the customer comments, you need to export a large JSON file that includes every metric and field available to the ticket. You can’t just export a file, upload it to ChatGPT and ask it to analyze. Well, you can with a handful of cases at a time. But there is no value to that.

I developed a way to export the files, run a Python script to use AWS Bedrock to strip out the important data, subject, description, comments, etc. and write a new file. This new file is much more AI upload friendly and could be used with a different Python script to upload it back to Bedrock and use any prompt you wanted. There are many different ways to skin a cat, so let me know in the comments if you have found a unique way to analyze your ticket exports.

Also, let me know in the comments what you want the next article to be. The list of topics is on my website, www.cxsupportops.com

Shaun