Natural Language Processing: Do chatbots have AI?

Chatbots have been one of the biggest buzzwords in Artificial Intelligence in the past five years, and while the hype has calmed down a little, chatbots are here to stay, especially in those areas with lucrative applications, like customer service automation and automatic lead qualification. And it might not be obvious whether they actually are an application of AI or not. Will adding a chatbot make your service smarter?

As I discussed in my previous post, AI can technically mean any tasks a computer does on a human's behalf. However, when people wonder about the intelligence of chatbots, typically they think about the Machine Learning (ML) kind of intelligence. And from that perspective, the answer is: some chatbots are reasonably smart, and some are not smart at all.

By far the most common part of a chatbot to contain ML is the very first part of the process: understanding what the user wants. It's possible to identify user intent by showing them options to choose from, but this only works for very simple bots that don't serve many different use cases. If you want your chatbot to feel like, you know, a chat, you will have to let the users say more or less what they want. This handy chart will tell you if a given chatbot uses ML to understand the user:

So, let's assume you have, or you want to have, a chatbot that can understand what the user chooses to write. The kind of ML that will help you with that is Natural Language Processing (NLP), is a common subcategory of AI in which computers process large amounts of text created by humans and analyse them. Areas of application within NLP include text-to-speech transcription, natural language understanding (NLU), and natural language generation (NLG).

Commercially, NLU is one of the most useful and lucrative areas of NLP, and contains tasks such as 

• Intent recognition: What does the user want?
• Sentiment analysis: Is the user happy, sad, angry or something else?
• Language detection: Is the user writing in Vietnamese or Italian?

For a chatbot, intent recognition is the most relevant area, though other NLP tasks could also be valuable. 

To put this in context, a chatbot that allows the user to choose between the buttons "Today's weather" and "Today's headlines" and replies with appropriate content is a form of AI but not ML. The entire behaviour is based on rules: Is the user selects weather, give them the weather. If headlines, headlines. Not much to see here.

A chatbot that lets the user write whatever they want and will recognise the keywords "weather" and "temperature" to show them weather-related information, or the keywords "news" and "headlines" to show them the latest news is a form of AI and technically a form of NLP/NLU, but not ML. The rules are clearly defined by a human and don't require any smart decision-making from the computer.

A chatbot that lets the user write whatever they want, will predict which possible user intent it best matches, and will initiate a rule-based decision tree in response, is a combination of ML and more generic AI. Here's what intent handling in such a chatbot might look like:

Currently the most successful and useful chatbots, especially customer care chatbots, work along these lines. They are smart in understanding what the user wants, having learned from vast amounts of training data, but beyond that, the response is generally based on pre-defined rules. 

One can also imagine a chatbot that also uses ML to create an answer, such as by employing Neural Networks to generate a response that would sound plausible to the user. Natural Language Generation (NLG) technology isn't quite there yet when it comes to producing text that could mimic human speech convincingly, but it's conceivable that future chatbots will not only flexibly understand what the user says, but also contain ML to generate responses without a human having to define them beforehand.

At this point in time, if you want your chatbot to be "smart," your best bet is to use NLP to understand what the user wants and use that to serve them something appropriate in response.

More articles in my series on AI fundamentals:

• Part 1: AI & ML: Get your terms straight
• Part 2: Natural Language Processing: Do chatbots have AI?
• Part 3: Supervised and unsupervised learning (coming soon)
• ?Part 4: From prediction to action (coming soon)