Building a chatbot without "Utterances"

Since the boom of chatbots in messaging apps such as WeChat or Facebook messenger, companies are becoming increasingly clear that they need to be where the customer is. As users, we spend 90% of our smartphone time using only 3 applications, with instant messaging being the main one. To this we must add that it is by far the most-adopted channel on a global level.

These factors, together with the fact that companies need to automate their customer service as much as possible, converge with the phenomenon and current boom of chatbots. Its mission is simple: be part of your day to day when it comes to solving doubts, questions or purchases, and help you get relevant information instantly without waiting or endless lines in phone calls. The recent emergence of smart speakers in our homes adds to the appeal of voice when interacting with chatbots.

But the question is: how easy is it for a company to create one of these chatbots? Currently there is a short list of key software providers to design and launch a chatbot. The most relevant of the current scene are: Google Dialog flow, Inbenta, Amazon Alexa, Microsoft and IBM Watson. All of them, a priori similar, offer the possibility of being tested without having to make an economic investment and ultimately they should be the key to giving life to your chatbot. The reality is that these software platforms are not able to understand the needs of your business or create intelligence for the chatbot to learn automatically. Here is the great myth or "false god" about chatbots: Machine Learning (an overrated and even inappropriate term in this context) or self-learning. There is no software that generates knowledge in a chatbot automatically and, if there is, it would not be advisable. The learning of a chatbot must always be assisted or supervised. And this is very well known to the Botmasters.

In effect, the Botmasters are already here. In the same way that a community manager manages social networks, or a webmaster manages the web, the Botmaster takes care of the chatbot. This role or figure is relatively new and increasingly becomes more important to the extent that companies invest in Artificial Intelligence, digital transformation, and offer new channels of interaction with customers. We are talking about a new type of employment or work.

The Botmaster is responsible for teaching, managing and maintaining the chatbot. One of the great challenges that Botmasters face is the creation of chatbot knowledge units, also known as "intents". They represent the individual ability to respond or solve a particular issue. For example, in an airline, an "intent" could be "How to travel with pets?". When asked by a user about this topic, the chatbot provides a relevant response that helps the user to resolve their doubt. So far so good. The Botmaster has been able to generate an "intent" for the machine and has provided a teaching on what the answer is.

The real problem is yet to come ... What will happen if we reformulate the same question with different words? "Can I fly with my dog?"

In this case, the default chatbot lacks understanding to understand that the two questions refer to the same intent. In specific terms, this capability is called NLU (Natural Language Understanding). Only the tools that have Artificial Intelligence and NLU will be able to process and conclude that both phrases refer to the same thing. Technologies such as Google, Amazon or Microsoft do not have NLU and force the Botmaster to carry out a manual and unitary training of "variations" of the same "intent" in order to provide the chatbot with a minimal comprehensive capacity. Even some of the above-mentioned platforms need to have a minimum number of utterances. Following the example of "How to travel with pets?", The Botmaster should create as many variations as possible: “May I travel with pets", "It is allowed to fly with pets", “Can I take pets on board" etc. As a general rule, it is recommended to have a minimum of between 10 and 40 utterances for each unit of knowledge of the chatbot. Not to mention cases such as “Can I fly with a dog", “Can I fly with a cat", “Can I fly with a Dalmatian", “Can I fly with xxxx ..." and thus swelling the list, whose limit will be the creativity of the Botmaster themselves. There are even cases in which a Botmaster resorts to the help of his relatives during the weekends in order to expand such lists of variations, since in the end a single individual has a limited ability to rewrite the same phrase in so many different ways

And in all this, where is the Artificial Intelligence? Is it the machine that is at the service of the human being? At first glance one feels the opposite is true, because it is the Botmaster who, in some way, needs to make a great effort to make the chatbot work even minimally well. Some readers could conclude that the creation of these variations or utterances is not a great effort either, and in part they might be right. Everything will depend on the purpose of the chatbot (what mission it has) and to which group of users it is directed. The reality is that in any business vertical, a company needs an average of between 200 and 800 "intents" to cover a use case. Taking an intermediate value, for example 500 "intents", and let's assume that we are fortunate and we do not have to create utterances of the style "Travel to Paris, Travel to Barcelona, Travel to xxx", and taking into account that we are going to create a minimum of 20 utterances per intent, we get 10,000 utterances. Possibly the implementation time of a chatbot is extended in weeks (and in cost) only by the need to have to create those utterances. In fact, in many cases it represents more than half of the time of implementation of a chatbot project. Not to mention the topic of languages: if we are building a chatbot that should carry out the same function but in 10 languages, our complexity and effort will be multiplied by 10 times (100,000 utterances… almost a book ?!).

Is there any way to alleviate this situation? Of course. When it comes to choosing technology and chatbot software, you have to look for a vendor that has NLU capabilities. In case of not having Natural Language, this does not necessarily mean that you can not build a chatbot, but surely the lack of comprehensive capacity will result in a considerable increase of time and cost in your project. The NLU allows you to obtain automatically from Computational Linguistics an infinite number of utterances for the same intent, and therefore a considerable saving of money at the same time as improving the "time-to-market" of the chatbot. It is very important to be able to validate these technologies, without forgetting the multilingual functionality. Most US software has a certain degree of effectiveness in English, but when dealing with languages such as German, Italian or Portuguese, we will find that their performance declines. Surely we will have more luck with European technology, in which there is a more developed sensitivity regarding the users idiomatic realities.

There is no definitive chatbot software, yet, but it is important to evaluate the benefits and disadvantages of each of them. Undoubtedly the NLU is a key element, especially if our chatbot will interact with languages other than English. If our project requires a very large volume of intents and we also need to build a chatbot in more than one language, surely the NLU will be the functionality to which more attention will have to be paid.