NLU Remains Relevant For Conversational AI

I’m currently the Chief Evangelist @?HumanFirst . I explore & write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.

The integration of LLMs into existing Conversational AI Frameworks (CAIFs) is becoming more and more prominent due to their generative and predictive capabilities. Most CAIFs have already implemented some form of LLMs or have plans to do so in the near?future.

Currently, the main purpose of NLU within these frameworks is to make predictions, assigning user utterances to pre-defined classifications and intents, and extracting entities such as named and industry-specific entities.

NLU can also be used independently to process conversational data offline. To consider the importance of NLU in today’s and future frameworks, below are a few practical considerations.

Efficient Open-Sourced NLU Pipelines

Many CAIFs come with generic, internal NLU pipelines that are likely based on open-source software that has no licensing requirements or third-party obligations.

For instance, with?Rasa , a powerful open-source?NLU API ?can be created that supports intents with structure and various entity types. The standard NLU pipeline can be used, and it is highly configurable.

There is no need for a large amount of training data or computing power, and training time is relatively quick. Moreover, there are several lightweight local installation options.

Built-In Efficiencies For Intents & Entities

Intents ?and entities have become increasingly efficient and organised over time. Many Gartner CAIFs have implemented nested intents or sub-intents, and intents can be split or merged with a drag-and-drop UI.

Entities ?are only associated with certain intents, and this pairing of an intent with specific entities requires two checks to be performed before the chatbot can respond.

Structures for entities include Compound Contextual Entities,?Entity Decomposition , entity groups, roles, etc.

Accurate entity detection is essential for the successful completion of an order, as it prevents the need to ask a user for data they have already provided.

Training Time & No-Code Environments

LLMs require data formatting and transformation to be done through a pro-code environment, which can be tedious and tricky. NLU, however, only needs a few training examples and is usually managed through a no-code studio environment.

Frameworks such as?Rasa ?and?Cognigy ?have made incremental training possible, and IBM Watson Assistant has recently been able to reduce NLU training time significantly.

Comparable Results between LLMs & NLU

When NLU-like classification tasks are performed with a LLM, the results from NLU engines and LLMs are often comparable, with NLU results typically being more reliable and predictable.

However, this is only the case when the LLM is used to its full capacity and NLU is optimised for constructing a classification model with minimal data.

Consistency with NLU

When the same data is submitted, NLU produces consistent and predictable results with minimal to no variation in results.

When?testing different LLM providers from a zero to few shot learning perspective, OpenAI appears to yield the best results, followed by AI21 and Cohere.

Unfortunately, it has been difficult to generate consistent and accurate content with LLMs such as Goose AI and Bloom.

Data Entry Is Easy

The majority of chatbot development frameworks have a user-friendly graphical user interface (GUI) for entering natural language understanding (NLU) data.

This point-and-click approach eliminates the need for formatting data into a JSON or CSV structure before importing it, thus avoiding potential errors.

Additionally, the lightweight nature of NLU Engine data exports makes the process of migrating between chatbot frameworks and NLU engines much less intimidating.

Intents are essentially just classes, and the necessity of classifying text and assigning it to one or more labels/classes/intents will never cease.

Recent advancements in terms of intents include the following:

Entities Have Evolved

Accurate entity detection is essential in avoiding the need to re-prompt the user, as it allows for the gathering of entities from highly unstructured input on the first pass.?Intents can be seen as verbs, while entities can be seen as nouns.

Read more?here .

Easy & Local Installations

If you are interested in exploring natural language understanding, there are several easy and accessible options available.

Rasa is an open-source NLU API that is easy to configure and train, while other NLU tools such as?Spacy, Snips, and Cisco MindMeld?are also available.

Moreover,?these tools are lightweight, open-sourced, and can be installed in any environment, even workstations, without the need for expensive computing, LLMs, or highly technical ML pipelines.