Efficiently Building Reliable and Scalable Conversational AI Agents: A Framework and No-Code Approach

Designing safe, trustworthy and scalable Conversational AI Agents with OpenDialog

?? I am the co-founder and CPTO of OpenDialog AI - Our mission is to transform the way users and organizations engage with business systems and processes by automating interactions through conversation. I write about product development, Conversational AI technologies, and, more generally, the impact of AI on the way we live and work.

Introduction

A lot of companies are excited and very tempted by the idea of launching Conversational AI agents to scale and automate interactions. A major concern, however, is how can they get there efficiently, with agents that are reliable, especially if operating in regulated environments.

reliable: “consistently good in quality or performance; able to be trusted.”

The challenge is that the requirement for reliability is in direct tension with the need for efficiency - the need to move quickly, show results early and be able to scale efficiently.

efficiency: “(of a system or machine) achieving maximum productivity with minimum wasted effort or expense."

Of course, if you assemble a large enough team of capable enough people and have them work long enough you will likely end up with something that is reliable enough. The process, however, will not have been efficient, modifications to the solution are unlikely to be easy, especially as people move on and knowledge is lost, and return on these investments will only come over multiple years.?

A core goal of OpenDialog is to empower organizations to efficiently create Conversational AI Agents that are reliable, while still taking advantage of all the benefits LLMs offer.

A core goal of OpenDialog is to empower organizations to efficiently create Conversational AI Agents that are reliable, while still taking advantage of the benefits LLMs offer. In this article I delve into how we go about achieving this. While the examples are specific to OpenDialog the learnings are more universal about what makes a good Conversational AI Agent management system.?

The focus is on three key, high-level, aspects of OpenDialog that enable it to support the creation of reliable AI Agents, efficiently.?

1. A solid conceptual framework and methodology for building Conversational AI agents, with control, trust and explainability considered from the start.?
2. A no-code AI Agent Management System that is optimized around that methodology, enabling user and teams to quickly assemble and reuse solutions.?
3. Extensive data collection and analysis to provide explainability, inform improvements and, through clear auditable traces, provide the right sort of assurances or reliability, especially those in regulated environments.?

The power of a conceptual framework for Conversational AI Agent applications?

Shared concepts and processes matter. They make teams more efficient in communication and sharing of learnings and they make the outcome more robust. This is why at OpenDialog the starting point has always been the conceptual framework that underpins a Conversational AI Agent. The actual software is simply a tool to provide builders the means to efficiently manipulate this conceptual framework and build out AI Agents using it.?

The OpenDialog conceptual framework enables much of the flexibility that the platform provides. It consists of levels and components that help you define the context within which your agent and the user communicate. It allows you to take a design-system approach to conversation design, going from high-level descriptions to individual turns within a conversation. This structured approach provides the following benefits.

1. It clarifies the relationships between the different components of our system leading to a system that is modular and loosely coupled, enabling the builder to intervene exactly where it is required. This in turn provides control and control leads to safety and reliability.?
2. It enables us to take advantage of standard solutions to common problems across different domains. These patterns support reuse and knowledge sharing that in turn translate to speed of execution.?
3. It facilitates testing and evaluation as we can build supporting infrastructure for analysis of outcomes at scale, taking advantage of the shared concepts provided by the framework and applying them across all the different AI agents.?
4. It improves communication amongst the team. The framework provides a shared set of concepts and terms meaning that the team can become more efficient in how they communicate with each other.?
5. It puts the conversation designer in control, increasing trustworthiness and ensuring that the right user experiences can be designed while still taking advantage of LLMs.

A no-code AI Agent Management System

With a conceptual framework in place, the next task is to provide easy access to those ideas through a software tool. This is what our no-code AI Agent Management System does. It is built around the OpenDialog Framework and enables designers to compose solutions quickly and efficiently using the OpenDialog framework as their shared language.

An AI Agent Management System should not just be a collection of features and some sort of workflow engine. What is far more important is how these features are combined to provide an outcome.

An AI Agent Management System should not just be a collection of features and some sort of workflow engine. We could talk describe OpenDialog as a tool that offers access to multiple LLM providers, prompt management, RAG, intent management, a conversation designer, a message designer, API integration, analytics, WebChat UI, etc. But these are just features. What is far more important, is how these features are combined to provide an outcome.

To describe the OpenDialog method let's walk through the lifecycle of an exchange between the user and the AI agent as illustrated below.

Understand: The first requirement is to understand the query at a high level. We use LLM-powered semantic classifiers to give us an indication (in terms of an intent) about what the user utterance is.

If you worked in conversation design for some time you will be familiar with intents where we would train machine-learning models based on example phrases. The semantic classification approach is quite different. It takes instructions in natural language that enable it to classify phrases across much more broad categories that then old-school intent classifiers. A classification could be as high-level as “a question” or “a statement” or as specific as “request to speak to a support agent”. Semantic classifiers can also identify users attempting to act as Bad Actors or potential distress.?

Semantic classifiers enable conversation designers to quickly create high-level categories that suit their AI agent application across different contexts and even combine multiple classifiers in a single context. This provides powerful tools to explicitly understand a query, at the appropriate level, before acting on it.?

Orchestrate: Following a high-level identification of intent we contextualize that within the conversation and overal process and we can further personalize the request based on user-specific information.?You can think of this as a second level of interpretation where we focus on what a certain intent means in the context of the specific context of a conversation.

One of the most powerful uses of contextualisation is that it empowers the AI Agent to actually say no (as counterintuitive as that may sound!). Imagine going through an information collection process for an insurance claim or a job review and having to start fielding questions that are on a completely different topic. As in a real-life interaction you would be nicely asking the user to refocus on the goal at hand. Understanding the question through semantic classification and then contextualising to a process is key to achieving that.

Act: Following orchestration we can decide how (and whether) we need to act. An action may consist of a combination of further LLM-powered reasoning and API calls to external systems. An action can take inputs from the information generated within the conversation and feedback information into the conversation.

Orchestrate: Having processed the user query and performed any actions that where relevant we can now turn our attention to what an appropriate response would be. We first identify based on activity so far what the new conversational context is that will, in turn, enable us to identify an appropriate intent for our AI Agent.?

Respond: That AI Agent intent will then kick-off a corresponding series of activities including further reasoning, content generation, RAG or post-processing of content, or further action. Finally, we will format an appropriate response based on the communication channel and send that over to the user.?

At every step of the process the designer has complete control of the information that will get sent to any LLM-related activity such as specific user attributes, the conversation context, the process context and the conversation history.?

The LLMs are stateless by default and the designer gets to decide if it is important to share conversational history or not, how much of that history to share (the entire conversation, just the last few exchanges, just the AI agent responses, etc) based on the context of the conversation. The OpenDialog approach balances our wish to take advantage of LLMs for fluid conversations while not being completely beholden to the LLM to decide every step in the conversation. The OpenDialog framework enables us to orchestrate between a series of smaller, far more understandable, robust and specialized prompts that work in coordination to provide a result.?

The OpenDialog approach balances our wish to take advantage of LLMs for fluid conversations while not being completely beholden to the LLM to decide every step in the conversation.

Each prompt can operate with just the information it needs to know in order to achieve its task rather than having to reason about the entire conversation or process.?

Analyze, Analyze, Analyze

The final aspect we will consider here is analytics.

The advantage of an opinionated framework with structure built in from the start is that you can then rely on that structure to annotate the data that is collected in real-time. Every utterance from the user, every action and every response is executed within a given context and that context can instantly be converted into actionable data.

There are two broad styles of views that we support in OpenDialog.

Context-first view

The first is a context-first view where we can view how conversations are distributed across context:

Conversation-first view

The conversation-first view, instead, provides a view of a single conversation and a play by play analysis of all the reasoning that the conversation engine performed before taking a specific action.

This conversation-first view provides for each interaction a rich set of information that is kept in a fully auditable log enabling us to refer back to it as required both for the purposes of analysis and identifying improvements and satisfying regulatory requirements.

Conclusion

Building reliable, scalable Conversational AI agents demands a smart approach that blends control, speed, and flexibility. The aim of the OpenDialog framework and no-code AI Agent management system is to make it easy for organizations to create powerful AI agents quickly, without compromising on reliability or efficiency.

We achieve this by starting from the solid conceptual framework and then building tooling around it that optimises our capability to design conversational AI Agents that make best use of that framework. Finally, rich data provides the necessary insights to drive improvements and support compliance needs. The end result is the ability to deploy and refine AI solutions quickly, measuring results in weeks, not years.