Master assistant or multi-assistant orchestration? 4 considerations

When you start working on expanding your enterprise conversational automation initiative, you’ll be faced with a dilemma.?Your goal should be, over time, to get your conversational assistant across all of your channels, spanning all of your use cases. That’s how you scale: increase coverage (use cases), increase availability (channels), increase adoption (users).

To achieve that, the dilemma is: do you build one big, massive, master assistant that can handle all of your intents and use cases across all channels? Or, do you break your assistant down into a series of smaller, narrow, more specific assistants, and tie them altogether with a orchestration layer on top?

What is multi-assistant orchestration?

Multi-assistant orchestration, also know as multi-bot orchestration is where you have a standalone 'bot' or assistant that takes requests and utterances from users, and then 'hands off' to smaller, tightly scoped 'bots' or assistants for the request to be fulfilled.

For example, let's say you're a retailer, you might have 3 different conversational assistants. One that handles product research, one that handles sales and another that handles customer account management. In order to make these 3 use cases available on your website via a single access point i.e. a single chat window, you'd need a fourth assistant that sits on top and in front of these 3 assistants. This fourth assistant is the 'orchestration layer'. It would take the customer request, then determine which of the 3 'sub-assistants' is best placed to handle the request. It'll then 'handover' to the best sub-assistant for the job.

From a customer's perspective, it'll seem like its just one single conversation. However, architecturally, the user is being passed around a number of different assistants.

Multi-assistant orchestration: a growing trend

A growing number of enterprises are opting for this orchestration of multiple assistants strategy. However, in my experience, this is an emergent, rather than purposeful strategy today.

It emerges as the business begins to?mature?and understand the value of conversational AI. They set up a centralised centre of excellence to coordinate and manage CAI implementations.

This group usually realises quickly that other parts of the organisation have been building and deploying chatbots or digital assistants in other corners of the business. Not wanting to disrupt what’s already in play, they aim to at least consolidate the access point by adding an orchestration layer. Over time, they seek to centralise on a specific platform and keep this orchestrated approach to put all of these various bots under one roof so that users can use all functionality from one access point.

Another way orchestration emerges is when a business has built one big, master assistant for one channel, such as a chatbot that handles 100 uses cases. As more use cases are added, concerns arise about the assistant’s performance or the long term viability of adding to such a large application. The question arises: should we split it up?

One master assistant or multi-assistant orchestration?

Rather than having this question emerge out of your natural maturity, it’s always prudent to consider these things up front so that you can plan ahead and implement the most appropriate solution purposefully.

So here are four considerations for teams thinking of the best way to scale their assistants across channels and use cases. Here, you should be able to answer the question: master assistant or orchestration?

1. Intent conflicts.

With one master assistant, you may have intents across different use cases that are similar, and therefore develop confusion in your language model. If you try to cater for this within one master assistant, then your language model will get confused and will not be able to ascertain which intent the user is trying to invoke.

This can open up a barrel of problems that, if you don’t have sophisticated?NLU?monitoring and testing, you might not even know about. But, your users will.

The importance of NLU QA

I wouldn’t recommend going down the master assistant route without extremely good quality assurance and testing on the?NLU?side. If you already have a large assistant with hundreds of intents, I’d recommend doing some testing to see whether you have conflicts and whether those conflicts are easily resolvable or whether segmentation of use cases is needed.

If you’re designing your assistant from scratch, you should be monitoring and testing your?NLU?performance throughout development. Once you’re live, with every new use case, you should be monitoring?NLU?performance to prevent conflicting intents from occurring in the first place.

If and when you reach a point where you’re beginning to negatively impact the rest of your language model by introducing new intents, if you can't clean it up and maintain the model's integrity, then you should consider segmenting your assistant into a series of specialist bots with an orchestration layer on top.

2. Channel interfaces.

Different channels have different interfaces and different functionality that they support. For example, with WhatsApp, you do not have the luxury of buttons or carousels. You do have this functionality in some chat interfaces, but you don’t with a voice interface.

Therefore, having one single solution, with one language model, deployed across multiple channels can often require you to provide for the lowest common denominator in functionality (or do a whole load of code customisations). This means that you won’t be providing the best experience on any channel.

That’s not to say that you can’t keep the conversation broadly consistent across channels. Your conversation design patterns, your design specification artefacts (you can keep one artefact, but specify the differences in responses by channel) and the assistant’s personality can all remain.

3. Channel suitability

Often, your use cases can be mirrored across channels, broadly speaking. Though, some use cases might lend themselves to one channel rather than another. For example, if you need to enable a user to send you a photograph of their driving licence, this can’t be done on a phone call without opening a parallel channel like chat or SMS. However, it can be done via WhatsApp in the same channel. Requirements like this will shape your ability to have one assistant on the backend deployed through multiple channels on the front end.

Designing for new channels

If you’re debating deploying your master assistant on another channel, I’d recommend against just deploying it as it is. Instead, you should first review the demand for use cases on that channel by analysing transcripts.

If there’s an overlap in use case demand, I’d then recommend sitting down, designing and testing the ideal experience for that channel, taking into account the specific interface and mental model differences of that channel. Then, review whether the use cases and interactions you provide in your current assistant can be iterated to deliver your desired experience in your target channel, without diluting the experience on your current channel.

If you can tick both of those boxes (use case overlap and ability to provide the optimum experience for the channel) without impacting the performance of your current assistant, then keep your master assistant. If not, it’s time to consider splitting things up.

4. Language model differences

Similarly to the differing interface requirements of a new channel, you may also find that the language people use in one channel is different from another. This will require you to consider whether a single language model can underpin all interactions across all channels.

For example, in WhatsApp, your users may use short, few-word prompts, and your language model will need to reflect this. With a voice user interface, people are more verbose and also speak differently to how they type. You also have the added challenge of extracting accurate speech from your?Automatic Speech Recognition (ASR)?engine. Issues like this will impact how you construct your language model.

How find out whether a new language model is needed

If you’re considering expanding your assistant from one channel to another, you should first do the following:

1. Take some transcripts from live human conversations on your target channel
2. Perform some topic clustering to highlight key themes and intents
3. Review whether there are any differences in topics and intents between the new channel and your existing model
4. Test your existing NLU against the channel-specific utterances from overlapping use cases and intents
5. Review your model’s performance

If you see that the training data you’ve gathered from interactions on your target channel performs well against your current training data in your current model, you can be confident that deploying on that channel without much change to your language model is feasible.

If you see that your current model performs poorly, there are two things you can do:

1. Test how your model would perform if you were to include some data from your new channel into your existing model. This should be done iteratively and with close monitoring because there’s a big risk that by adding new training data from another channel, that it impacts your existing model in other areas. If you can include training data from the target channel in your existing model and maintain performance across intents, then you can launch on a new channel without requiring a new language model.
2. If including training data from your target channel in your existing model negatively impacts performance, the only other option you have is to create a new language model with the training data from your new channel. This is the equivalent of creating a new assistant for this specific channel. If there’s no overlap between training data on each channel, this is the best and safest option.

What's the right approach for you?

Following the 4-point process above will help you understand whether a master assistant or assistant orchestration strategy is the right approach for you. I recommend considering this throughout your CAI strategy implementation, with every addition of a new use case. You're far better off planning for this, than having to unpick a 4-year old master assistant.

However, if you do have a large assistant, you shouldn't be afraid of this process. The short term pain will be worth the long term gain. I think we'll see a lot more of these multi-assistant orchestration implementations in future as teams continue to mature.

---

About Kane Simms

Kane Simms is the front door to the world of AI-powered customer experience, helping business leaders and teams understand why voice, conversational AI and NLP technologies are revolutionising customer experience and business transformation.

He's a Harvard Business Review-published thought-leader, a top?'voice AI influencer'?(Voicebot and SoundHound), who helps executives formulate the future of customer experience strategies, and guides teams in designing, building and implementing revolutionary products and services built on emerging AI and NLP technologies.

• Subscribe to my newsletter
• Listen to the?VUX World podcast?on?Apple,?Spotify?or wherever you get your podcasts
• Take our free conversational AI maturity assessment