Why we need a no-code multimodal orchestration tool
Kane Simms
??Triple Award-Winning AI Automation Consultancy ?? VUX. helps businesses leverage AI… Properly.
During Ben's chat with Jason F. Gilbert, Jason expressed frustration at how hard it is to orchestrate robotic movements with voice, sound effects, lights and other modes of expression.
This matters, and not just for Jason, who is the Lead Character Designer at Intuition Robotics, where he works on ElliQ – ‘the sidekick for healthier, happier aging.’
There are many more designers working in robotics. The same challenge will face conversation designers who work on digital humans as well – they also use various modalities to express themselves.
How can we design characters that use all their modes of expression to convey meaning? That’s what humans do. Words are only one component of how we communicate – we use much more to express ourselves.
No tools for the job
Here’s what Jason said (edited for brevity):
“There’s not a single platform for designing multimodality experiences on robots. I've talked a lot about this. I've asked a lot of people about this - people from different robotics companies. No one has this. Ideally, we would have some kind of Voiceflow, or some kind of no-code tool, where you can just go ‘okay, this is where the lights come in’. And now when [ElliQ] says this line, she also has this sound effect, this gesture and this thing, and that would be amazing. But you don't have that [tool] right now.”
That’s a huge challenge for designers like Jason. Conversation designers should be able to orchestrate every facet of a bot’s expressive repertoire. That’s how we communicate! We don’t use words with a little garnishing of something else. Sometimes words are the garnishing in what someone means.
In order to design a bot’s expressive language, we surely not only have to consider what’s natural for a human to understand, but also what new possibilities a bot has for expression that humans don’t have (such as lights or the buzzes, whirs and clicks a robot's motors make while they move).
Where are the solutions?
It’s also a huge opportunity for anyone who wants to design such a tool. It would need to allow designers to orchestrate everything a bot can do together, so that we can combine different modalities that let bots express themselves in various ways.
Our industry needs this. Bots and digital humans will be incredibly dull conversation partners if their facial expressions and body language express nothing. Without this tool the other modalities could easily just become window dressing if they’re not expressing what the bot means.
It could be worse if their various modalities contradict each other. Imagine if a user asks whether it’s time to take their daily medicine, and the bot nods it’s head (rather than shaking it) while saying “no”. Then they’ll confuse users, and it could be dangerous.
How do other industries do it?
It’s so easy to fall into the trap of thinking every challenge in conversational AI is brand new. That’s not always the case.
Animated films and videogames have workflows where every facet of a character is considered and brought to life, with varying results.
Check out this video on the making of Rango. Whereas actors will usually just be brought in to replace temporary voices for animated films, on Rango the actors were filmed together on a motion-capture stage acting out every scene. Those performances were the source materials for the animation, so each actor’s body movements, facial expressions and voice were captured before being applied to their CGI character. As you can see, it brings the characters to life and they’re so expressive!
领英推荐
Compare Rango to Fireman Sam. Both are CGI, but watching Fireman Sam is like watching wooden marionettes. Their body language is often redundant. Their facial expressions say very little. The character's potential for expression with their bodies and faces hasn't been exploited at all.
Why can’t we make bots and digital humans that are as expressive as a character from Rango?
What’s suitable for our industry?
Of course, most of us aren’t making entertainment products. Our bots often have important roles to play. They have to empathise. They have to build trust. They have to sell things. They have to represent a brand and its values.
You could say in those cases the stakes are higher than with entertainment. When someone watches a film they don’t like, they can ask for a refund or moan on social media, and then the story is over. On the other hand, if someone has an underwhelming conversation with an AI assistant, then they might never talk to it again or stop dealing with that brand.
For a companion bot such as ElliQ trust is paramount. The user and bot communicate with each other, and a relationship grows from those exchanges.
So, where’s the tool to help conversation designers orchestrate a bot’s multimodal expressions? We’re not animators and we don’t have mo-cap studios or actors. We shouldn’t have to learn every trick a CGI animator knows to do this, and yet it's our job to create excellent communicators.
Someone get on it! We need this.
Here’s my full interview with Jason – he gives many great insights.
You can also check out Kane’s interviews with Stefan Scherer and Danny Tomsett?for more on designing for robots and digital humans.
This article was written by Benjamin McCulloch. Ben is a freelance conversation designer and an expert in audio production. He has a decade of experience crafting natural sounding dialogue: recording, editing and directing voice talent in the studio. Some of his work includes dialogue editing for Philips’ ‘Breathless Choir’ series of commercials, a Cannes Pharma Grand-Prix winner; leading teams in localizing voices for Fortune 100 clients like Microsoft, as well as sound design and music composition for video games and film.
---
About Kane Simms
Kane Simms is the front door to the world of AI-powered customer experience, helping business leaders and teams understand why voice, conversational AI and NLP technologies are revolutionising customer experience and business transformation.
He's a Harvard Business Review-published thought-leader, a top?'voice AI influencer'?(Voicebot and SoundHound), who helps executives formulate the future of customer experience strategies, and guides teams in designing, building and implementing revolutionary products and services built on emerging AI and NLP technologies.
Advancing Voice UX design one word at a time.
2 年Before a tool, don’t we actually need some sort of semantic representation that combines linguistic and spatial communication? So, instead of simply NLU, we need Natural Communication Understanding. If I point to something instead of saying “the brown chair” or nod instead of saying “yes” the robot needs to understand that. Once we have this, then we can start talking about the kind of tool that can deliver it. And, if we do this correctly for a real world “spatial” context, wouldn’t it also be used for what others are planning to do in the Metaverse? Now, there’s got to be people smarter than us (yeah, hard to believe) working on this sort of thing. Where are they? Anyone else thinking we need something like this?
To write software that users will adore.
2 年Kane Simms In order to get there I think you need to see some things such as: standards - tools have work together; (multi-disciplinary) teams that understand both the creative side and can write code. It is difficult to write a good tool, otherwise. *It’s so easy to fall into the trap of thinking every challenge in conversational AI is brand new. That’s not always the case* From what I notice is 1. folks in voice don't look at adjacent industries for insight. I remember talking to a veteran game script writer and they asked "Andrew, why don't you use a narrative engine?" 2. folks in voice, at least in the customer facing end, really don't look at what has been done in the past.
Founder and Chief Creative Officer
2 年The trick with conversational + multimodal is the same as the trick with the web. It took 5 years after NCSA Mosaic premiered for "web standards" to officially become a movement, and 10 years for Zeldman's seminal book. We're still about 18 months from the 10 year anniversary of the drop of the first Echo device, and more from the beginning of really popularizing multimodal voice experiences. Honestly, I've come to believe that Amazon's APL is like Microsoft's ActiveX. It got a good deal of adoption because of the wide use of the platform, but created a ton of apps that couldn't be run on anything other than devices using old versions of their software. This turned into a huge technical debt for Microsoft. They've spent TEN YEARS trying to migrate enterprise customers off of being IE dependent and to use web standards. Even with Edge now sharing a LOT of DNA with Chrome, Microsoft still struggles with the adoption of Edge, partially because of how IE haunts it. Not arguing for one-stop shopping, but a similar maturation of the field with open standards.
genAssess - AI Skills Assessments | Conversational AI & Automation Specialist | Founder, Speaker, Educator & Problem Solver in Recruitment and Talent Tech
2 年Couldn't agree more here. It's about time we think about the amalgamation of multi-modal experience design. Great article Kane Simms, Benjamin McCulloch & Jason F. Gilbert
Conversational AI | Generative AI | Opus Research Senior Analyst
2 年There’s Rapport and there’s Cocohub, as two examples of doing pretty much what you describe.