AI and the Fallibility Double Standard

Autonomous AI agents have begun to take over entire tasks start to finish. And this is just the beginning. We expect to see a plethora of such agents in the next half decade, and these will take on all sorts of tasks that humans currently perform, if unhappily.

In many ways the machines will be better at these tasks than we are. Self-driving cars have superhuman sensors, reaction time and true multitasking, and they will always apply their full attention to the task at hand, driving! Our AI scheduling assistant Amy Ingram has an equally strong sensory input and will obviously have total recall (the ability to remember every single person you met with and how it was negotiated) and the ability to read at speeds not possible for you and I.

At times these systems may seem omniscient. A human assistant only works for one person and so is blind to everyone else’s calendar and preferences; Amy (and her brother Andrew) can work for everyone. If she schedules a meeting for two x.ai customers, she knows the preferences of both parties and can optimize for both (we call this scheduling nirvana because we are geeks!).

And yet, having these superpowers does not mean these systems are or ever will be infallible. Human assistants make mistakes all the time. Human drivers make them far too often, with disastrous consequences. Can we expect our machines to perform these tasks flawlessly? Should we? How do we know who to blame when something does go wrong? How do we turn mistakes into productive learning?

The Inevitability of Machine Errors

Tolerance for error will depend on the function of any given autonomous agent. For self-driving cars, we’ll have a very limited tolerance for mistakes, since any given mistake can endanger human lives. In fact, Google is advertising its self-driving cars as a safety boon, since human error accounts for 94% of all accidents; however, the company doesn’t  project a zero error future. In some very deft communications, Google reports all of its self-driving car accidents by month with an analysis of what went wrong in each. This sets consumer expectations long before anyone can purchase such a vehicle.

Andrew and Amy must schedule your meetings nearly flawlessly, so that you don’t end up waiting alone at Starbucks while your contact waits at another one two blocks over. But Amy may occasionally inconvenience you in trying to balance competing objectives. (For example, to get a meeting on your calendar on Thursday, as per your request, she may have to schedule it 30 minutes outside of your preferences.) And her defiance of your scheduling preferences may appear, at first blush, as a “mistake.”

The Blame Game

How to attribute blame is, in some ways, a more interesting question. When people we’re close to make mistakes, we tend to give them the benefit of the doubt, or at least we try to. If it’s a first-time mistake or a minor one (replying “all” when you meant to reply to sender), we reason that we too might have done the same (or have done something similar in the past). Because we empathize, we are not so quick to blame or judge these human foibles. We can even be protective of the mistake maker, imagining they are in the right, and it’s the other party that has erred.

We see a different pattern with our AI personal assistant. If something goes awry, customers often assume the machine is at fault. But when we research most “mistakes,” the majority have been made by Amy’s boss, the guest or is simply not a mistake at all.

A recent study by researchers at the University of Wisconsin confirms our anecdotal findings. Their data showed that people will “fire” an automated advisor after a single mistake but will allow their human counterparts many more mistakes before ending the relationship. It’s a glaring double standard—no one would fire a human assistant or advisor after a single mistake—that really only hurts us, since even intelligent agents don’t have feelings.

We’d be much better served by curiosity about the root of an error. These systems can only learn when a mistake is correctly identified and its causes understood. Assuming the machine is always wrong is no more helpful than assuming it’s always right.

Forgiveness

If we imagine AI agents as being more akin to employees or team-mates than apps, our relationship with them will need to be similarly dynamic and two sided. In this context, setting preferences marks the beginning of a relationship which will require us to communicate when things change or go wrong, with the expectation of some communication back from the system. You can think of it as granting these autonomous agents a sort of rudimentary self-consciousness; this should, in turn, enable us humans to extend compassion to AI agents and to, on occasion, accept the agent’s choice even when it’s not aligned with our own world view. We’re designing Amy this way, and others are doing the same.

As Alexander Pope put it, “To err is human; to forgive, divine.” If we hope to get the most out of our AI agents, we’ll need to learn how to extend forgiveness to them as well.