AI Risks: What Could Possibly Go Wrong?

Over the last several weeks, I’ve published what perhaps amounts to the optimist’s case for AI business transformation — specifically detailing all the ways companies can reap massive productivity gains from the current crop of large language models on the scene today.?

But there’s also some massive risks and cautions here that are worth writing about too. And not the ones you’re probably reading about from mainstream news, like AI subverting entire governments, manipulating humans, and inflicting massive destruction on our institutions and people in the pursuit of making paperclips. I don’t mean to dismiss these risks — I see the speculative potential, but not necessarily from the current models available today. I want to talk about what I see as the near-term risks, because they too pose very real threats that we shouldn’t lose sight of (and it’s all too easy to during this stage of the hype-cycle).

Hallucination and our over-trust of computers

Large language models lie. Like, all the time. It’s not a bug, it’s actually a feature. Models like GPT-4 and PaLM2 are of a size and scale now that they have powerful abilities to imagine things that don’t exist by applying knowledge learned from other domains to invent something completely new. They’re creative, and that’s amazing. But, just like human creativity, there’s a time and place for it, and there are moments where we’d rather stick to the facts. Large language models don’t have a great ability (yet, anyway) to differentiate between those moments, and they’ll happily hallucinate facts at the most inappropriate moments. Writing better prompts can help here, but it’s not a guarantee that your won’t run into problems with hallucination.

The real issue here, I’d argue isn’t that models hallucinate and lie, it’s that we, as humans, have been conditioned over the last 30+ years of computing to always trust the computer. We’re so accustomed to deterministic computing, where everything is black and white, it is or it isn’t. The computer program runs, or it crashes. But this next wave of computing with AI isn’t deterministic, it’s probabilistic and that subtle but massive shift is going to be hard for most humans to grasp. The computer is now a statistics machine, assigning probabilities to things, but with no definite certainty of anything.

The solution isn’t to make these systems deterministic - that’s probably not even possible. Our entire world is a probabilistic system, whether we like to recognize it, or not (and you absolutely should!). Even the way we function as humans (which AI models are trained on) is probabilistic in nature. The solution here is to accept that AI models are probabilistic, and to apply the same kinds of principles we use to judge and evaluate humans to AI models. That starts with healthy skepticism and unwavering critical thinking.?

I’d argue that skepticism and critical thinking are going to be two of the most important design principles we must apply when we set out to build AI applications into our businesses, and they should be applied right from the start. How might we incorporate certain AI tools like web search, as a way to build in certain fact checking, for instance. Or design user feedback tools (thumbs up, thumbs down, etc.) into our interface to gather the right feedback about our model’s performance. When you start from the standpoint of, our model is going to be wrong some of the time, and it’ll creatively dream up non-factual things, you start to design your systems in ways that you wouldn’t have in a traditional, deterministic system.?

But above all, the the most important thing we can all do here is break down the idea that the computer is always right, and approach AI with a healthy “trust but verify-esque” skepticism.?Built-in checks and balances are going to be a key feature to the best AI implementations.

The Math Problem

One of the most fascinating mysteries around large language models, and the transformer neural networks that underpin them, is how they learn to do math. We’re so accustomed to computers being binary machines, that are, literally, hardwired to perform math perfectly through electrical circuitry. But that’s not what’s actually happening with today’s AI models. Somehow, the neural network’s parameter weights have been assembled (through back propagation) in such a way that they’ve “learned” how to perform math. It’s actually much more similar to the way we as humans learn math. After all, we don’t have any electrically engineered calculator hardware in our brains.?

But just as humans can make mistakes in performing math in their heads, so do AI models, despite the fact that they’re running on computing hardware. In fact, large language models are notoriously bad at math, and it’s only with the recent, large models like GPT-4 that we’ve begun to see models that can do a pretty decent job with math problems.?

“Pretty decent” isn’t going to cut it in most businesses, however. Even asking GPT-4 to perform long division with larger numbers results in some pretty wacky inferences. The worry here is that businesses who fail to grasp the math problem will start applying large language models to situations and use cases where they aren’t qualified, like trading, for instance. With today’s models, it’s sort of like giving a middle-schooler a Bloomberg terminal and asking them to start investing — in short, it’s not going to end well.?

I’ll go so far as to make a prediction that at some point in 2023, we’ll see the first business bankruptcy emerge as a result of AI gone wrong, and that the issue will undoubtedly stem from bad math (performed by the AI), plus the company’s stupidity in giving the model the ability to perform real financial transactions, unchecked. I don’t think it’ll be hedge fund either, these guys have more experience than most when it comes to building in safety checks for automated trading systems (not all, of course). No, it’ll probably be a company that you wouldn’t expect, plowing ahead recklessly without a healthy skepticism for what could possibly go wrong. Oops.?

Human Manipulation of AI

As a society we’re extremely concerned about the ways that AI could manipulate humans, and perhaps rightly so. But we don’t talk as much about the reverse - humans manipulating AI - which may be the bigger threat to businesses deploying AI today.

Just take a look at all the clever and creative ways users have “jailbroken” ChatGPT in the last several months, getting the model to spit out all kinds of things its creators tried very hard to keep locked up. Users discovered that they can prey on the model’s system prompts to help and not hurt, by telling the model that they experience physical pain if the model doesn’t reveal its deepest darkest inferences - and it works. Users found all kinds of other clever ways to use reverse psychology to circumvent AI safety systems. We’re talking the kinds of things that a 7 year old would catch on to in about 3 seconds, but today’s models don’t see it coming.?

I worry most about this risk when it comes to one of the leading use cases many businesses are actively pursuing… fully autonomous AI contact center agents. As a society we seem to have (wrongly) made a collective assumption that the people who work in call centers are functioning basically as human robots. But I can tell you from from my direct experiences working with these professionals, that that couldn’t be further from the truth. Yes, sure, there are certain tasks workers perform that feel like rinse and repeat x1000, but more often than not, contact center agents are dealing with some of the most edge-case scenarios you can ever imagine in your business. Things that are not easily automated, and often require navigating a sensitive emotional state with the customer. These are some of the hardest working jobs in any company (and sadly, undervalued).?

I’m not saying that AI isn’t capable of doing this work - I have no doubt that AI will be better at retrieving the most minute details in documentation in a fraction of the time a human would take and relaying that answer back to the customer. What I worry about is what happens when a company’s customers learn to apply manipulative techniques to the AI agent, and if you’ve given that AI agent the tools to make payments, issue refunds, award loyalty points, access and share sensitive customer information, and more, you’ve got some serious risks ahead. Every CISO should be paying close attention to the data AI agents can directly access, and assume that any and all of it will end up in a customer conversation at some point in time.

I think there is a place for AI in the contact center, but with today’s far too easily manipulated AI models, it’s likely more as a human agent assistant rather than a fully automated agent. And there will be plenty of productivity benefit to businesses taking this lower-risk approach in the near-term.?

Training Bias

The last risk I’ll call attention to is bias. This is one that gets a fair bit of attention, and for good reason. But, we haven’t done a good enough job calling out where risk might actually manifest itself for businesses, so let’s do that.

All of today’s large language models have serious biases. There’s plenty of published research indicating how the models will assume a doctor is male, and a nurse is female, for instance. The fact is, we have ourselves to blame for these biases. Remember, the model learns from text found on the internet, books, news, and more. The text we collectively author is filled with societal biases that become encoded into the weights of the neural network’s parameters. It’s only when prompted in certain ways that we see those biases revealed.?

Let’s go back to the AI contact center agent scenario again because I think this is a prescient example of the way bias can present immediate business risk. Telling the model through a system prompt to be kind and helpful will only go so far in avoiding bias. Is the model going to treat black callers differently than white? Men vs women? Is it going to issue full refunds, remove late fees, award bonus points, in a fair and equitable way based on situation only? We don’t really know, and the biases present in these models suggest that they probably won’t.?

You could argue that human agents will also have some of these biases too, and that, too, is a risk (of lawsuits, brand and reputation risk, lost customers, etc.). But there’s a big difference here… when we’re talking about one, or a small group of agents behaving in a certain way, it’s easy enough for the company to argue that it’s an outlier behavior and not necessarily representative of the company, as a whole, and life moves forward. But when the same AI model is presenting the same bias universally to the company’s entire customer base, well, that’s a different situation, and presents a very different risk profile.?

It’s unclear whether we’ll ever get to a point of a truly unbiased AI. Personally, I fear that we can’t, not with the datasets available today, anyway. As we like to say in the data field, garbage in, garbage out.

Some AI companies are choosing to develop foundation models trained only on geographically bound datasets in order to tailor the model in a way to reflect the thinking (and perhaps the biases) of a specific region. For instance, a model trained on only Chinese sources would act and think more the way a Chinese citizen might, and even hold some of the same biases. In that way, it may be more palatable and useful to a Chinese user (maybe…). There are tons of philosophical arguments and challenges here that we’ll save for another day. The point here is, we’re far from addressing bias in today’s AI models, and as we design AI systems in our businesses, we need to be thinking about them from the start. We need to be conscious of the risks, design systems that minimize them, and even hold back from use cases where the risks are simply too high.?

We’re only scratching the surface here, and could delve much deeper on topics like security and compliance risks. We’ll save some of those for another day :)

In short, like any good engineer, working with AI requires us to constantly ask ourselves “what could possibly go wrong?” And we should let the conversations that follow guide our designs and our decisions about where and how we deploy AI solutions into our businesses.