What ChatGPT Can Never Do

Never. As in never ever. And not just ChatGPT, but any AI. Sounds foolish to make a prediction for the indefinite future, especially for something evolving as quickly as the field of AI, but hear me out.

Just for kicks I asked ChatGPT the same question “What can ?ChatGPT never be able to do”? And as usual it gave a perfectly reasonable, if somewhat boring, response – “ChatGPT does not have consciousness or self-awareness. Additionally, it cannot have intentionality or have emotions”.

I’m sure ChatGPT is frequently asked similar questions about its intention for world domination, so you can be sure that the good folks at OpenAI have spent a great deal of time fine-tuning appropriately benign answers.

The Is-Ought Problem

But our story starts quite some time ago, with the 17th century Scottish philosopher David Hume. Hume framed the so called ‘is-ought’ problem which later came to be known as Hume’s guillotine. He asserted that statements can be of two kinds – ‘Is’ statements and ‘Ought’ statements.

‘Is’ statements or ‘positive’ statements are descriptive statements, about how the world is, how the world was in the past, or how the world will be in the future. Statements like “It was hot yesterday”. Or “It will rain tomorrow”.

Then there are the ‘Ought’ statements or “should’ statements. These are prescriptive statements about how the world should be or how we want it to be. They are statements about goals or values, and statements that expresses a relationship in which one idea is considered to be the correct or the desired outcome of another idea. “You should carry an umbrella” or “Everyone deserves to be happy” or “One should always try to be honest”.

Hume’s principle states that while it is possible to deduce ‘Is’ statements from other ‘Is’ statements, it is impossible to derive ‘Ought’ statements from ‘Is’ statements.

For example, the ‘Is’ statement “The driveway is wet today” can be deduced from another ‘Is’ statement “It rained last night”. But let’s say you have an ‘Is’ statement, “It will rain today”- then you might normally say “Therefore you should carry an umbrella” (an ‘ought’ statement). This sounds perfectly reasonable in regular conversation, but as pure logic goes, it actually relies on a hidden assumption – that getting wet is something that is not desirable.

Imagine this conversation between me (who can only speak in factual Is statements) and my friend Rick, who specializes in being a jerk.

Me: “Be careful with that hammer.”

Rick: “Why should I be careful with a hammer?”

Me: “Because you can hurt yourself.”

Rick: “So, why shouldn’t I hurt myself?”

Me: “But you will end up in a hospital.”

Rick: “And why shouldn’t I go to a hospital?”

You can see how this can go on forever. In order to derive the ‘ought’ statement “You should be careful with a hammer” you need to assume at least one other ‘ought’ statement, that going to the hospital is something unwanted.

This is Hume’s guillotine – you can never derive an Ought statement from an Is statement.

Alice In Wonderland

The previous discussion may seem unnecessarily theoretic, even contrived. But it is in fact critical to how we build and use AI models. Indeed, getting AI to have the same goals as humans has its own moniker in the AI domain, the alignment problem.

AI’s can be thought of as agents that take actions in the world in order to achieve goals. Intelligence is what enables them to choose ‘good’ actions. Intelligence requires AI systems of having or creating an accurate model of reality, using that model to make predictions about the future and evaluating the likely impact of alternative actions.

Which means AI systems have to ask questions like “What is the world like?”, “How does it work?”, “What will happen?”, “What will happen if X happens”, “What will happen if I take action Y?” and so on.

Notice that all of these questions are ‘Is’ questions. But they are all trying to answer the single ‘Ought’ question “What action should I take?” This is a dilemma because, as we have seen before, you can never get from an ‘Is’ statement to an ‘Ought’ statement.

It’s a bit like Alice in Wonderland who comes to a fork in the path and asks the Cheshire cat “Which road should I take?” The grinning cat answers “That depends on where you want to go”. An action makes sense only in context of a goal.

In practice we cross this bridge by teaching AI with proxies of human goals. These proxies take different forms - in reinforcement learning it can be called reward modeling, in linear programming these goals may be termed as objective functions.

Which brings us finally to what AI can never do – it can never self-identify goals. It can be trained to meet objectives but those objectives are set by people. Humans who bring their values, judgement and biases with them, all of which will ultimately reflect in the actions that an AI will take.

Implications for Decision Intelligence with AI

The Is-Ought problem tells us why decision intelligence and prescriptive analytics is so much more demanding than the more familiar predictive analytics. It is because while prediction is necessary for decision making, it is not sufficient. Decision making requires the agent to model open-ended ‘should’ questions not just factual ‘is’ statements.?

This also has implications for how we build Decision Intelligence with AI –

1.?Invest a lot more time deciding what your goals should be. Be very cautious when translating these goals in to objectives that your AI model learns from. ?For most real-world Enterprise AI problems, you have to navigate cross-functional tradeoffs, shifting goals over time and different goals for different business segments within the organization. This means that, in many situations, finding the reward function is the hardest part of the problem.

2.?Make goals as explicit as possible, even if (and especially if) they seem obvious.

3.?Incorporate sufficient flexibility in the model design for goals to evolve over time. This can be due to genuine business reasons as the external environment changes. But more often than not it is because people realize that what they really want is different than what they thought they wanted.

4.?Constraints are your friend. They are what keep goals realistic, prevent runaway models and reduce unintended consequences.

5.?Measure value against stated goals (built into the model reward function) but also unstated goals (metrics that you did not necessarily make part of the model reward).

?

Side Notes & Clarifications

1.?It is possible to distinguish between two types of goals – instrumental goals and terminal goals.

a.?Terminal goals are the goals you want ‘just because’ (no reason or justification). They are the final objectives that an individual or an organization wants to achieve. They are ends in themselves.

b.?Instrumental goals, on the other hand, are the goals you want because it gets you closer to your terminal goals. These are intermediate goals that help an individual or an organization achieve a higher-level goal. Essentially, they are a means to an end.

AI systems can self-generate instrumental goals. When I say that AI cannot have goals, I mean they cannot come up with terminal goals (at least not terminal goals that are always aligned with human terminal goals). Terminal goals must come from the outside, as an input.

2.??Plot twist - Not everyone agrees with Hume. Another philosophical view, called "moral realism" states that "ought" statements can be derived from "if" statements by appealing to facts about the natural world. If you accept this perspective then AI can even have terminal goals.

3.??Misalignment of goals (what we tell AI we want as compared to what we really want) has led to some well-publicized failures - Microsoft's chatbot Tay was designed to learn from and interact with users on Twitter. It was released in March 2016 and was programmed to maximize engagement with users. Within 24 hours of its release, Tay began to show racist, sexist, and offensive language, as it was exposed to and learned from toxic online interactions. There was no penalty for bad behavior which caused it to adopt the worst aspects of human behavior on the platform. It was taken down by Microsoft in just one day.

#ChatGPT #AI #Goals #Rewards #DecisionIntelligence #PrescriptiveAnalytics

Disclaimer: This publication does not represent the thoughts or opinions of my employer. It is solely based on my personal views and as such, should not be a substitute for professional advice