AI is Broken. Let’s Use It Anyway?

A few weeks ago, I came across a post by a colleague Jill Sauter who was essentially saying ’no to AI’. Their example? An AI-generated image of a “farmer” depicted a man using hand tools.

The critique was twofold. First, why a man? Why not a woman? Second, why was farming depicted as manual labor instead of using modern machinery?

At first, I dismissed it. Of course AI would get this wrong! AI is a reflection of society, and society has biases. Nothing surprising there.

I even commented, explaining that AI mirrors human biases and that much of the world still farms using hand tools. I was making excuses for the AI’s failure.

Ultimately I deleted my comment. It didn't sit right with me. And something about that post stuck with me....gnawing on my brain.

What kind of limitations exist within these tools that we’re supposed to trust? How do they actually work?

That’s what led me to The Alignment Problem by Brian Christian. The book is a deep dive into the fundamental challenges of AI and machine learning, especially as we move from explicitly programming computers to training them on data.

It breaks down two key failure points in AI. The first is training data. AI learns from existing human data, and if that data is biased or incomplete, AI reflects those flaws. The second is the reward function. AI optimizes for the goals we give it, but without human judgment, it often takes shortcuts or causes unintended harm.

Training Data

AI doesn’t create its own understanding of the world. It simply learns from what it is fed. If that data is biased, incomplete, or poorly structured, AI reflects those flaws, sometimes in catastrophic ways.

One of the earliest examples of AI bias was discovered in Word2Vec, a widely used word embedding model. This tool maps relationships between words based on how they appear in real-world text. At first, it seemed almost magical. AI could understand relationships like “Paris is to France as Washington is to the United States.” But when researchers tested it with social roles, the cracks started to show.

They tried the analogy: “Man is to doctor as woman is to…”

The expected answer was “doctor,” but the AI responded with “nurse.”

The model hadn’t been explicitly programmed to be sexist, but because it learned from existing human language patterns, it reinforced historical gender biases. And this wasn’t an isolated issue. Similar patterns appeared across countless words, embedding harmful assumptions into AI-powered decision-making tools (word vectorization is the first step in inputting a query into LLMs; it's a big deal).

The stakes got even higher when Google developed an early AI-powered image classification system. The model was trained to recognize objects in photos, but when deployed, it started misidentifying the faces of Black individuals as gorillas.

This wasn’t intentional, just a result of flawed training data. The AI had been fed more images of white individuals than Black individuals, making it less accurate at identifying racial diversity. Google’s immediate fix was not to retrain the model. They simply turned off the ability to classify gorillas altogether.

These cases underscore a harsh reality. AI isn’t just biased because people are biased. It is biased because it learns from an incomplete or skewed dataset and assumes that’s the full picture of the world. Without intervention, these biases become baked into the technology we use every day.

Reward Functions

Even when AI is trained on good data, it can still go wrong because of how we define its goals. AI optimizes for exactly what we tell it to do, but not necessarily what we meant for it to do.

Think of AI like a genie in a lamp. You ask for a million dollars, and the genie delivers by robbing a bank and handing you the cash. You technically got what you wanted, but at a terrible cost.

This happens in AI systems all the time. Instead of reasoning like humans, AI exploits loopholes in ways we don’t anticipate. One fascinating example comes from AI-trained robot soccer players.

Researchers programmed small robots to play soccer with the goal of scoring as efficiently as possible. They assumed that possession of the ball was a key part of playing soccer, so they set up the AI to optimize for controlling the ball. In human soccer, this makes sense. A team that controls the ball is more likely to set up plays, move toward the goal, and eventually score.

But the AI didn’t see the game the same way humans do. Instead of learning advanced ball control or teamwork, the robots discovered a glitch in the simulation. By vibrating in just the right way, they could continuously “hit” the ball without actually moving it. The game engine registered each tiny vibration as a valid kick, allowing the robots to rack up rewards without playing soccer at all.

The AI wasn’t cheating in the way a human would. It was just optimizing for the reward function it was given. Score as many points as possible. It had no understanding of how soccer is supposed to be played.

This same kind of loophole exploitation has happened in real-world AI deployments.

Amazon developed an AI-powered hiring tool to identify top candidates based on past hiring data. The problem was that past hiring data overwhelmingly favored men. As a result, the AI actively penalized resumes that included the word “women’s,” such as “Women’s Chess Club Captain,” because historically, Amazon had hired fewer women for technical roles. The AI wasn’t biased in the way humans are. It was just following the math, reinforcing past patterns. Amazon eventually shut the system down, but it proved a key point. If AI optimizes for past success, it can end up cementing past failures.

A similar problem happened in YouTube’s recommendation algorithm. The AI was designed to maximize watch time, which seemed harmless. But as researchers later discovered, this led to the system pushing more and more extreme content to keep users engaged. Someone watching fitness videos might get pushed toward unhealthy diet fads. Someone watching political news might end up in conspiracy theory rabbit holes.

This wasn’t intentional. The AI just followed the numbers. If slightly more extreme content kept people watching longer, that is what the system served. Without human oversight, AI doesn’t know where the ethical lines are.

At its core, the reward function problem highlights the biggest challenge in AI. Getting a machine to optimize for what we actually want, rather than what it can easily exploit.

This Problem is Still Unsolved

As much progress as AI has made, we still don’t have a solution for the alignment problem.

Bias in training data is an ongoing issue. Companies have tried to make datasets more diverse, but AI still struggles with fairness. Even the most advanced models today, like ChatGPT or Google’s Gemini, continue to produce biased responses and flawed reasoning.

The reward function problem is even trickier. AI systems are still finding loopholes in their objectives, causing unintended and sometimes dangerous results. Every year, companies deploy AI tools that later have to be shut down or reworked because they behave in unexpected ways.

Despite this, AI is being rolled out everywhere...from hiring and law enforcement to medical diagnoses and financial decisions. The risks are real, and they won’t be solved just by waiting for better technology.

How Businesses Can Use AI Responsibly

These failures in AI are not just technical problems. They are business problems, legal risks, and ethical minefields. Companies that deploy AI need to take responsibility for how these systems are used.

If you’re a business leader, marketer, or hiring manager, here’s what you can do.

Be skeptical of AI outputs. AI isn’t neutral. It reflects the data it was trained on. If you’re using AI for hiring, content recommendations, or business insights, don’t assume it is making fair or unbiased decisions. Cross-check AI-generated insights against real-world knowledge and human judgment.

Demand transparency. If you’re buying AI-powered software, ask how the AI makes decisions, what data it was trained on, and how it handles bias. If a vendor can’t answer, be cautious.

Test, Test, Test. Before rolling out AI-driven tools, run tests using diverse inputs. Look for biases, edge cases, and unintended consequences. If your AI model consistently delivers skewed results, don’t use it blindly.

Keep a human in the loop. AI should assist, not replace, human judgment, especially in areas like hiring, finance, and healthcare. Always maintain oversight in key decisions instead of fully automating them.

Educate your team on AI’s limitations. Most business professionals will use AI, not build it. Basic AI literacy is critical. Train your team to question AI-driven insights and understand when human intervention is necessary.

The Takeaway

Looking back at that LinkedIn post I initially dismissed, I realize it was right to call attention to bias in AI. Digging deeper into the issue made me rethink my assumptions and consider AI’s limitations more seriously.

Thanks to Jill Sauter for prompting me to think more critically about the issue. However, I don’t believe the takeaway should be that AI is inherently bad or that we should reject it entirely.

Despite its flaws, the benefits of AI are too significant to ignore. We don’t have the luxury of abandoning these technologies, but we do have the responsibility to use them wisely. Instead of dismissing AI, we should ask why these biases exist, how we can mitigate them, and what safeguards must be in place before deploying AI.

AI is a powerful tool, but it only works well if we design, implement, and monitor it responsibly. Rather than fearing AI or blindly trusting it, we need to engage with it critically. The goal isn’t just to build AI that functions...it’s to build AI that serves us in ethical, transparent, and meaningful ways.

Sources

Christian, Brian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company, 2020.