Judging the Machines: How Do We Measure What’s Beyond Us?

Have you ever written the lyrics of a song??

Have you ever written a poem??

Both examples were outside of the capabilities of most of us - especially including me.?

Now, I am able to create complete songs with lyrics, music and vocals that make people cry - in a positive way.?

Of course, I am using a number of GenAI tools to “support” me. Clearly the former heavy lifting is now not so heavy anymore. But there are other skills needed.?

This example brings me to the topic of our newsletter today.?

As I rely more on AI, a new question arises: If GenAI helps me produce results beyond my own skills, how can I accurately judge its output? And this isn’t just my dilemma—it’s a challenge facing anyone who interacts with AI today.?

Let’s explore this in more depth.

As we accelerate into the exponential age, we stand at the crossroads of human intellect and artificial brilliance. Generative AI (GenAI) has rapidly evolved from a novel tool into a force capable of producing content—text, art, and even decision-making outcomes—that rival, or in some cases, surpass human creation.?

Take the example of AI-generated art in the creative industry. In 2022, an AI-generated artwork titled ‘Théatre D’opéra Spatial’ won first prize in a digital art competition at the Colorado State Fair. The judges, unaware it was AI-made, praised the piece for its originality and technical brilliance. When they discovered the work’s origin, a debate ensued:?

Should AI-generated art be judged alongside human-made art? The challenge wasn’t just that the AI created a masterpiece; it was that the judges, despite their expertise, couldn’t discern between human creativity and machine innovation. This raises a critical question—how do we judge creations when AI can exceed human capabilities? And should traditional metrics still apply?

The human brain, shaped by millennia of evolution, is the gold standard by which we've evaluated everything, from art to logic. However, the rise of GenAI has forced us to reconsider that framework. AI, fueled by vast data and pattern recognition, can now generate results with remarkable precision, creativity, and nuance. Whether it’s writing a compelling essay or creating intricate art, the outcomes are often so impressive that even experts might find it difficult to discern if they’re judging an AI masterpiece or a human one.

This begs the question: if AI can now match or surpass human outputs, what qualifies us to be the judge?

The Hidden Irony: Humans May Seem Smarter Than They Are

One interesting paradox is that AI is designed to amplify our abilities, yet the gap between AI output and human understanding may expose our own cognitive limitations. Often, human judgement operates on a mix of intuition, experience, and knowledge, but these are not foolproof metrics. If AI can analyse data sets that no human can fully comprehend and produce better results, our role becomes less about judging the output’s quality and more about trusting a process we don’t fully grasp.

The irony is clear: while GenAI makes us appear more intelligent through collaboration, it may also reveal the boundaries of human intellect in a more striking light.

The “Expert Bias” Problem

Human biases also play a significant role in how we assess AI-generated content. Experts, in particular, may feel threatened or unconsciously sceptical about a machine’s ability to outperform their years of experience. This bias can cloud judgement and create resistance to acknowledging AI’s potential, even when the output is demonstrably superior.

In essence, as the quality of AI output becomes harder to distinguish from human work, our judgement could become increasingly subjective. We may favour human-made content simply because we believe it’s more authentic, despite its inferiority.

The Future Challenge: Judging What We Can’t Understand

With the exponential development of GenAI models, the gap between what AI can achieve and what we can comprehend is only set to widen. As we move forward, this challenge will shift from “can we judge AI’s output?” to “should we?” In the near future, we might need new frameworks, possibly other AIs, to help evaluate the work of these advanced systems. Human judgement, while valuable, could become one piece of a much larger, more complex puzzle.

In conclusion, we’re on the brink of an era where humans must grapple with the fact that the machines they’ve created may be beyond their ability to assess. GenAI’s potential is enormous, and as it continues to improve, it will not just change what we create but how we judge creation itself.

Call to Action

I’ve shared my thoughts, but this is just the beginning of a much larger conversation.?

How are you grappling with these questions in your work or personal life? Have you faced situations where GenAI’s outputs challenged your ability to evaluate them? I’d love to hear your stories, your perspectives, and your solutions.?

Let’s connect and discuss on LinkedIn—whether it’s to share experiences, debate these ideas, or explore the future of human-AI collaboration.?

And if you or your organisation are navigating these complex waters and need guidance on integrating GenAI, feel free to reach out. The journey is just starting, and we’ll need all hands on deck.

Until next time, stay curious and keep experimenting.?

Michael?

P.S. This article was partially supported by GenAI (and a new agent I employed).?