Theoretical & Practical (Current) Limitations of Generative AI

Do generative AI models like GPT-4, Gemini, and Llama 3, trained or currently being trained on our collective knowledge actually scale past our knowledge plateau? When do the improvements feel less jaw-dropping? When does a model hit peak maturity? Let's dive in.

Can AI Models Surpass the Human Knowledge Plateau?

In short, right now with what is available to the public, the answer is no. Generative AI models do not scale past our knowledge plateau in the sense that they don't generate original, groundbreaking insights or make discoveries. They are trained on existing data, which means their understanding and knowledge are bounded by what has already been discovered, written, or recorded by humans up until their training data was collected.

These models excel at synthesizing and summarizing the information they have been trained on, and they can generate text that appears insightful, coherent, and well-informed. However, the "insights" are ultimately repackaging of existing human knowledge. The model isn't able to conduct experiments, feel intuition, or have eureka moments of genuine innovation.

The Breakdown of a Generative Pre-trained Transformer

Keep in mind that GPT stands for "Generative Pre-trained Transformer". GPT models are trained on a dataset that is essentially a snapshot of human knowledge up to a certain point. This means they can only generate text based on what they've been trained on. They can't come up with novel scientific theories, artistic movements, or ethical frameworks that aren't already present in the training data in some form.

These models don't "understand" information or have a conscious experience (at least by human definition anyways). They operate by calculating statistical relationships between words in a dataset. This allows them to generate text that can be coherent and informative, but this is not the same as having an understanding of the content or the ability to create new knowledge. It is mathematically predictive versus thought driven.

Scientific knowledge often advances through experimentation and the collection of new data. GPT models don't have the capability to perform experiments, analyze new data, or even perceive the world. They can't, therefore, contribute to empirical advances in our understanding of the universe.

While GPT models can perform some level of logical reasoning based on the data they've seen, this is limited. They can't build complex chains of reasoning that would allow them to develop new theories or insights that weren't already present in their training data. Their "reasoning" is a form of pattern-matching, and recognition guided by the data they were trained on or fine-tuned with.

When Do the Improvements Feel Less Jaw-Dropping

Maybe never? However, the law of diminishing returns applies to machine learning as it does to many other domains. Initial versions of models like GPT2 and GPT3 were revolutionary because they represented a "quantum leap" in text generation capabilities. Each subsequent version has brought improvements in coherence, accuracy, and depth of understanding. However, these improvements are incremental and typically focus on fine-tuning rather than fundamentally changing capabilities.

Each new model or version of an existing generative model isn't just an upgrade; it's like getting a brand new superpower. Think of the jump from reading text to generating human-like responses. Now, these models can help compose music, write code, assist with research, create art, and a host of other creative and practical uses. The advancements aren't just incremental; they're transformative in how we interact with technology, the internet, and even each other.

And while it's true that we might get accustomed to these wonders—just like we did with smartphones—that doesn't make the advancements any less revolutionary. Sure, we may not drop our jaws at every new feature, but that's because these amazing capabilities are becoming integrated into our daily lives. They're becoming indispensable tools that help us think, create, and communicate in ways that were unthinkable just a few years ago.

When we say the "novelty wears off," it's often because the extraordinary has become the ordinary. And that's amazing in its own right! Just because we get used to flying doesn't make the act of hurtling through the sky at hundreds of miles an hour any less miraculous.

When Does A Model Hit Peak Maturity?

Determining when a model reaches "peak maturity" is tricky. Technological advancements often come in waves, punctuated by paradigm shifts that redefine what's possible. For generative models, this could involve new architectures, better training algorithms, more computational power, or entirely new approaches to machine learning that we haven't conceived of yet.

However, there are practical constraints, such as computational resources and energy consumption, that might limit the current growth of existing models. In terms of capability, a model could be considered mature when improvements in performance no longer seem to justify the increased costs in resources.

Let's change the way we approach "maturity" though for a moment. I believe there are other facets at play here. The idea that smaller models can compete with larger ones in specific domains suggests that the field is maturing in the direction of specialized, efficient architectures. Initially, the focus was on creating models with as many parameters as possible to maximize performance across a broad range of tasks. Now, we're seeing a nuanced understanding that not all tasks require such computational heft. Specialized models can be "right-sized" for their tasks, offering a balance of performance, speed, and resource consumption. This is a sign of a maturing field where "one size fits all" is giving way to tailored solutions.

The move to make advanced models like LlaMA 3 open source reflects another aspect of maturity: community collaboration. Open-source models are a way to accelerate innovation by allowing a broader community of researchers and developers to build upon foundational work. It also addresses some of the ethical considerations related to equitable access to AI technologies. In a mature field, competition is complemented by collaboration and shared goals.

The Future of AI Models and Training

Ever heard of AlphaGo? Heard of the game Go? Check this out as it relates to model maturity.

Developed by DeepMind, a subsidiary of Alphabet (Google's parent company), AlphaGo was a groundbreaking AI model trained to play the board game Go, which is extremely complex and strategically challenging. The training process for AlphaGo involved a combination of techniques, including supervised learning and reinforcement learning.

Absolutely, the story of AlphaGo is a compelling example of how AI models can achieve exceptional levels of performance, even surpassing human experts. Developed by DeepMind, a subsidiary of Alphabet (Google's parent company), AlphaGo was a groundbreaking AI model trained to play the board game Go, which is known for its complexity and strategic depth.

Training Process of AlphaGo

The training process for AlphaGo involved a combination of techniques, including supervised learning and reinforcement learning:

1. Supervised Learning: Initially, AlphaGo was trained on a dataset of expert Go games. The model learned to predict the moves that a human expert would make in a given situation.
2. Reinforcement Learning: After this initial training phase, the model played millions of games against itself, refining its strategies based on the outcomes. Essentially, it received positive reinforcement for good moves and negative reinforcement for bad ones.
3. Monte Carlo Tree Search: During actual gameplay, AlphaGo used a technique known as Monte Carlo Tree Search to explore possible future moves and their implications. This allowed it to make far-sighted decisions.

AlphaGo demonstrated several key indicators of model maturity:

1. Specialization: AlphaGo was highly specialized for the task at hand. This intense focus allowed it to exceed human performance in a very specific domain.
2. Efficiency and Scalability: The model was designed to make efficient use of computational resources, another sign of maturity. Later versions became increasingly efficient while still improving performance.
3. Adaptability: Through reinforcement learning, AlphaGo could adapt and improve without human intervention, an important feature for any mature model.

In Summary...

• Generative AI models don't currently scale past human knowledge; they repackage it.
• Improvements might feel less jaw-dropping as we get accustomed to what these models can do even though we will likely count on them more and more.
• "Peak maturity" is hard to define but could involve a mix of technological, practical, computational, and should always include ethical considerations.

------- ?? -------

Hey, I'm Dave! I'm a former digital agency owner, now co-founder of Bold Crow AI. I help businesses & organizations implement customized AI solutions in a responsible way. I've built cool tools for nonprofits too, helping them gather and leverage social proof.

Let's connect:?Dave Norris

#futurefrontier #llms #ethicalai