The Great AI Plateau? Why LLMs Aren't Actually Hitting a Wall

Welcome back, trailblazers! It's time for another edition of Leading Edge, where we dive deep into the world of Generative AI, leadership, and cutting-edge technology.

Today, we're tackling a question that's causing heated debates in boardrooms and research labs alike: Have Large Language Models (LLMs) hit their scaling limit?

Let's unpack this complex topic and explore what it means for the future of AI.

The Battle of Tech Titans

The AI world is in the midst of a fascinating debate about the future of LLM scaling. On one side, we have Salesforce's Marc Benioff boldly declaring we've reached the limit of what's possible with current approaches.

Jensen Huang scoffs. Loudly.

But of course he does.?? He has 20 Bn reasons in the form of GPU orders quarter to maintain optimism about continued scaling.

Microsoft's Satya Nadella has joined the conversation too, offering a more nuanced perspective that suggests we're not at a dead end, but rather at a crossroads.

Who's right?

Truth is messy. Like your first attempt at prompt engineering. ??

The Evidence on Both Sides

The skeptics have some compelling points. The jump from GPT-3.5 to GPT-4 was indeed revolutionary, showcasing improvements across virtually every benchmark and capability.

But since then? We've seen iterations rather than innovations, improvements rather than breakthroughs.

The latest performance metrics tell an interesting story:

Looking at the current leaderboard, we see a fascinating clustering of performance:

• Gemini-Exp-1121 leads with a score of 1365
• ChatGPT-4.0-latest follows closely at 1361
• Other models like Claude 3.5 Sonnet and GPT-4 variants cluster in similar performance bands

This clustering suggests we might be approaching what statisticians call a "local maximum" - a point where incremental improvements become increasingly difficult and expensive to achieve.

The Traditional Scaling Playbook: When More Was Always More

Historically, improving LLM capability meant following a relatively straightforward playbook with two main strategies:

Data Acquisition: LLMs feed on data. More data. ALL the data. The strategy of "more data equals better performance" has led to widespread scraping of internet content, resulting in a wave of lawsuits from content creators, media companies, and authors.

Result? Everyone's getting sued. Authors. Publishers. News outlets. Even their cats are suing the LLM providers.

Parameter Scaling: This is reflected in the ever-growing "B" numbers in model names, indicating billions of parameters.

Billions of them. Then more billions. Bigger B, better model. Right?

But then comes smaller models. from 2B to 8B - and they punch above their wieght class. Anthropic recently turned this assumption on its head by charging 4x more for their smallest model, Claude 3.5 Haiku, claiming it outperforms their previously largest model, Claude 3 Opus.

Not surprisingly, the larger model was also discontinued (due to lack of demand?)

But wait, there's one more.

While the debate about data and parameters rages on, there's a third factor that often gets overlooked: Algorithmic Improvements. But we're not seeing the kind of algorithmic advantages that characterized earlier tech revolutions - like Google's PageRank algorithm, which single-handedly revolutionized search. The playing field has leveled, with most major players having access to similar fundamental technologies.

It's like everyone's cooking with the same ingredients, following the same recipe, and wondering why all the dishes taste the same.

Something's gotta give.

The Intel Story: When Faster Wasn't Better

The 1990s.

Glorious times for Intel. They owned the world. Or at least the part that mattered.

Their mantra was simple: Faster equals better.

And it worked. Boy, did it work.

The narrative was beautifully simple: faster clock speeds meant better processors. Intel's Pentium line, backed by clever marketing and relentless innovation, dominated the market. Moore's Law wasn't just a prediction; it was a promise - transistor counts would double every two years, bringing corresponding performance improvements.

For almost two decades, this approach worked brilliantly. Processors became smaller, faster, and more capable, seemingly without limit.

Until it wasn't.

Physics showed up to the party. Uninvited.

Heat became a problem. A big problem. Power consumption went through the roof. Quantum effects started messing with the perfect plan. Clock speeds hit a wall at 3-4 GHz. And stayed there.

Stuck.

The mighty Intel, master of the silicon universe, had hit its limits. The experts declared Moore's Law dead. Perhaps they kept declaring it a bit too much.

They were wrong.

The ARM Revolution: Apple's Game-Changing Move

But progress wasn't over - it just needed a new direction. The limitations of increasing clock speeds led to alternative approaches, particularly in ARM-based architectures. These processors prioritized efficiency and multi-core processing over raw speed, becoming the foundation of mobile computing.

The watershed moment came in 2020.

Apple had enough. Done with Intel's promises. Done with the heat and power consumption. Done with the lame 2% incremental performance gains with new processors.

They went their own way. The M1 was born. The results were revolutionary.

A MacBook that lasted all day. A Mac Mini that punched way above its weight. An M1 Ultra that could run 70B parameter LLMs better than desktop setups with dedicated NVIDIA GPUs.

Yes, you heard that right: Even an M1 Ultra with 128GB, currently a 3 year old system could outperform any desktop class setups combining latest NVIDIA GPUs in running LLM models of 70B parameters or higher.

This wasn't just an incremental improvement - it was a paradigm shift.

Alternative Ways to Scale LLMs

So how can LLMs break through their current limitations?

The answer might lie in a concept called inference-time reasoning, or "test-time reasoning." This approach emphasizes "Chain-of-Thought" (CoT) reasoning, enabling models to process information more deeply during inference, similar to human deliberative thinking.

OpenAI's o1-preview model represents an early step in this direction.

At the risk of anthropomorphizing, this shift mirrors the dual-process theory in cognitive psychology:

• System 1 Thinking: Fast, automatic, and intuitive responses
• System 2 Thinking: Slow, deliberate, and analytical reasoning

Traditional LLMs operate primarily in System 1 mode, providing quick responses based on learned patterns. In contrast, models using CoT reasoning engage in System 2-like processing during inference, allowing for more thoughtful and accurate outputs.

Last week, Deepseek R1 dropped.

First open-source Chain-of-Thought model. With some quirks of being a state owned chinese products of course.

Well, almost open-source. They're still working on that part.

Similar to o1, DeepSeek-R1 approaches tasks through careful reasoning and planning, representing a new direction in LLM development.

While o1 and R1 itself might not be groundbreaking, the concept behind it is revolutionary. As foundation providers fine-tune and improve upon this approach, we can expect to see more powerful implementations in the future.

The Path Forward

History rhymes.

Intel hit a wall with clock speeds. LLMs are hitting a wall with scale.

Intel found a new way. LLMs will too.

It's not about abandoning growth. It's about growing smarter.

The AI industry is discovering alternative paths to enhanced capability. This isn't about abandoning scaling entirely - it's about scaling smarter.

Key developments to watch include:

• Further refinements in inference-time reasoning
• Integration of specialized domain knowledge
• Development of more efficient training methodologies
• Innovation in model architecture and design

The question isn't whether LLMs have hit a scaling wall - it's whether we're looking at scaling in the right way.

The next breakthrough might not come from building bigger models, but from building smarter ones that can think more effectively with the resources they have.

The current limitations in LLM scaling aren't the end of progress - they're catalysts for innovation. And before i go, here's someone a lot msarter than me, Satya Nadella saying pretty much the same thing.

Until Next Week...

Hope you enjoyed this deep dive into the fascinating world of AI scaling! Your feedback is crucial as ever. Hit reply and let me know what you think! Want to see a specific topic covered next week? Don't be a stranger - share your ideas!

And of course, if you found this newsletter valuable, spread the knowledge! Share it with your network and help us grow this community of forward-thinkers and innovators.

See you next week!

#artificialintelligence #generativeai #leadership #ai #productdevelopment #startups