Phased Approach | Is AI Progress Slowing?

In Phased approach this week we look into the new reports that the biggest AI companies are seeing slowing progress in their relentless race towards AGI. We look at the reports and what this means for business adoption.

Could a slowdown actually help adoption? Let's discuss

Are we hitting the limits of Scale with our current approach?

In a significant shift that could reshape the artificial intelligence industry, major AI companies are encountering unexpected challenges in their pursuit of larger language models, forcing a fundamental rethinking of how AI systems are developed and trained. The most telling signal comes from OpenAI's upcoming Orion model, which is showing only modest improvements over its predecessor despite substantial increases in resources and computing power.

Signs of a Plateau

Industry insiders and researchers are reporting a growing body of evidence suggesting that the traditional approach to AI advancement—simply scaling up models with more data and computing power—may be reaching its limits. This development challenges the fundamental assumption that has driven AI progress for the past several years: that bigger models consistently lead to better performance.

"The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again. Everyone is looking for the next thing," says Ilya Sutskever, OpenAI's co-founder who recently departed to found AI labs Safe Superintelligence (SSI). "Scaling the right thing matters more now than ever."

The Technical Challenges

Behind the scenes, major AI laboratories are grappling with a perfect storm of technical and practical challenges that are forcing a fundamental rethinking of how AI models are developed. The economics of training runs have become particularly problematic, with individual training sessions for large models now costing tens of millions of dollars, with no guarantee of success.

These massive training operations are proving increasingly fragile. As systems grow more complex, hardware failures have become commonplace, forcing costly restarts and delays. What makes these setbacks particularly painful is the uncertainty: researchers often won't know if their months-long training period has produced meaningful improvements until the very end of the process.

Adding to these complications is a growing power crisis. The energy demands of these training runs have become so substantial that even companies with deep pockets are finding themselves constrained by power availability in their chosen data centre locations.

Perhaps most concerning is what some researchers are calling a "data quality crisis." The industry has effectively exhausted the world's supply of readily available, high-quality training data. Attempts to circumvent this limitation through synthetic data generation have led to an unexpected problem: new models are beginning to mirror the limitations and biases of their predecessors, creating a kind of artificial echo chamber.

Industry Response: A Shift in Strategy

In response to these mounting challenges, the industry is pivoting away from traditional scaling approaches toward more innovative solutions. OpenAI's development of their O1 model (previously known as Q* and Strawberry) exemplifies this new direction. Rather than simply throwing more computing power at the problem, O1 emphasises "test-time compute" techniques that allow for more sophisticated, multi-step reasoning processes that more closely mirror human thought patterns.

This approach represents a fundamental shift in how AI systems process information. Instead of relying solely on pattern recognition learned during training, these new systems can spend more time analysing problems during actual use, weighing multiple possibilities before arriving at a conclusion. The company has also begun integrating feedback from subject matter experts and PhDs into their training processes, suggesting a move toward more carefully curated learning approaches.

"It turned out that having a bot think for just 20 seconds in a hand of poker got the same boosting performance as scaling up the model by 100,000x and training it for 100,000 times longer," said Noam Brown, a researcher at OpenAI who worked on O1, at the TED AI conference in San Francisco last month.

The competitive landscape is rapidly adapting to this new paradigm. Companies like Anthropic, xAI, and Google DeepMind are all developing their own versions of these more sophisticated training techniques. However, not everyone sees the current challenges as insurmountable. Microsoft's CTO Kevin Scott remains notably optimistic about traditional scaling approaches, while Anthropic's CEO Dario Amodei offers a more sobering perspective, predicting that model training costs could reach an astronomical $100 billion next year.

Economic Implications

These technical shifts are creating significant ripple effects throughout the AI industry's economic ecosystem. On the infrastructure front, we're seeing early signs of a transition from massive pre-training clusters toward distributed inference clouds. This shift could have profound implications for companies like Nvidia, which has dominated the AI chip market largely through its strength in training hardware.

"This shift will move us from a world of massive pre-training clusters toward inference clouds, which are distributed, cloud-based servers for inference," notes Sonya Huang, a partner at Sequoia Capital.

The business landscape is also evolving rapidly. OpenAI's operating costs are approaching $10 billion for 2024, raising serious questions about the sustainability of current development approaches. This has led to an increased emphasis on efficiency and return on investment across the industry. Companies are increasingly focused on extracting maximum value from existing capabilities rather than pursuing raw performance improvements at any cost.

Competitive Positioning

Despite the challenges, major players remain confident in their ability to maintain competitive advantages. "We see a lot of low-hanging fruit that we can go pluck to make these models better very quickly," says Kevin Weil, OpenAI's chief product officer. "By the time people do catch up, we're going to try and be three more steps ahead."

Nvidia, whose AI chips have fuelled its rise to becoming the world's most valuable company, is also adapting to the new landscape. CEO Jensen Huang recently addressed the shift at a conference in India: "We've now discovered a second scaling law, and this is the scaling law at a time of inference... All of these factors have led to the demand for Blackwell being incredibly high."

The Silver Lining

Despite these challenges, many industry experts see the current situation as an opportunity for healthy industry maturation. The slowdown in traditional scaling advances is forcing companies to focus on practical applications and real-world implementation rather than chasing theoretical capabilities. This has led to more predictable development cycles, better alignment with enterprise needs, and increased attention to safety and reliability.

The industry is also seeing more sustainable business models emerge as companies focus on creating practical value rather than pursuing headline-grabbing breakthroughs. This shift toward business maturity is particularly welcome for enterprise customers, who benefit from more stable, reliable AI solutions that better align with their actual needs.

Looking Ahead

As the industry adapts to these scaling limitations, several clear trends are emerging. Research priorities are shifting toward novel architectures and training approaches that emphasise efficiency over raw scale. The market is evolving from a race for breakthroughs toward a more measured approach focused on maximising current capabilities and developing specialised, domain-specific solutions.

This evolution is also reflected in changing investment patterns, with venture capital and corporate funding increasingly flowing toward companies focused on efficiency and optimisation rather than those simply promising bigger models. The emphasis on specialised solutions suggests a future where AI development becomes more diverse and nuanced, with different approaches optimised for different use cases rather than a one-size-fits-all push for scale.

What does it mean for the AI industry

The AI industry stands at a crucial inflection point. While the era of easy gains through simple scaling could be ending, this transition could mark the beginning of a more mature and sustainable phase of AI development. As companies pivot from raw scaling to targeted innovation and optimisation, the focus is shifting from who can build the biggest models to who can most effectively harness and apply existing capabilities.

This change could ultimately prove more beneficial for the sustainable development of AI technology, leading to more practical and efficient solutions rather than just bigger models. The next chapter in AI development will likely be written not by those who can scale the highest, but by those who can innovate most effectively within existing constraints.

That being said this could just be a bit if a collective panic in the AI industry. There does seem to be a consensus coming from all the big AI shops, however the real elite players in this field are represented by a really small group of insiders, and likely a bit of an echo chamber. This could just be an S-Curve growth pattern. In the meantime though, perhaps we can all draw a breath and decide how we can actually use this amazing technology with accelerated adoption.