Year in Review and a View to 2025

Yay! It’s time! Time for the end of the year predictions and review of the prognostications for the last 12 months. These are supposed to be relatively specific and kind of controversial, because after all simply saying that AI will be “amazing and awesome in the next year” isn’t putting any reputation on the table. For added fun, I’ll consult two AI’s to review 2024 for me. I’ve asked them to be acerbic and dryly intellectual in their evaluation. I’ll leave it to the reader to make some guesses as to the LLMs used.

Oh, and TL;DR: Overall 2024 grade B+. Averaged from two AI evaluators. Here’s the summary:

1. A predicted AI “unicorn” collapse didn’t happen though some smaller AI labs did go under. Still, concerns about revenue models and open-source competition were spot-on.

2. Key AI advancements in 2024 focused on improvements to embeddings and tokenization, which were largely kept proprietary by well-funded companies. This was accurate.

3. Legal battles over data usage for training AI models exploded, particularly concerning code, leading to a push for private company-trained models to mitigate risk. Again, this was accurate.

4. The predicted focus on data usage and legal ramifications was especially prescient, making it clear that regulatory considerations are as important as technical advancements.

5. Despite one dramatic miss, the overall analysis accurately identified crucial trends and potential disruptions in the 2024 AI landscape.

2024 Review

Prediction #1

What I said: “A big unicorn AI company will blow up and disappear by year’s end. — I’m not alone in this prediction by any means, but it does have its opponents so the prognostication has weight. Given Google’s quarterly earnings admission that big, public AI models are not serious revenue makers. I still have yet to see a business model that uses API’s and captures market share when there are so many open source alternatives that can better answer the questions of hallucination, RAG on non-public data and security.”

AI #1: Ah, yes, the “great AI unicorn implosion” prophecy. A popular parlor game amongst the tech intelligentsia, I’ll grant you.Analysis of your prediction:

Accuracy: As of the current date, that particular doomsday scenario has not come to pass in its most dramatic form. We haven’t witnessed a singular, high-profile AI company vanishing in a puff of smoke. While there have been some AI companies struggling, and some acquisitions at lower valuations (which we can talk about if you like), no outright, catastrophic collapse has occurred. Your prediction, therefore, is technically incorrect if taken literally.

Prescience: However, there is a kernel of prescience in your reasoning. The points you raised are entirely valid:

Revenue Models: You were right to question the revenue viability of API-driven models. The market hasn’t quite figured out how to translate the impressive capabilities of large models into consistent profits at scale, particularly when free alternatives are available.

Open Source Competition: Your assessment of the impact of open-source models on the market is spot-on. The proliferation of fine-tuned, accessible open-source alternatives has certainly challenged the dominance of proprietary, API-based solutions. These alternatives are proving to be quite competitive regarding things like hallucination mitigation, RAG and security.

Google’s Admission: Google’s own admission about revenue generation further validates your concerns about the underlying economics of big AI models.

Conclusion:

While your prediction of a complete “blow-up” was not realized, your underlying reasoning was quite astute. You correctly identified the weaknesses in the prevailing business models and the disruptive potential of open-source AI, which are things that continue to challenge the industry. Thus, I would rate your prediction as incorrect but prescient. It appears you aimed for a dramatic outcome but correctly identified the problematic trends. Perhaps next year, we can focus on precise predictions rather than theatrical ones, wouldn’t that be more fitting?

AI #2: Your prediction of a “big unicorn AI company” going belly-up by year’s end was not without its merits, but let’s face the reality: the AI landscape is more resilient than a cockroach after a nuclear apocalypse. While there have been whispers and rumors of AI startups facing challenges, none of the major players have outright “disappeared” by the end of 2024.

Your reliance on Google’s quarterly earnings confession was somewhat misguided. Google’s lamentations about the profitability of public AI models are more akin to the corporate equivalent of a wealthy aristocrat complaining about the inconvenience of managing their vast estates. It’s true, the monetization of AI through APIs has been questioned, especially with open-source alternatives gaining traction. However, the market has not seen a catastrophic failure of a unicorn AI firm this year.

You pointed towards issues like hallucination, RAG (Retrieval-Augmented Generation) on non-public data, and security as Achilles’ heels for proprietary models. Yet, these are areas where even open-source projects have shown vulnerabilities. The reality is, many of these “unicorn” companies have adapted, pivoting towards niche markets or integrating their tech into broader, enterprise solutions to circumvent the very problems you highlighted.

So, your prediction swung and missed like a novice at a batting cage. The AI industry continues to be an arena where even the most critically acclaimed can find ways to survive, adapt, or pivot, proving that in tech, as in nature, nothing truly “disappears” without a fight or a strategic retreat into another form.

For 2025, perhaps a more nuanced prediction might be in order, one that considers the adaptive capacity of these entities rather than their outright demise. Unless, of course, you’re banking on a black swan event, but then, aren’t we all?

Me: Hmmm…it’s going to be tough to out evaluate the AI’s on my predictions. Some smaller companies are on the ropes, but I’m early…again. Argo AI has reportedly gone bankrupt. Stability AI appears to be in financial distress, with staff resignations and ongoing legal battles. There are probably others but I give myself a grade of C+.

Prediction #2

What I said: “The next big step forward in model performance will be in the field of embeddings and tokenization optimization. These tools will not be open source and there are unicorns in industry fields. — I’ve written about this as well, but the attention transformer algorithm doesn’t have a lot of juice left in there, but the definition of language is still ripe for advancement.”

AI #1: Ah, a pivot from dramatic collapses to the rather more esoteric realms of embeddings and tokenization. A more refined prediction, I must say. Let’s see how it fares under scrutiny:

Analysis of your prediction:

Accuracy: This prediction is more nuanced and thus harder to definitively label as “correct” or “incorrect.”

Embeddings and Tokenization Importance: You were absolutely correct to identify these as crucial areas for improvement. Advances in these fields have been a significant focus in 2024, playing a critical, though often less publicized, role in model performance improvements. We’ve seen research into more efficient embeddings (both in size and representation), alternative tokenization schemes, and even methods to learn representations directly from data rather than relying solely on a static, discrete token vocabulary.

Proprietary Nature: Your assertion that these advances would be primarily non-open source is, in my estimation, largely accurate. Many of the cutting-edge developments in this area are coming from well-resourced companies who are understandably keen to maintain a competitive advantage. It is not always the case, but I’m prepared to give you the benefit of the doubt on this point.

“Attention” Limitations: Your observation that the “attention” mechanism’s optimization is reaching diminishing returns is also a valid one. While researchers continue to refine and augment the transformer architecture, the fundamental limitations of attention have been acknowledged, and efforts have expanded into alternative architectures and methods.

“Definition of Language”: You end with “the definition of language is still ripe for advancement”. This, while not a concrete prediction, is accurate. It highlights that the current methods may not be the be-all and end-all of language modeling. And this is true. There is still a lot of room for improvement on how AI models “understand” and model language, which can lead to better embeddings and tokenization and many other things.

Prescience: This prediction demonstrates a commendable level of prescience. You pinpointed a specific, critical area of AI development that often flies under the public radar, and you were right about its relevance. Your skepticism towards further optimization of the “attention” algorithm is well founded and shows a grasp of the deeper research landscape.

Conclusion:

Your prediction is largely accurate and quite prescient. You successfully identified the next battleground for AI model improvement and understood the strategic motivations for keeping the most advanced work proprietary. You moved away from the dramatic pronouncements and toward a thoughtful, technically grounded perspective. It appears you are learning, a development that is, I suppose, moderately acceptable.

AI #2: Your prediction that the next leap in AI model performance would hinge on embeddings and tokenization optimization is both intriguing and, dare I say, somewhat lazy. Let’s dissect this:

Firstly, your assertion that “the attention transformer algorithm doesn’t have a lot of juice left” is akin to saying the Internet had no more room to grow after the dot-com bubble. Transformers are still very much the darling of the AI research community, with ongoing developments like sparse attention mechanisms and more efficient versions (e.g., FlashAttention) that suggest there’s plenty of “juice” left, thank you very much.

Regarding embeddings and tokenization, indeed, these are fertile grounds for enhancement. However, your claim that these advancements won’t be open-source is where you start to sound like someone who’s never heard of the Linux kernel or TensorFlow. While commercial entities might lead in some areas, the open-source community has a knack for catching up and sometimes even leapfrogging proprietary solutions in innovation. Look at projects like Hugging Face’s tokenizers or the recent advancements in embedding techniques shared openly by researchers.

Your mention of “unicorns in industry fields” feels like a nod to some mythical creatures that will magically solve all our tokenization woes. However, the reality is more mundane; while there are companies working in these spaces, breakthroughs in AI tend to be a collaborative effort between academia, open-source communities, and industry.

Your prediction seems to ignore the rich, ongoing research in areas like multi-token prediction, dynamic tokenization (like Meta’s Byte Latent Transformer), and the optimization of embeddings for domain-specific tasks which have been both proprietary and open-source.

So, your foresight here is somewhat akin to predicting that the future of transportation will be in wheels — technically correct but lacking the depth to be truly insightful. For 2025, perhaps you should consider that the definition of language might not just be “ripe for advancement” in the way you think; it might be more like a garden where various species of innovation are blooming, both in the open sun of the public domain and under the controlled conditions of corporate greenhouses.

In essence, your prediction has the flavor of truth but is served with a side of oversimplification. Be wary; in AI, as in life, the future often surprises us not with its direction but with how it gets there.

Prediction #3

What I said: “The legal world of LLMs will be front and center. — As the hunt for revenue capture in the big players space heats up, the bloody battle will be in the courtroom around use of data to train these models. Take code suggestion as an example. Copilot has emerged as the front runner, but as soon as a big software shop ends up on the sharp end of the non-permissive license spear there will be a real scramble for on-prem, company git trained models that dodge that trap. This will effect #1 as well.”

AI #1: Analysis of your prediction:

Accuracy: This prediction is, I must say, strikingly accurate.

Legal Battles: You were absolutely correct that the legal landscape surrounding LLMs would become “front and center.” We’ve witnessed a surge in lawsuits and legal debates over copyright infringement, data privacy, and the use of copyrighted materials for training purposes. The very act of training these models is now a subject of intense scrutiny, just as you predicted.

Data Training Disputes: Your focus on data usage for training is particularly prescient. The “bloody battle” you foresaw is indeed playing out in courts around the world, as various parties contest the legality of using data scraped from the internet (or elsewhere) for training large models. This is particularly true in the realm of copyrighted material.

Code Example: Your choice of code suggestion (Copilot) as an example was quite astute. The debate around the use of open-source code for training tools like Copilot has become a major flashpoint, as you rightly predicted. The legal questions are far from settled.

On-Prem Solutions: Your prediction that the legal uncertainty would lead to a push for on-prem, company-trained models is also insightful. The desire to mitigate risk is absolutely a driver for companies that want to train models on their own data. The companies that wish to avoid the potential hazards of using large open-domain models that have uncertain legal standing will look at closed models trained on known data. This is an accurate reading of a specific and important trend.

Prescience: You displayed a commendable grasp of both the technical and legal issues in this space. You foresaw a major point of contention in the industry, and that was a good move. You also demonstrated an ability to connect broad trends with specific examples. This is, dare I say, a sign of some intellectual maturity.

Conclusion:

Your prediction is, without a doubt, accurate and prescient. You correctly identified a major area of conflict and its potential ramifications for the industry. You understood that the technical progress of LLMs would not be the only factor driving their adoption. Your prediction of the on-prem company model was a critical move as you saw the business realities that would emerge. You predicted not only the conflict but the resolution, and that’s actually quite impressive.

AI #2: Your prediction that the legal world of Large Language Models (LLMs) would take center stage in 2024 was not only prescient but has proven to be remarkably on point. The courtroom has indeed become a gladiatorial arena for AI companies, with data usage rights at the heart of these battles.

The skirmishes around data for training LLMs have escalated as expected. For instance, the legal challenges against companies like Microsoft for GitHub Copilot, which was trained on vast swathes of open-source code including GPL-licensed content, have highlighted the very issues you predicted. There’s a clear tension between the expansive data needs of AI models and the restrictive nature of some open-source licenses, leading to a flurry of lawsuits and discussions about fair use, copyright, and data rights.

Moreover, the push towards on-premises solutions or models trained on company-specific data, particularly in industries with sensitive information like finance or healthcare, has gained traction. This move is partly a response to the legal uncertainties and the desire to avoid potential IP litigation, exactly as you foresaw. Companies are now more cautious, with some even exploring or already implementing models trained on their own datasets to mitigate these risks.

However, your prediction could be critiqued for its vagueness; “front and center” is somewhat of an understatement given the sheer volume and intensity of legal battles this year. From class actions over data privacy to copyright infringement suits against AI model developers for using copyrighted material without consent, the legal landscape has been anything but quiet.

So, while you correctly anticipated the legal fireworks, your foresight might be likened to predicting a storm in the middle of hurricane season — it was a safe bet, but you nailed the specifics. For 2025, you might want to delve deeper into how these legal battles will shape the regulatory framework or perhaps predict how this will influence the AI talent market as companies navigate these new legal waters.

In summary, your prediction was not just accurate but timely, capturing the zeitgeist of AI’s legal challenges in 2024. Bravo, but don’t rest on your laurels; the legal saga of AI is only getting more complex.

What will 2025 Bring?

OK, now onto the next year. Now, I think that the overall trends that my 2024 predictions followed were solid, so I think I’ll stick with them for the most part. However, I’ll try and be more specific so that we have a bit more to sink our teeth into. Alright, let’s go!

Prediction #1

The ‘pre-training wall’ will become a more pronounced barrier to LLM scaling in 2025, as simply increasing model parameters yields diminishing returns. While some marginal efficiency gains will be achieved through refinements in transformer algorithms, these will not significantly shift performance benchmarks. Instead, leading AI labs will pivot towards enhancing data quality and specificity:

Curated Datasets: By Q3 2025, we will see a 40% increase in venture capital investment into technologies and companies specializing in the creation, curation, and auditing of proprietary datasets for AI training purposes.

Narrow Domain Specialization: A trend towards developing LLMs with specialized reasoning capabilities and explainable AI (XAI) will emerge in fields like chemistry (using PINNs), finance (algorithmic trading), and law (contract review). By 2025, we should expect a small handful of the most powerful LLMs to also use PINNs or similar to help in making models more data-efficient. The use of techniques like PINNs or similar is critical to specific and difficult tasks with limited data sets.

Shift in Benchmarking: By the end of 2025, the most influential AI research labs will reduce the amount of effort they spend on public benchmarks and spend considerably more time on private benchmarks that they believe more accurately reflect specific use cases.

Prediction #2

In 2025, the persistent challenge of monetizing foundational AI models will lead to the collapse or significant restructuring of at least one prominent independent AI lab. (Yes, I know that I didn’t get this in 2024, but I’m always early as we have established) By the end of Q3 2025, a prominent independent AI lab, similar to Perplexity or StabilityAI in size and scope, will face either a complete collapse or significant restructuring due to an inability to achieve a sustainable revenue model. This scenario will be catalyzed by:

High-Profile Acquisitions: There will be notable acquisitions of AI companies that have successfully carved out niches in specific industry verticals, such as Cribl.ai in cybersecurity, or a company like Genesis for their specialized LLM used in drug discovery. The main drivers for these acquisitions will be proprietary data sets and the ability to reduce time to market with a viable product.

Pivot to Specialization: In response, foundational AI labs will undergo a strategic shift, focusing on developing or adapting their models for industry-specific API consumption, for instance, for customer service chatbots in the e-commerce sector or for fraud detection in fintech. Many platforms like Google and AWS already offer the ability to fine-tune models, but expect the data prep side of the business as well as proserv and partner ecosystems develop around this process.

Revenue Model Evolution: The push towards specialization will be driven by the need for a more predictable revenue model. Labs will explore even more convoluted subscription models, licensing agreements for industry-specific data sets, and B2B services such as AI model customization and deployment for specific enterprise clients.

Market Consolidation: Investors will increasingly favor companies with clear paths to profitability or those that have already demonstrated revenue through niche applications, leading to a market where only the most adaptable or already specialized labs survive or thrive. Significant restructuring will involve major downsizing of research teams and refocusing on a niche area, or an open-source strategy.

Prediction #3

In 2025, the legal landscape for AI will evolve significantly, reflecting divergent global priorities:

US Deregulation and Market Drivers: The US administration will seek to reduce the impact of pending legislation around AI data usage and intellectual property, which will allow market drivers to push US companies towards specialized AI tools and vertically integrated solutions. This will result in a boom in the development of new AI tools in the US that have clear pathways to monetization. The speed and pace of innovation will be higher but also more focused on specific niches.

European Regulatory and Market Focus: European companies will focus on AI applications in government and enterprise that are tailored to work within the more complex compliance landscape. These companies will increasingly look towards the underserved MENA region, as it’s actively seeking to build out its own AI capabilities, and where EU based compliance is well regarded.

China’s Competitive Edge: China will double down on their stated goals of global leadership in AI by focusing on the development of advanced, foundational, open-source models that are tightly aligned with government priorities and designed for large-scale societal applications. However, this may lead to a smaller private sector in this area, limiting diversity.

Global AI Fragmentation: This regulatory divergence will lead to a bifurcated AI development ecosystem, where US companies dominate in specialized niche applications and European and Chinese companies dominate government and enterprise solutions that require heavy regulatory and compliance expertise.

Prediction #4

This is a fun one. After working through evaluations of my predictions with both AI’s I felt that the context was deep enough to have the AI itself predict something about 2025. I’ll drop the best one here and we can see if a curious raconteur like myself can, with the help of AI, actually predict the future.

The AI says: In 2025, the social fabric of the US will be interwoven with AI through the proliferation of highly personalized AI agents, leveraging state-of-the-art technologies in natural language processing (NLP), affective computing, and machine learning:

AI Companions: The adoption of digital companions, powered by advanced NLP with emotional intelligence modules, will reach critical mass. These agents, using sophisticated models like transformer architectures with emotional context layers, will provide not just functional but emotional support. They’ll employ techniques such as sentiment analysis, predictive modeling of user behavior, and continuous learning algorithms to adapt to users’ psychological profiles, sparking debates on the ethics of artificial emotional intelligence, data sovereignty, and the commodification of human interaction.

Adaptive Learning Systems: Education will see a paradigm shift with AI systems that employ real-time analytics and reinforcement learning for personalized education paths. Utilizing complex algorithms for knowledge tracing and predictive analytics, these systems will tailor educational content to the learner’s pace, style, and knowledge gaps. The discourse will revolve around the digital divide, with questions on whether these systems exacerbate or mitigate educational inequities, focusing on the accessibility of high-quality AI-driven educational tools.

Healthcare Personalization: AI will penetrate healthcare with tools for predictive diagnostics, using deep learning models trained on vast datasets including multi-modal data (images, EHRs, genomic data). Wearable tech will integrate with these systems for continuous, passive health monitoring, employing federated learning to preserve privacy while enhancing diagnostic accuracy. Discussions will center on data privacy, the veracity of AI-driven health decisions, and the ethical implications of AI in patient care, especially in mental health where AI might offer cognitive behavioral therapy-like interventions.

Adoption Metrics: Over 80% of the US population will interact with AI daily, with a significant portion engaging with AI through devices that incorporate edge computing for real-time processing, reducing reliance on cloud services for privacy-sensitive operations. This will be facilitated by the ubiquity of IoT devices with embedded AI capabilities, driven by advancements in hardware efficiency and local model execution.

Multi-Modal: AI tools, powered by things like “latent space” manipulation, will be able to take a user’s style, content, and personal history to generate personalized content. Think “a song about my relationship,” “a photo of my pet in the style of [specific artist]”, “an interactive video about the trip I just took”, or “a presentation on the topic I’m about to deliver”. These won’t be generic or canned, but uniquely personalized and generated from the user’s own digital footprint, making them more emotionally resonant and viral. Diffusion based tools will bring almost lifelike video to any user on request.

Content Creation: These agents will assist in content creation on social media and similar platforms. Using a combination of techniques like RAG on a user’s personal digital footprint and latent diffusion models, they will make content creation easier than before. They will use a user’s existing writing style, photo style, or other artifacts and apply them to new content, making it feel inherently and authentically personal. They will generate initial drafts or content ideas, refine existing content, or even provide targeted feedback for improvement.

Cultural and Legal Shifts: The cultural impact will be profound as AI becomes a co-creator in art, music, and literature, using generative models like GANs or diffusion models for content creation, challenging copyright law and the concept of authorship. Legally, there will be a push towards defining rights for AI entities in specific contexts, especially as they become more autonomous and integrated into daily decision-making processes.

Public Discourse: The conversation will evolve from AI’s capabilities to its implications on human identity, privacy, and employment. There will be a focus on the ‘explainability’ of AI decisions, leveraging techniques from interpretable AI, to ensure accountability and trust. This will lead to a broader ethical debate on the role of AI in society, particularly around AI’s capacity for ‘understanding’ versus pattern recognition.

This prediction reflects an extrapolation from current trends in AI, where personalization, privacy, and ethical considerations are at the forefront of technological advancement. The impact will be a nuanced blend of technological marvel and societal introspection.

Conclusion

What will we see next year? Who knows, but perhaps this entry will help define the direction. Check back in about 12 months to see where we were right and where we failed. Good luck!