"There is no Moat in LLMs" - Rapid Commoditization of Large Language Models (LLMs)

The rapid commoditisation of Large Language Models (LLMs) is indeed a trend that many large companies may be overlooking. Having looked behind the scenes and spoken to many experts, I share here some of the insights I have gained and a breakdown of the key points that support this perspective:

Falling Token Prices

Token prices for LLMs have been dropping dramatically. For instance, GPT-4's token cost has decreased by approximately 79% per year since its initial release. This steep decline in pricing suggests that LLMs are becoming more accessible and less of a premium product.

Public Data and Lack of Defensible Moat

LLMs are primarily trained on publicly available data, which means that the underlying knowledge base is not proprietary. This lack of a defensible moat makes it challenging for companies to maintain a significant competitive advantage solely based on their models.

Emergence of Smaller, Efficient Models

We're witnessing the development of smaller models that perform comparably to larger ones for many tasks. This trend indicates that the correlation between model size and performance is not always linear, potentially reducing the need for extremely large and costly models.

Specialized Hardware for Inference

The introduction of specialized chips designed for LLM inference is set to dramatically decrease operational costs. This development will further contribute to the commoditization of LLM technology by making it more affordable to deploy and run these models at scale.

Edge Computing and On-Device Models

Advancements in edge computing and on-device models are making it possible to run smaller, optimized versions of LLMs directly on user devices. These models can generate initial outputs that are then verified by cloud-based systems, potentially reducing the reliance on large, centralized models.

Continuous Model Optimization

The LLM landscape is characterized by constant optimization and improvement. This ongoing refinement means that the performance gap between different models is continually narrowing, making it harder for any single company to maintain a significant lead.

Open-Source Alternatives

The availability of open-source LLMs, such as Llama 2, Llama 3, and Mistral, is providing alternatives to proprietary models. These open-source options eliminate per-query token rates, allowing users to deploy models on their own infrastructure or 3rd-parties to host these more efficiently than per-user-subscriptions.

Shifting Value Proposition

As LLMs become commoditized, the real value for businesses will likely shift from the models themselves to how they are integrated, fine-tuned, and applied to specific use cases. This parallels the evolution of personal computers, where the focus moved from raw CPU power to overall system capabilities.

My Personal Outlook

While Nvidia's current success is undeniable, and Big Tech is hyping a lot - the long-term outlook for the AI industry suggests a more distributed and commoditized future. The massive investments being made today may not yield the monopolistic returns that some in Silicon Valley anticipate. Instead, we're likely to see a landscape where AI capabilities become a standard part of technology stacks, with differentiation occurring at the application and integration levels rather than in the underlying models or hardware. This shift could potentially render some of today's hefty investments in proprietary AI infrastructure less valuable as the market matures and diversifies.