They(LLMs) are all the same - myth or fact ?

This is the third article in the AI series. And I need to address something that keeps coming up a lot in discussion.

FACT : Performance of all major LLMs are currently converging. In-fact the above image shows us that gap between both closed and open source foundation models are also decreasing.

Myth : With every new model release, we see this gap between the performance of the models decrease. Hence foundational models are going to be commoditized.

The myth stems from the following assumptions :

1. We are running out of data to train, hence no one has an edge
2. Data is the only blocker, and we are at the end of what can be squeezed out of the Transformer model

The reality is far from this. To understand it, let's look under the hood of the different forces at work. Three voices which alluded to them are :

1. Leopold Aschenbrenner(former OpenAI engg.) from his Menifesto. The full 165 pages here.
2. Eric Shmidt (former Google CEO) - recent talk at Stanford(they took down the video, but parts of it are available back now)
3. Ilya's view on the Transformer model

Here's the summary:

1. We are NOT running out of data. See below
2. The true blockers are: Capital & Power

Data isn't a blocker

• Re-training on the same data works wonders on improving model performance (Leopold)
• Training on artificial structured data works pretty well. i.e. when annotated data is fed to models, instead of raw data from the internet, the model performance is better & a lot less data is required
• Capital required for new models is exponential. 1 Billion & 100 Billion dollar models are seriously being thought of. They are no more thought experiments.

Why Power & Capital

• The first two reasons above requires an enormous amount of power, think about training GPT-4 10-100 times on a slight variations of reasoned data
• Those NVidia clusters are hungry babies !! :)

When you put all the above in to perspective, the emerging picture shows that the picture hasn't played out fully.

And as we get closer to 2029, things are going to:

• Divergence in performance
• Capital will dictate consolidation of companies
• War for AI supremacy will move from Companies to Nation state (currently being played out in the shadows)

#AI #AItrends #AI-for-CIOs

Here are the links to the previous articles in this series:

2nd Article - Fourth Revolution

1st Article - AI : Where are the use cases