The AI Beat | Moreee powerrrrr!

Happy Friday, readers. Imagine reading the title of this edition like it was being screamed out by a Dr. Frankenstein-type mad scientist in their laboratory as electricity crackled around them — you know the type, represented often in literature, film and pop culture in various iterations.

That's essentially the takeaway many AI observers had from a lengthy (1-hour, 18 minute) video interview by podcaster Dwarkesh Patel of Meta boss and founder Mark Zuckerberg in the wake of the release of the company's new "open" Llama 3 LLM, which is still reverberating around the web today. The relevant portion, for our purposes, is embedded below:

Why? Because in it, Zuck stated that the energy required to train ever-more powerful AI models would be tremendous. As he stated:

"No one has built a 1GW data center yet. I think it will happen. This is only a matter of time but it's not going to be next year...Just to put this in perspective, I think a gigawatt would be the size of a meaningful nuclear power plant only going towards training a model."

That has sparked a good degree of debate, opining, and hand-wringing around the notion that transformer-based LLMs may have hit a plateau in terms of performance, at least what is available without building or procuring an entire nuclear power plant.

Big Technology CEO/journalist Alex Kantrowitz asked in the latest edition of his newsletter: "Are LLMs About To Hit A Wall?" As he writes:

"Until we get to such massive energy allocation, it may be difficult to say how much room LLMs have left to improve. But it seems like sooner or later, we will find out."

Others are more outright pessimistic, stating that LLMs may have already peaked. As journalist Will Lockett wrote on his Medium blog this week (a post entitled "AI Is Hitting A Hard Ceiling It Can’t Pass":

"Unless the AI industry can find a way to be more efficient with its AI training and its computational load, it won’t be able to break past this limit, and AI development will completely stagnate. Now, possible solutions are on the horizon, such as far more efficient AI hardware incorporating analogue and quantum technologies and new AI architectures that requires significantly smaller training datasets. However, these concepts are still in their infancy and are potentially decades away from being used in the real world.

In short, be prepared for AI to massively fall short of expectations over the next few years."

I'm personally unconvinced transformer-based LLMs have peaked or plateaued, or that nuclear power plants will be required to train the next, or even upcoming generations of AI models.

Indeed, smaller startups such as Sakana AI have already demonstrated it's possible to train new models based on aggregating the properties of some leading open source ones — a technique it calls Evolutionary Model Merge. Will this result in new top-of-the-line performance for new foundation models? So far, it hasn't, but I could definitely see a world in which it does.

Meanwhile, those who can afford it will be securing more power...per our headline in today's AI Beat...to the extent they can. Just see Amazon's recently announced purchase of a 960-megawatt, nuclear powered data center called Cumulus Data Assets in Pennsylvania (it's unclear if this will be used for any AI model training, but I wouldn't be surprised). Amazon is pitching it as a way to meet its aggressive carbon emissions reduction goals.

At a time when the once-feared and maligned nuclear power is being reconsidered as a means of delivering the energy people want without the emissions they don't, the drive to power new AI training runs with split atoms seems worthwhile to me — even if it doesn't ultimately result in smarter AI, perhaps it will put the world on a cleaner, more sustainable path.

That's all for this week. Enjoy your weekend. Like/share/subscribe etc.

— Carl Franzen

AI news from the week that was

Read more