LLaMA 3: in the search of AGI

Mark Zuckerberg's recent move could redefine AI's future, but is it a visionary leap or a huge misstep? I couldn't resist discussing Mark's latest announcement.

Honestly, I had to read it twice just to ensure I wasn't jumbling up the numbers. I took a moment to digest the information before formulating and sharing my thoughts on it.?

It might sound like a joke, but with such a substantial investment, he realistically could have considered acquiring OpenAI, teaming up with a few well funded partners. Regardless the craziness amount of Infra Mark announced Meta is currently building (350k Nvidia H100s by the end of the year and overall 600k), let's take a step back. We're all aware of LLaMA – an incredible model, and yes, it's open-source. But the truly remarkable feat of LLaMA is how it has paved the way for fine tuning numerous other models, which are now in leading positions of many leaderboards.

I've consistently maintained that the pivotal edge in training Large Language Models (LLMs) boils down to budget constraints. Budget is often the primary limitation for teams, and it's no secret that OpenAI boasts an expansive pool of resources (thanks of course to the Microsoft folks). Don't misunderstand, exceptional talent is essential to craft these models. However, the question remains: are such brilliant minds exclusively found at OpenAI?

A few days back, I went through an interesting paper which essentially conveyed the following:

Beginning with an incredibly small, manually curated preference dataset (OpenAssistant), the LLM undergoes iterative enhancement. Each iteration leverages data produced by the model's previous version to train its successor. Remarkably, after just three iterations, this model, based on the pre-trained-only Llama 2 70B, outperforms the most capable models available.?

And then, I came across Mark's announcement. Coincidence? Who knows....In my view, Mark aims to leave a lasting legacy, and it seems he's closer to that ambition now more than ever before.

Take a look to what Andriy Burkov just posted couple of days ago:

But what's driving Meta and Mark to open-source their models? In a recent interview, their chief AI scientist Yann LeCun shed some light, stating:

“When you have an open platform that a lot of people can contribute to, progress becomes faster. The systems you end up with are more secure and perform better. Imagine a future in which all of our interactions with the digital world are mediated by an AI system. You do not want that AI system to be controlled by a small number of companies on the West Coast of the US. Maybe the Americans won't care, maybe the American government won't care. But I tell you right now, in Europe, they won't like it. They say, “OK, well, this speaks English correctly. But what about French? What about German? What about Hungarian? Or Dutch or whatever? What did you train it on? How does that reflect our culture?”

On the other hand, when they asked about AGI, he just answered:?

“I don't like the term AGI because there is no such thing as general intelligence. Intelligence is not a linear thing that you can measure. Different types of intelligent entities have different sets of skills.”

Indeed, Yann is techie and seems to lack Mark's visionary perspective and, most crucially, his mission in the world. But for now, we don't need to delve too deep into that...

Envision a scenario where Mark beats Sam, or Llama outperforms GPT. In such a case, users might lose interest in other proprietary models, drawn instead to a state-of-the-art model that's, importantly, free. This could foster a vast ecosystem of developers (as Yann predicted), who are then motivated to craft new, fine-tuned models and applications. Such a shift could usher in a new era of research and enhancement, potentially bringing us closer to achieving Artificial General Intelligence (AGI). This future becomes more plausible when thousands of developers worldwide collaborate, rather than a single company steering the course of artificial intelligence.

I know, I know, it might sound far-fetched... perhaps it is. But then again, isn't what Mark just did with his announcement equally, if not more, astonishing?

The step Mark has taken is both bold and ambitious. Time will swiftly reveal whether this was a leap of overconfidence or, akin to his past endeavors, a move that will once again revolutionize the way we interact with the world.