?? The foundations of future AI

ChatGPT, Claude and other language models have dominated mainstream discussions and use. It’s not surprising: they’re devilishly capable, a Swiss Army knife and useful to nearly everyone.

However, we believe that AI’s next chapter will be written by a diverse ecosystem of domain-specific foundation models, each mastering its own data domain.

Today’s post is all about why we need such an ecosystem, what domain-specific AIs are already paving the way forward and what considerations you need to take into account whether you’re a founder, investor or just a user of AI.

The foundations of AI

Foundation models are large-scale AI systems trained on varied datasets, designed to adapt to multiple tasks within specific fields or across domains. Their versatility and adaptability can catalyse change across a wide range of applications.

You can think of foundation models as bright undergraduates. GPT-4, Claude 3 and Llama represent the “liberal arts” of AI, offering wide-ranging but generalised capabilities with their mastery of language and all the knowledge encoded within it.

In parallel, we’re seeing the emergence of distinct AI majors in specific fields like climate science and biology. These specialised majors are paving the way for a new generation of diverse foundation models. What’s shaping up is an ecosystem of AIs that could contribute to scientific research, improve climate forecasting and mitigation strategies and accelerate discovery and R&D across industries.

Why an ecosystem of AIs?

Foundation models are extraordinarily good at reducing complexity into something computable. But to use this superpower in complex fields we need to crack like climate and weather forecasting, we need more than language. Computer scientist Anima Anandkumar captured this well in her Stanford talk recently:

You can think of what the weather will be tomorrow, whether there will be storms or rain. For that, you’re looking at processes that involve everything from particles within a cloud that cause precipitation to very large scale effects like atmospheric rivers, which are important for California because they cause storms in winter and can be thousands of miles wide. So you have microscopic to macroscopic phenomena all interacting together to cause the weather on this planet. That’s the aspect where just language models by themselves will not be enough.

Language models like GPT-4, Claude and Gemini benefit from the abundance of data on the internet. There are over 1.9 billion websites and 14 billion YouTube videos - an enormous corpus for training.

However many areas lack abundant data. For instance, there are approximately 10,000 rare diseases , but each affects fewer than 200,000 people. This scarcity of cases translates to limited data for AI training.

These specialised domains require AI models that can make the most of limited data. So we need foundation models to be in a Goldilocks zone:

• (1) Be broad enough: The model has been exposed to a wide variety of data, allowing it to learn general patterns and principles

and at the same time…

• (2) Be specific enough: The model’s architecture and training process are tuned to leverage the data effectively to make accurate and useful predictions in specific tasks. Their training data needs to be focused enough on what domain knowledge is “useful”, yet broad enough to be expressive in their domain. This is especially important when there is not enough data to impose structure.

Because of this duality, we’re not calling these foundation models specialists – which they’re not. They’re really well-trained generalists who specialise to be particularly useful in one domain.

The power of generation

To really understand how this ecosystem of AIs could be game-changing, I will return to my thesis on why humanity needs AI . A few weeks ago, I wrote:

We are near a critical juncture in knowledge production. In the foreseeable—hundreds of years—horizon, we might contend that we are flattening the curve of our knowledge creation—a first for our species. Not only is that an existential turning point, but it’s also a dangerous one. There will still be things to understand, challenges to address and risks to mitigate, for which knowledge will be required.

Generative AI can help us overcome this crisis. We need not just the ability to understand complex phenomena, but also to generate new ideas, simulate future scenarios, predict events... Generation is the real power.

Take drug-discovery. Only 7.6% of drugs that entered clinical trials between 2010 and 2023 successfully made it to market approval. The average cost to take a drug from discovery to commercial launch is $2.3 billion . The chemical space of potential drug-like molecules is estimated to be between 10^23 and 10^60 molecules , far exceeding what can be physically synthesised and tested. Significant money, time and lives can be saved by increasing the probability of a drug’s success.

In this space, one domain-specific foundation model Chai-1 predicts molecular structure, which could help researchers understand the 3D shapes of potential drug targets. Researchers can assess the ‘druggability’ of targets and improve drug design through rapid iteration and optimisation of drug structures. As a result, this could boost success rates in preclinical and early clinical phases to reduce the costs and timelines in the drug development process.

This is an excerpt of a longer essay I wrote for members of Exponential View. Continue reading here + unlock 1 year of free access to Perplexity Pro when you become a member.