Foundation Models

Foundation models are the smartphones of AI.

Consider all of the separate devices that were replaced by iPhones and Androids - phone, camera, videocamera, PDA, GPS, MP3 player, watch, alarm clock, TV, flashlight, even the PC.

Now consider all of the individual AI models for specialized tasks - text classification, sentiment analysis, named entity extraction, topic modeling, OCR, handwriting recognition, image classification, object detection, speech to text, text to speech, summarization, text completion, image generation - slowly but surely multimodal foundation models are replacing all of them.

There's still an argument for using smaller, fit for purpose models, but that argument is primarily based on cost. Costs are declining rapidly (once a model has been out long enough and replaced by a new SOTA model).

In the near term, I'm focusing on efforts like distillation that take the largest, highest performing models and use them to train much smaller models that perform almost as well as the original models but at a significantly reduced cost.