What is a foundation model -- and how does it lower the cost of machine learning applications?

A foundation model is a partially completed?deep learning?algorithm that has been pre-trained with extremely large data sets covering a vast range of topics.

Foundation models cost millions of dollars to create because they contain hundreds of billions of?hyperparameters?that have been pre-trained with hundreds of gigabytes of data. Once created, however, each foundation model can be modified an unlimited number of times to automate a wide variety of discrete tasks.

Foundation models use self-supervised transfer learning techniques to apply information from learned from one situation to another.?Today, this type of machine learning model is primarily used by hyperscale IT companies working on natural language processing (NLP), natural language understanding (NLU) and natural language generation (NLG). Popular use cases include:

• BERT?-- helps artificial intelligence programs understand the context of ambiguous words in text by processing text in left-to-right and right-to-left directions simultaneously to determine a word’s context. BERT stands for Bidirectional Encoder Representations from Transformers.
• GPT-3?-- uses deep learning algorithms to produce text that appears to have been written by a human being. GPT-3, which is often used on websites to generate product descriptions and news summaries, stands for Generative Pre-trained Transformer 3.
• DALL-E 2?-- uses a process called “diffusion” to create realistic images and art from a description in natural language. DALL-E 2 is a portmanteau of WALL-E and Salvador Dalí.

Public access to foundation models through?OpenAI?is expected make it more cost-effective for smaller companies to use deep learning in a broad range of applications. Responsible AI proponents are concerned, however, that modified foundation models are vulnerable to data poisoning?attacks that purposely introduce misinformation and amplify?machine bias.