Open Source Generative AI Foundation Models: The Toolbox for Innovation

In the 30th edition of this newsletter, entitled?“Enterprise (Multimodal) Generative AI: Paving the Way to Artificial General Intelligence,”?it was concluded that the?“Customized Generative AI”?models can push the current productivity levels to the next frontier by carefully selecting the?“Best-Fit Foundation Models”?that can turn the?“Enterprise Data Assets”?into?“Actionable Knowledge.”?More importantly, several types of these foundation models can be combined to form what is called the?“Multimodal Generative AI Systems.”?Furthermore, these multimodal systems can be built from the?“Small and Specialized Foundation Models”?that provide better total cost of ownership and less harm to the environment. These two trends in generative AI research and development are paving the way to?“Artificial General Intelligence”?at an accelerated pace.?

So, in this edition, the focus will be on exploring how the advancements in?“Foundation Models”?are shaping the future of AI technology. Before going further in the discussion, it is worth mentioning something about the history of these foundation models. These models are essentially large neural network architectures that are pre-trained on vast amounts of text data, allowing them to learn the nuances and complexities of language. Although becoming increasingly popular since the end of 2022, especially in the field of natural language processing, they have a long history from several years earlier to this date.?

The last quarter of 2013 was a turning point for the development and use of these foundation models. Google released the first foundation model with a number of parameters exceeding the?“100M Parameters Threshold.”?This foundation model was called Word2Vec. The parameters of a foundation model are logical weights that are typically learned by training the model on a large corpus of text data. In the next decade, the increase in the processing power enabled researchers to increase the number of trained parameters exponentially. For example, the Open-AI’s GPT-3 model, released in 2020, has 175 billion parameters, and the Google PaLM model, released in 2022, has 540 billion parameters. Finally, Open-AI’s GPT-4 model, released in the spring of 2023, is expected to have over 1 trillion parameters. This was the reason behind giving the language foundation models with these sizes the name?“Larage Language Models.”

In 2024, we can expect even larger language models to be developed with even more parameters. However, what actually happened was the opposite. Foundation models began to prioritize efficiency and sustainability, resulting in smaller models with fewer parameters being developed instead to simultaneously make the training and operational costs manageable and affordable. This shift towards smaller and “Specialized Models”?with fewer parameters has led to significant advancements in the field of Generative AI. These advancements have opened up many new possibilities for applications in various industries.

The number of parameters can usually be considered as fundamental selection criteria for the suitable foundation model from the number of available?“Open-Source Foundation Models “that represent a ready-to-use toolbox for innovation. The number of parameters can also be considered as a quick proxy for the model’s complexity and potential performance. More importantly, it is a proxy for the minimum required computational resources needed for training, fine-tuning, and deployment. By focusing on developing smaller models with fewer parameters, researchers can create more efficient algorithms that can be run on a broader range of hardware, making AI technology more accessible to a larger audience. This shift towards sustainability and efficiency in AI model development is beneficial not only for reducing costs but also for promoting innovation and creativity in the field. As the capabilities of Generative AI continue to improve, we can expect to see even more groundbreaking applications emerge in the near future.?

Hence, and to conclude, the?“Customized Generative AI”?strategies can carefully select the?“Best-Fit Foundation Models”?that can turn the?“Enterprise Data Assets”?into?“Actionable Knowledge.”?More importantly, the focus nowadays is on selecting the most suitable foundation model from the available open-source?“Small and Specialized Foundation Models”?that are available as a toolbox for innovation and, at the same time, provide better total cost of ownership and less harm to the environment.?