The Rise of Open-Source Generative AI: Evaluating the Leading Models - Part 2

In Part 1 of this series, we explored the true essence of "open source" in artificial intelligence, guided by the Open Source Initiative's (OSI) newly defined standards. We delved into the importance of transparency, accessibility, and shareability in fostering innovation within the AI community. But how do existing models measure up against these standards?

In this second instalment, I will examine some of the most prominent large language models (LLMs) released in the past six months, many from big tech companies. I will evaluate how well they align with the OSI's definition of open-source AI and discuss their capabilities and the challenges they present.

Evaluating Leading Open-Source LLMs

The surge in open-source LLMs has been significant, but not all models fully embrace the OSI's standards. Let's examine six notable models to see how they stack up.

Vicuna-v1.3

Vicuna, developed by LMSYS.org, was created by fine-tuning the Llama base model. It is an auto-regressive language model. There is a family of Vicuna models, like Vicuna-7B, Vicuna-13B, and 33B. Released in May 2024, Vicuna is designed as a chat assistant capable of engaging in various conversational tasks. With a 2K context window, the 33B version requires approximately 65.2GB of VRAM.

Open Source Characteristics

• Free to Use: Partially. Free for non-commercial purposes.
• Inspectable: Yes. Model architecture is publicly documented.
• Modifiable: Limited. Modification terms are unclear.
• Shareable: Limited. Non-commercial license restricts sharing.
• Transparent Training Data: Partially. Trained on ~125K conversations from ShareGPT.com.
• Accessible Source Code: Partially. Training code is available on GitHub.
• Open Weights: Yes. Model weights are available for download.

Llama 3.1

Released in July 2024 by Meta (previously Facebook), it has a 128K context length and supports multiple languages. It has remarkable capabilities in general knowledge, math and multilingual translation, rivalling models like GPT-4.

Open Source Characteristics:

• Free to Use: Yes. It is for both commercial and research purposes, but it has limitations. Products with over 700 million monthly active users require a separate license.
• Inspectable: Yes.
• Modifiable: Yes.
• Shareable: Yes. Must include the license agreement and display "Built with Llama."
• Transparent Training Data: Partially. Trained on ~15 trillion tokens from public sources.
• Accessible Source Code: Yes. Available on Hugging Face and GitHub.
• Open Weights: Yes.

AYA 23-8B

AYA 23-8B supports 23 languages and has an 8192-token context length. It's designed for efficiency and accessibility in multilingual research. Developed by Cohere For ALL, the model was released in May 2024.

Open Source Characteristics:

• Free to Use: Yes. For non-commercial purposes.
• Inspectable: Yes.
• Modifiable: Yes.
• Shareable: Yes. Under CC-BY-NC terms.
• Transparent Training Data: Partially. Trained on the Aya Collection (513 million prompts in 114 languages).
• Accessible Source Code: Yes. Available on Hugging Face.
• Open Weights: Yes.

Mistral Large Instruct

Mistral Large is an advanced text generation model developed by Mistral AI and released in July 2024. It is known for its top-tier reasoning capabilities and proficiency in code generation. With a whopping 128K context window, Mistral can perform complex multilingual reasoning tasks.

Open Source Characteristics:

• Free to Use: Yes. For research and non-commercial use. Commercial use requires a separate license.
• Inspectable: Yes.
• Modifiable: Yes. Under research terms.
• Shareable: Limited. Sharing is allowed under the research license.
• Transparent Training Data: Partially.
• Accessible Source Code: Yes. On Hugging Face.
• Open Weights: Yes.

Gemma 2

Gemma 2 is a family of open-language models from Google that was released in June 2024. It aims to deliver high-performance NLP in a compact size, supporting multiple languages and over 80 coding languages.

Open Source Characteristics:

• Free to Use: Yes. Gemma License—a custom "open-source" license allowing both research and commercial use with certain restrictions.
• Inspectable: Yes.
• Modifiable: Yes.
• Shareable: Yes.
• Transparent Training Data: Partially. Trained on ~8 trillion tokens.
• Accessible Source Code: Yes. On Hugging Face.
• Open Weights: Yes.

Qwen 2 Math 72B

Qwen2-Math-72B is a specialized math model with a 128K context window. It significantly outperforms previous models on math benchmarks, surpassing GPT-4 in some areas. Developed by Qwen AI, the model was released in July 2024.

Open Source Characteristics:

• Free to Use: Yes.
• Inspectable: Yes.
• Modifiable: Yes.
• Shareable: Yes.
• Transparent Training Data: Partially.
• Accessible Source Code: Yes. On Hugging Face.
• Open Weights: Yes.

While many other open-source LLMs have been released recently, I've focused on those from big tech companies and those released in the last six months. These models are shaping the current landscape but represent just a fraction of the ongoing developments in open-source AI.

Conclusion

Evaluating these models against OSI's open-source standards reveals a mixed landscape. Many "open source" models fall short of full compliance, primarily due to restrictive licenses that limit commercial use or lack full transparency in training data.

However, the progress is undeniable. Even with some limitations, the increasing availability of powerful LLMs is democratizing AI development. Developers and researchers now have access to advanced models that were previously the domain of tech giants.

Why Does This Matter?

Clear standards and honest labelling are crucial. They help developers understand the freedoms and limitations associated with each model. This transparency fosters innovation by allowing developers to build upon existing work without legal ambiguities.

As the open-source AI community grows, adherence to OSI standards will ensure that the "power to the people" promise is fully realized.

Stay Tuned

The landscape of open-source AI is rapidly evolving. As new models are released and standards continue to develop, we can expect even more exciting advancements in the future.