Open Source Large Language Models in 2023

With a growing diversity in the AI field and an expanding array of open source alternatives, here are some of the notable contenders leaving a significant mark.

Since the introduction of OpenAI's chatbot, ChatGPT, late last year, there has been a substantial increase in public interest regarding large language models (LLMs).

Although the lucrative potential of generative AI-based tools is evident, numerous smaller businesses and independent researchers within the broader AI community exercise caution when considering closed-source LLMs. This caution stems from concerns about operational costs, substantial computational requirements, and other factors such as data ownership, privacy, and the occasional tendency of these models to generate inaccurate information, colloquially known as "hallucination."

It's not surprising that open-source alternatives have gained momentum in the past year. Surveys indicate that, although open-source LLMs may not match the overall power of their closed-source counterparts, they can be customized to excel in specific tasks, surpassing proprietary models.

With the AI landscape witnessing increased diversity due to the emergence of a broader range of open-source alternatives, here are some of the notable contenders leaving a significant mark in 2023.

1. LLaMA and LLaMA 2

In February, Meta unveiled the initial iteration of LLaMA, a substantial language model featuring 13 billion parameters. It demonstrated superior performance to GPT-3, a model with 175 billion parameters, across various benchmarks. Initially released as an open-source package under a noncommercial license, developers could request access; however, the model's weights were leaked online, essentially making it freely accessible to anyone.

Subsequently, in July, Meta introduced LLaMA 2, emphasizing its training on 40 percent more data than the original version. Additionally, specialized versions like LLaMA 2-Chat, optimized for human-like conversations, and LLaMA Code, tailored for code generation, were introduced.

While there is ongoing debate about the true open-source nature of LLaMA 2, Meta has relaxed usage restrictions to include commercial purposes. This shift has led to the development of open-source derivatives based on LLaMA, such as Alpaca, Alpaca-LoRA, Koala, QLoRA, llama.cpp, Vicuna, Giraffe, and StableBeluga.

2. Pythia

In April, the nonprofit lab EleutherAI introduced Pythia, a suite of large language models (LLMs) of varying sizes trained on public data. Designed as an interpretability tool, Pythia aids researchers in gaining a deeper understanding of the training process and outcomes of LLMs.

3. MPT

MosaicML launched the MPT series of large language models in May, starting with a 7-billion-parameter model and later introducing a 30-billion-parameter version in June. Claiming superiority, especially in scenarios requiring longer text prompts, MPT incorporates advanced techniques to enhance efficiency, context length extrapolation, and stability to minimize loss spikes.

4. Falcon

Released in June by the Technology Innovation Institute in Abu Dhabi under the Apache 2.0 license, Falcon quickly gained popularity with its 40-billion-parameter model. In September, a larger Falcon model with 180 billion parameters was announced, positioning it among the largest open-source LLMs. Despite trailing closed-source models like OpenAI’s GPT-4, the team asserts that it outperforms Meta’s LLaMA 2 and matches Google’s PaLM 2 Large.

5. BLOOM

While officially released in July 2022, BLOOM earns a spot on our list for its significant impact. Developed through the collaboration of over 1,000 AI researchers from 60 countries and 250 institutions, BLOOM, standing for BigScience Large Open-science Open-access Multilingual Language Model, facilitates public research on large language models. The largest BLOOM model, with 178 billion parameters, is trained on multilingual data from 46 human languages and 13 programming languages, establishing itself as the most extensive open-source massively multilingual model to date.

6. Mistral

Founded by former researchers associated with Meta and Google, Mistral unveiled a 7-billion-parameter LLM in September. The Paris-based startup claims that Mistral 7B outperforms other open-source LLMs like LLaMA 2 across various metrics. Recently, the team generated substantial attention by releasing a newer model, Mixtral 8x7B, through a torrent link, overshadowing the often predictable publicity surrounding releases from larger tech companies.

Conclusion

As the realm of open-source LLMs undergoes further expansion, many developers are actively seeking ways to decrease reliance on OpenAI's API. This shift towards open-source alternatives is driven by considerations of cost-effectiveness, transparency, and the ability to fine-tune models.

While proprietary models may maintain a marginal advantage at present, open-source counterparts are rapidly closing the gap. Some open LLMs have already demonstrated superior performance compared to their counterparts with larger parameters, underscoring the significance of the quality of training data over sheer size. The past year has witnessed compelling advancements in open LLMs, underscoring their pivotal role in the evolving landscape of large language models.