Open-Source Large Language Models (LLMs) for Dummies

The GNU Project.

In 1983, Richard Stallman, a programmer at MIT’s Artificial Intelligence Laboratory, founded the GNU Project based on the philosophy that software should respect users’ freedom. Richard defined four essential freedoms of software development: the freedom to run the program as you wish, the freedom to study the program’s source code and change it, the freedom to redistribute copies so you can help others, and the freedom to distribute copies of your modified version to others.

Stallman created the GNU General Public License (GPL), one of the most influential software licenses, to ensure the software remained free and open. This license allowed anyone to utilize and modify the code under a GPL for free but required any derivative works must also be distributed under the GPL, creating a cycle of ongoing collaboration.

The GNU Project’s primary goal was to create and distribute a Unix-like operating system for free, and the GNU Project spent much of the 1980s developing the essential components to make this happen. Along with the initial operating system, this included a robust compiling system that supported various programming languages, a highly customizable text editor, and a widely used Unix shell that offered significant enhancements over the traditional Bourne shell being used at the time.

A key milestone for this project came in 1991, when Linus Torvalds released the Linux kernel, which, when combined with the components of the GNU project formed a fully functional operating system commonly referred to as GNU/Linux.

The open-source movement was born, and today Linux remains one of the most utilized operating systems in the world. Current Linux providers Red Hat and Ubuntu are a testament to the power of open-source development, offering enterprise-level support, security, and performance features for Linux.

?

The Benefits of Open-Source LLMs.

Large language models have become the most recent boom in the technology industry. Since the release of the ChatGPT and its subsequent semi-acquisition by Microsoft, a flurry of VC investment has poured into the space, making data centers one of the most attractive investments in real estate today.

While many people know the largest proprietary LLMs, such as Chat GPT and Anthropic, a significant push in development within the open-source LLM space is generating a lot of promising innovations. These models have a code base that is publicly available and can be modified by anyone, making them ideal products for research and development. They also offer promising opportunities for startup businesses.

Open-source LLMs hold several key advantages over their proprietary counterparts. They are highly accessible and cost-effective. They eliminate the need for expensive licensing fees, making the AI technology available to individuals, startups, and smaller organizations to build upon these models without restrictions.

The Players.

Several leaders of open-source LLMs have emerged over the last few years. In this article, we’re going to analyze a handful of them and assess their current market positions in more detail, but for those short on time, here’s a summary of the results in advance:

?

Bert by Google.

Bidirectional Encoder Representations from Transformers (BERT) by Google is a groundbreaking open-source model designed for Natural language processing. It has made a significant contribution to the field through its ability to understand the context of words in a sentence, making it highly effective for various natural language processing (NLP) tasks.

BERT reads text bidirectionally, meaning, it considers texts from both the left and right of each word, which improves its understanding and predictions. Functionally, this helps it understand more subtle nuances in language through contextually rich word embeddings. It is effective for categorizing text into predefined categories, and identifying and classifying entities like names, dates, and locations. It excels at understanding and generating answers to questions and is useful for tasks requiring the comparison of two sentences, such as entailment and semantic similarity.

?

RoBERTa by Facebook AI.

RoBERta by Facebook AI builds upon BERT’s framework by enhancing its pretraining methods and overall performance. It aims to enhance NLP capabilities through more extensive training techniques.

One of RoBERTa’s key capabilities is enhanced pretraining. It is trained on a significantly larger dataset with longer training durations. Facebook focuses primarily on Masked Language Modeling (MLM), which involves removing the next sentence prediction task used in BERT, which simplifies the training process. In its place, it implements dynamic masking during training, ensuring that each input sequence is masked differently every time it is processed, which enhances the model's robustness.

From an architecture perspective, it is identical to BERT but includes optimized hyperparameters and training procedures such as Layer-wise Learning Rate Decay to stabilize training and improve convergence.

?

T5 by Google.

T5 offers a unified framework driven by a text-to-text approach to NLP that treats every task, whether it’s translation, summarization, question answering, or classification, as converting input text to output text allowing it to have a simpler model architecture and training process.

It’s pre-trained on a clean version of the Common Crawl dataset, ensuring extensive and diverse text exposure. It is trained on a variety of unsupervised tasks, enabling it to generalize well across different NLP applications, as well as easily fine-tuned on specific tasks with relatively small datasets, making it highly adaptable.

A major benefit to T5 is its ability to be adapted to various size models, allowing for both smaller and larger versions depending on the individual needs, computational capacities, and performance requirements of the user.

?

GPT-Neo by EleutherAI.

GPT-Neo is a LLM based on OpenAI’s ChatGPT 3.0. It offers varying size models depending on individual needs, computational capacities, and performance requirements. GPT-Neo has comparable performance to proprietary models and provides robust capabilities for natural language understanding and generation tasks.

It has large pretraining on an 825GB data set known as the Pile and excels at text generation of human-like text, making it ideal for various applications like creative writing and conversational agents.

Its primary benefit is as a free and accessible alternative to GPT-3 but is more resource demanding than some of the alternatives. It is a newer model and is therefore not as widely adopted or as polished as some other LLMs.

?

DistilBERT by Hugging Face.

DistilBERT is a distilled version of BERT, which means it has been trained to replicate the performance of BERT using a smaller model. DistilBERT is trained using a technique called knowledge distillation, where the smaller “student” model learns to mimic the behavior of a larger “teacher” model (BERT). It contains 6 transformer layers, which is half the number of layers in BERT (12 layers), and has approximately half the parameters.

However, due to the smaller nature of the model it benefits from faster inference times as well as lower resource requirements. This means it returns results faster than BERT and for a lower cost of computation. From a performance perspective, DistilBERT retains about 97% of BERT’s performance on language understanding benchmarks while being smaller and faster.

DistilBERT’s greatest strength is its efficiency, ease of use, and fast response times making it ideal for real-time applications.

?

XLNet.

XLNet is the product of a collaboration between Google and Carnegie Mellon University. It combines the best aspects of autoregressive language modeling (like GPT) and bidirectional context understanding (like BERT), aiming to outperform previous models in various natural language processing tasks.

XLNet is built on a unique architecture known as Transformer-XL, which allows it to capture long-term dependencies more effectively than traditional transformer models. Unlike BERT, which is purely bidirectional, XLNet uses a permutation-based approach to enable bidirectional context while maintaining autoregressive properties. It predicts words by permuting the order of tokens, which helps it learn bidirectional contexts without the limitations of masked language modeling used in BERT.

For long-term dependency handling, it uses segment-level recurrence to better handle long sequences of text, improving performance on tasks that require understanding long contexts. It also incorporates relative positional encoding to enhance its ability to model relationships between distant tokens.

The primary strength of this model is its ability to handle long-term dependency, making it suitable for tasks involving long documents.

?

OpenLLaMA by Together AI.

OpenLLaMa is an open-source version of Meta’s LLaMa and leverages a lot of the strengths of the original model. Like XLNet, it uses a hybrid approach that combines Bidirectional and Autoregressive contextual understanding. This allows it to generate text based on a comprehensive understanding of the context while maintaining the ability to predict subsequent words. It’s highly suited for long-form content and maintaining coherence over extended passages.?

OpenLLaMA offers significant benefits over RoBERTa in terms of text generation capabilities, flexibility, and adaptability for a wide range of applications. While RoBERTa excels in natural language understanding and specific natural language processing tasks, OpenLLaMA’s advanced text generation and extensive training make it a powerful tool for conversational AI applications.?

OpenLLama is a direct competitor with XLNet, and offers similar strengths in its ability to handle long-term dependency.

?

BLOOM by BigScience.?

BLOOM is built on a transformer architecture like GPT-3 and BERT, but what makes BLOOM unique is that it's trained on an extensive multilingual dataset. This makes BLOOM the LLM of choice for tasks that require translation, text generation, and summarization. For developers building multilingual NLP tasks, BLOOM should be strongly considered as your preferred LLM.

Hugging Face 谷歌 EleutherAI Together AI @Meta