Large Language Model (LLM) Overview

OpenAI ChatGPT caused quite a stir with its launch last year. It had over a million user registrations within days of being made public. The overwhelming response to the ChatGPT brought the concept of large language models (LLM) to a spotlight.

What are LLMs?

Simply put, a Large Language Model is a type of artificial intelligence that applies deep learning techniques and is trained on vast amounts of textual data to understand, summarize, and generate new information or content. Transformer architecture, which was introduced in the paper “Attention is all you need” back in 2017, serves as the building blocks for LLM. Transformer architecture is a type of neural network that relies on an attention mechanism as described in the paper. It outperforms the Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTMs) architectures. Models based on the Transformer architecture are faster to train on large data sets and are also better at retaining relationships between words for Natural Language Processing (NLP) models.

A RNN architecture processes the information sequentially word by word which can be very slow when training on large amounts of data. In addition, they are not very effective in learning the long relationships in large texts and can only retain short range relationships between the words. On-the-other-hand, the LSTM architecture can retain longer relationships between words, but are complex to train and struggle with vanishing gradient descent as the number of hidden layers are increased.

The Transformer architecture overcomes the challenges faced by traditional neural networks. It generates positional encoding of word tokens along with word encoding. The self-attention heads within transformer neural networks allows it to retain the long range relation between the words. The positional encoding of each word as a separate dataset allows the training to be performed in parallel and much quicker than the other neural networks.

How can LLMs be used?

LLMs can perform various NLP tasks such as Text Generation, Classification, Language Translation, Text Summarization, Conversation Assistants etc. Depending on the task and the use case being addressed, commercially available models such as OpenAI ChatGPT, Google Bard, Meta Llama can be selected. Many open source LLMs are also available to use. Hugging Face Transformers Library is a leading open-source library, which also hosts a large number of pre-trained LLMs for different use cases. These models can be downloaded and extended with domain specific training and leveraged for providing AI/ML based solutions.

A transformer model neural network is composed of an encoder and a decoder block. Each of these parts can be used independently, depending on the task.

• Encoder-only models are optimized for tasks that need understanding of the input text such as - text classification, name entity recognition, sentiment analysis etc.Models such as BERT, RoBERTA, ALBERT are examples of encoder-only transformer models
• Decoder-only are good for text generation tasks.GPT, CTRL, BLOOM are examples of decoder-only models
• Encoder-decoder ( or sequence to sequence models ) are leveraged for generative tasks that need input such as text summarization, translation,etc.BART, mBART, Marian, T5 models are some examples of encoder-decoder models

Conclusion

This article provides a high-level overview of LLMs and some scenarios where it can be applied. LLMs are revolutionizing technology with new use cases popping up every day. The transformer architecture is fundamental to building LLMs that are key to Generative AI, and can perform tasks such as text generation, classification, summarization, translation, etc. Training these models is expensive and time consuming, it needs a lot of computational resources. But with the availability of commercially and open-source pre-trained models, we are bound to witness an exponential growth of organizations leveraging or providing solutions using LLMs.