Leveraging the Potential of Large Language Models

At Sadiq, we are a team of innovators dedicated to exploring the vast domain of Large Language Models (LLMs), their potential, and their applications. In this blog, we will focus on understanding how they propel advancements in neural processing networks and generative AI.

Before we dive deeper, let's understand the essence of these remarkable Large Language Models.

Understanding Large Language Models

LLMs play a pivotal role in contemporary AI systems. They are crafted to interpret user input and deliver intricate, human-like responses with intelligence.

You might even already be familiar with some LLMs in the market, such as GPT 3.5 used for the now popularized ChatGPT. These advanced AI models, along with others such BART (will be looked at in depth later in the blog), LaMDA and PaLM have been trained on vast amounts of data, enabling them to comprehend context, nuances, and even emotions, allowing for more natural and engaging interactions with users.?

By employing state-of-the-art deep learning techniques, LLMs have revolutionized various applications, including natural language processing, chatbots, language translation, content generation, and more. Their remarkable ability to understand, learn, and adapt from the vast sea of information available on the internet empowers them to tackle complex tasks and support decision-making processes across diverse industries.

Historical Context of LLMs

The development of these revolutionary Language Models has spanned several decades. It started with N-gram Models in the 1950s and 1960s, using statistical techniques to predict word probabilities based on preceding words.?

Hidden Markov Models (HMMs) emerged in the 1970s and 1980s, incorporating observed words and hidden states to capture linguistic patterns and context better. However, these models were limited in modeling long-range dependencies and the complexity of natural human language.?

Neural Network-based Models then gained prominence in the 2000s, particularly Recurrent Neural Networks (RNNs) like Long Short-Term Memory (LSTM), improving context-aware text generation. The transformative breakthrough came with the introduction of transformer models, which leveraged self-attention mechanisms for contextual relationships, enhancing machine translation, text generation, and other NLP tasks.?

Large Language Models emerged more recently, exemplified by OpenAI's GPT series, showcasing the power of pre-training LLMs on extensive text data and fine-tuning for specific downstream tasks. Today, LLMs have become a versatile tool in AI research and Natural Language Processing, demonstrating their ability to generate coherent and contextually relevant text across various applications.

Now that we have covered some history, let's get a bit more technical.

The Functionality

A Large Language Model is simply a type of neural network, a machine learning model composed of interconnected neurons that process data through mathematical functions. These neurons, much like the ones in a human brain, are the fundamental building blocks of computation.

The power of a neural network lies in the connections between its neurons. Each neuron calculates an output based on its input, and these connections determine how much influence one neuron's output will have on the following neurons. While some neural networks are small with just a few neurons and connections, LLMs are significantly larger, having millions of neurons and hundreds of billions of connections, each with its own specific weight.?

LLMs employ a particular neural network architecture known as a transformer, specifically designed to process sequential data, like text. The idea behind the transformer architecture is based on "attention," wherein certain neurons pay more attention to others in a sequence. Since text is read and processed in a sequential manner, with various parts of a sentence influencing each other, the transformer architecture suits text-based data well.

What sets LLMs apart is their remarkable ability to build themselves. Instead of programmers defining the model's instructions, they create algorithms that allow the model to review large amounts of data, typically text in the case of LLMs, and define its own connections and weights. This process, known as training, involves refinement through trial and error, allowing the model to create quality results.?

This leads to another important aspect of LLMs, which are the models’ parameters. These refer to the internal variables that the model uses to learn and make predictions. After the training process, the model retains a set of parameters that represent the learned knowledge and relationships within the data they were trained on. This includes elements such as weights, biases, activation functions, learning rate, etc. Which can all be used to optimize its performance.

One usage of LLMs would be in the context of generative text models. Generative text models work by leveraging deep learning techniques to learn patterns and relationships within a given dataset of text. These models are typically based on architectures like GPT and operate at a very low level to generate coherent and contextually appropriate text. At their core, LLM models are built upon the Transformer architecture, which comprises multiple layers of self-attention and feed-forward neural networks.

The model's training process involves two main stages:?

Stage 1: During pre-training, the model learns to predict the next word in a sentence based on the context provided by previous words. It does this by processing large amounts of text data, such as books, articles, and websites, and capturing statistical relationships and patterns within the text. The model's objective is to minimize the difference between its predictions and the actual next words in the dataset. Once the pre-training phase is complete, the model has developed a strong language understanding capability, but its outputs are not fine-tuned for specific tasks.?

Stage 2: In the fine-tuning stage, the pre-trained model is further trained on a smaller dataset, tailored to the specific task it needs to perform, such as generating poetry, answering questions, or writing code. This dataset includes input-output pairs, allowing the model to learn how to produce appropriate responses or generate text in line with the desired task.

During text generation, a seed prompt is given to the LLM model, which acts as the starting point for generating subsequent text. The model takes the seed prompt and utilizes its learned knowledge to predict the most probable words or tokens that should come next, given the context. This process continues iteratively, with the model generating one word at a time until the desired length of text is achieved.

One essential aspect of LLM models is their use of a probabilistic approach. The model generates text by sampling from a probability distribution over the model's vocabulary for each word, considering the context and previous predictions. This probabilistic nature allows the model to produce diverse and creative output, which turns out to be efficient and useful.

Current LLMs

CLIP

CLIP (Contrastive Language-Image Pre-training) is an innovative AI model developed by OpenAI. Its purpose is to understand the connection between images and text descriptions better. Unlike many other models that are optimized for specific tasks, CLIP is designed to predict the most relevant text description for an image in a more generalized manner.

The fascinating aspect of CLIP is that its architecture combines two crucial domains of AI: Natural Language Processing (NLP) and Computer Vision (CV). By utilizing both of these domains, CLIP becomes incredibly versatile and can learn a wide range of tasks that involve understanding both images and text.

One of the key advantages of CLIP is its zero-shot learning capability. Traditional machine learning models are typically trained to recognize a fixed set of classes, but CLIP breaks free from this limitation. Zero-shot learning allows the model to generalize and make predictions on entirely new and unseen labels without needing specific training for each label. For instance, while traditional image-based models can recognize only a thousand specific classes, CLIP is not bound by this constraint, making it way more flexible.

To achieve this level of understanding, CLIP is trained using a technique known as Contrastive Learning. This approach teaches the model to comprehend that similar representations of images and text should be placed close together in the latent space, while dissimilar ones should be pushed further apart. This helps CLIP to grasp the intricate relationships between images and their associated text descriptions, leading to more accurate and contextually relevant predictions.

You can learn more about CLIP here .

LLaMa

LLaMA (Large Language Model Meta AI) is a state-of-the-art foundational Large Language Model released by Meta as part of their commitment to open science. It is designed to help researchers advance their work in the subfield of AI, particularly in natural language processing.

LLaMA is a smaller and more performant model, which makes it accessible to researchers who don't have access to extensive computing power and resources. It is trained on a large set of unlabeled data, making it suitable for fine-tuning for various tasks, and is available in different sizes, ranging from 7B to 65B parameters. These parameters represent the learnable elements or weights in the language model, with larger values indicating a more powerful model capable of better understanding and generating human-like text.

You can read more about LLaMA here .

GPT-4

GPT-4, is the latest evolution of their now-famous Generative Pre-trained Transformers (GPT) series. Anchored in transformer architecture, GPT-4 is pre-trained on vast text corpora, enabling it to grasp intricate language structures, semantics, and contextual nuances. This pre-training phase equips GPT-4 with a foundation of linguistic knowledge, laying the groundwork for its text generation capabilities.

GPT-4 exceeds its predecessors by embracing multimodal capabilities, integrating visual inputs with its text processing functionality. This achievement is done by fusing the transformer architecture with mechanisms for interpreting images, presenting a significant stride in advancing AI's understanding of both textual and visual content.?

You can learn more about GPT-4 here .

PaLM

The Pathways Language Model (PaLM) stands as a remarkable achievement in AI language models, developed by Google as part of their research initiative. This initiative was inspired by the concept of "pathways," aimed to construct a single, powerful model capable of serving diverse applications. PaLM comes in various versions, with PaLM 2 being the latest iteration. Notable variants include Med-PaLM 2, finely tuned for medical and life sciences data, and Sec-PaLM, designed to expedite threat analysis in cybersecurity.

PaLM has the capability to generate text on any topic given a prompt, summarize large volumes of content, and even analyze text for sentiments and tones. The model's ability to reason has been enhanced through exposure to scientific literature and mathematical expressions, contributing to its proficiency in logical problem-solving.?

At its core, PaLM employs a transformer neural network model, a technology shared previously talked about model, GPT-4. This model undergoes rigorous training using Google's Pathways machine learning system, which leverages a few-shot learning technique. This enables PaLM to swiftly adapt and generalize new tasks with minimal labeled examples, making it highly adaptable and effective in generating coherent and contextually relevant responses.

You can learn more about PaLM here .

BART

BART (Bidirectional Auto-Regressive Transformers) is a model developed by Meta using the sequence-to-sequence Transformers architecture. BART can find itself being used in various comprehension tasks, abstractive dialogue, question answering, as well as summarization tasks. It uses a standard seq2seq/Neural Machine Translation architecture (read more about seq2seq here ) architecture along with a bidirectional encoder (similar to the BERT model), providing an alternate solution towards other pre-training schemes available in the market.?

BART can be said to be a combination of Google's BERT model as well as OpenAI's GPT. BERT's utilization of bidirectional and auto-encoder format allows the model to downstream tasks regarding the information of the whole input passage, yet it is not good for doing sequence generation tasks. GPT, on the other hand, is best to be used for text generation, but lacks the capability for downstream tasks which require knowledge of the whole input. BART, as it is made of a combination of both GPT and BERT, allows the adoption for the best of both worlds.

You can learn more about BART here .

Google LiT

In addition, Google has developed a new model called LiT (Locked-Image Tuning) that analyzes images and understands language together. It learns from a large amount of image and text data, without needing humans to manually label or organize the data.

LiT combines a pre-trained frozen image encoder and a text encoder, allowing it to work with both images and text effectively. In tests, LiT performed better than previously discussed CLIP, achieving higher accuracy on tasks like identifying objects in images. Even when using publicly available data, LiT remained strong and surpassed earlier models that had used less data.?

You can learn more about Google LiT here .

Implementation of LLM

Abstract

Now we’ll discuss a practical example on how to actually utilize an LLM-based model in practice. We will be using BART for this example now that we have given an overview of what it is. We will be using the coding language python for this section, and we would recommend that you run this on an integrated development environment (IDE) such as Google Colab or Jupyter Notebooks. We will not be going over how to set up these environments in this blog, but there are many online resources that can help guide you to set these up for yourself.

What we will be showcasing in this example is the technique of automatically assigning pertinent tags or labels to textual content based on their descriptions or characteristics, also known as auto-tagging.

Let’s begin this step-by-step tutorial.

Using BART

Step 1:?

The first step is to initialize our model. We will be using a pre-trained model from the HuggingFace Transformers library to set up our BART model as well as its tokenizer.?

The tokenizer will be responsible for converting human-readable text into a format that can be processed by our model.

mname = "cristian-popa/bart-tl-ng
tokenizer = AutoTokenizer.from_pretrained(mname)
model = AutoModelForSeq2SeqLM.from_pretrained(mname)        

Step 2:?

The next step is to take a lengthy text input, split it into individual sentences or documents, and then tokenize and encode those documents using the previously initialized tokenizer. This process of tokenization involves breaking down the text into smaller units, such as words or subwords, which are then assigned numerical values.?

These encoded documents are stored in the "encodings" variable, creating a structured format that can be seamlessly fed into a sequence-to-sequence language model for further analysis, interpretation, or generation of desired outputs.

input = ""
The rapid advancement of technology has transformed various aspects of our lives. From communication to transportation, technology has revolutionized the way we interact and navigate the world. The internet, in particular, has played a pivotal role in connecting people globally and providing access to a vast array of information.

In today's digital age, websites have become an integral part of businesses and organizations. A well-designed website can serve as a powerful tool to attract customers, showcase products or services, and enhance brand visibility. From e-commerce platforms to informative blogs, websites cater to diverse needs and purposes.

Search engines like Google have become indispensable for online users. They enable us to quickly find information, discover new websites, and navigate the vast online landscape. The process of search engine optimization (SEO) has emerged as a crucial aspect for website owners, aiming to improve their visibility and ranking in search engine results.

Furthermore, social media has transformed the way we connect and share content. Platforms such as Facebook, Twitter, and Instagram have created new avenues for communication, networking, and content distribution. Users can now easily share their thoughts, experiences, and multimedia content with a global audience.

As the digital ecosystem continues to evolve, user-generated content has gained significant prominence. Individuals can actively contribute to online platforms by creating and sharing their own content. This has given rise to online communities and content-driven platforms, fostering collaboration and engagement among users.

In summary, the internet, websites, search engines, and social media have become integral parts of our daily lives. They have reshaped communication, information access, and content creation. Understanding and leveraging these digital tools are essential for businesses, organizations, and individuals seeking to thrive in the digital age.
"""

# Split the input into sentences or documents
documents = input.split("\n\n")

# Tokenize and encode the documents
encodings = tokenizer(documents, truncation=True, padding=True, return_tensors="pt")"        

Step 3:

Our next step is to calculate the cohesion score. This score quantifies the interconnectedness of the document set and plays a vital role in revealing its overall thematic unity.?

To compute this score, we will leverage the concept of cosine similarity, a mathematical measure that gauges the similarity between vectors. Specifically, we will calculate the average cosine similarity between the document embeddings, which are numerical representations capturing the semantic essence of each document.?

The final results of these computations will be stored in the cohesion_score variable, serving as a tangible representation of how similar certain values are with each other.

# Generate embeddings for the document
with torch.no_grad():
embeddings = model.get_encoder()(
input_ids=encodings.input_ids.squeeze(),
attention_mask=encodings.attention_mask.squeeze()
).last_hidden_state

# Reshape embeddings to 2-dimensional shape
embeddings = embeddings.reshape(embeddings.shape[0], -1)

# Calculate cohesion as the average cosine similarity between document embeddings
cohesion_matrix = cosine_similarity(embeddings, embeddings)
cohesion_score = np.mean(cohesion_matrix)        

Step 4:?

The next step is to extract the top words from each document using our language model. For each document, it employs the model to generate a set of keywords, taking into account various parameters for optimal word selection.?

These generated keywords are then organized into a list, enabling a subsequent calculation of coherence. We then convert our sequence back into human-readable text using our tokenizer.

# Calculate coherence as the average cosine similarity between the top words of each document
top_words = []
for i, document in enumerate(documents):
# Generate the top words for each document using the model
outputs = model.generate(
input_ids=encodings.input_ids[i].unsqueeze(0),
attention_mask=encodings.attention_mask[i].unsqueeze(0),
max_length=15,
min_length=1,
do_sample=False,
num_beams=25,
length_penalty=1.0,
repetition_penalty=1.5
)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
top_words.append(decoded)t        

Step 5:?

We will now display the printed cohesion score, which is the average similarity between the document embeddings, and the coherence scores, representing the generated top words (summaries) for each document using the pre-trained language model.

# Print the cohesion and coherence score
# The cohesion score represents the average similarity between document embeddings
# The coherence scores represent the generated top words for each document.
print("Cohesion Score:", cohesion_score)
print("Coherence Scores:")
for i, document in enumerate(documents):
print(f"Document {i+1}: {top_words[i]}")        

To wrap things up, we should be left with this result:

Now that we’ve shown you how to utilize an LLM, you should have a good, basic understanding on how they work, and how they are typically used in implementation.?

Innovative features like auto-tagging are becoming increasingly important for businesses to stay competitive and provide a superior user experience as e-commerce continues to change the retail environment.

(Code written/reviewed by Asad Shahid, Zain Ali and Siraj Zahid)

Parting Thoughts

The world of LLMs holds massive potential and has transformed the landscape of artificial intelligence. These advanced models, such as GPT-3.5, BART, CLIP, and others, are reshaping the way we interact with technology and enabling unprecedented capabilities in natural language understanding, text generation, and even tasks such as image-text connections.?

At Sadiq, our innovators are eager to explore deeper into this domain, unveiling fresh opportunities and uses for these exceptional AI wonders. Keep an eye out for our upcoming blogs!

About the Authors

Zain Ali - ML Software Engineer

Located in the Dallas-Fort Worth Metropolitan area in Texas, Zain is a sharp student who is currently a Computer Science major studying at The University of Texas at Dallas (UTD). He hopes to create the world of tech a better place by showcasing his coding and mathematical knowledge in a delightful manner.

Asad Shahid - AI Research Associate

Emerging from the vibrant San Francisco Bay Area, Asad emerges as a perceptive scholar, propelled by an unwavering resolve to catalyze meaningful transformations and amass profound insights in the dynamic realms of software engineering and machine learning. Currently pursuing a Bachelor's degree in Computer Science, his endeavors encompass the expansive spectrum of software engineering and machine learning, all while nurturing an insatiable curiosity for perpetual growth and enlightenment within these captivating domains. He is always staying abreast on the latest technological innovations and business currents.

Siraj Zahid - AI Research Associate

Born and raised in the San Francisco Bay Area, Siraj currently studies at the University of California, Davis. He is a brilliant student pursuing a degree in Statistical Data Science as well as a minor in Computer Science and has already received a college diploma in Mathematics. He loves to work on everything data and software and is always up-to-date with the latest technology and business trends.

References

1. What is generative AI?
2. The Language Interpretability Tool (LIT): Interactive Exploration and Analysis of NLP Models ?
3. What’s next in large language model (LLM) research? Here’s what’s coming down the ML pike
4. CLIP: The Most Influential AI Model From OpenAI — And How To Use It
5. Introducing Fashion CLIP: Revolutionizing Fashion Material Detection
6. Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance
7. What is GPT-4? Here's everything you need to know