Connecting the Dots: How NLP, Tokenization, Embeddings, Hidden Vectors and Translated Work Together in Generative AI (Part 1)

In this 6-part series, we'll explore the details of Generative AI, starting with its fundamental building blocks. To understand how Generative AI works its magic, let's dive into... "Building Blocks of Generative AI"!

To build your understanding of Generative AI, let's explore these essential components.

• Natural Language Processing (NLP) : The Power of Language.
• Language Models: Bringing NLP to Life?
• Generative vs. Classification Models: What's the Difference?
• Overall Architectural Flow

Natural Language Processing (NLP) : The Power of Language

• NLP, which stands for Natural Language Processing, is a field within computer science (and often overlaps with Artificial Intelligence) that focuses on enabling computers to understand and process human language.? It's not a single concept or algorithm, but rather a collection of techniques that allow machines to analyze, manipulate, and even generate human language.
• NLP is in the technology field for several years but it came into buzz after Open AI - Chat GPT (Generative AI). The reason is Generative AI is basically accepting the human inputs as prompts , understands the language symantics, context and provide the response as Q/A or Summarize or other valuable response.
• NLP achieves human-computer interaction through two key functionalities: Natural Language Understanding (NLU) and Natural Language Generation (NLG). NLU allows computers to grasp the meaning of our words (text), like intent and sentiment. NLG lets them generate human-like text, translating languages or creating new content.
• Think of NLP as the overall blueprint for building systems that understand and generate human language, defining the tasks (like NLU and NLG) and techniques needed to achieve them. Language models are a powerful implementation of NLP concepts, particularly for NLU and NLG tasks that are crucial for Generative AI applications. While other tools can also implement NLP functionalities, for the purpose of discussing Generative AI, we will focus on language models.

Language Models: Bringing NLP to Life

We explored the building blocks of NLP - NLU and NLG. Now, let's see them come alive! Language models put these functionalities into action.

• At the heart of many NLP applications lie language models. These are powerful computer programs that leverage algorithms, often deep learning, to analyze and generate human language. They can handle tasks like understanding the meaning of text (NLU) and crafting human-like responses (NLG).
• Unlike a one-size-fits-all solution, language models adapt to specific tasks.? They leverage powerful algorithms, often deep learning,? to tackle different NLP challenges.? This specialization is why you see such a diverse range of language models available,? each focusing on tasks like translation, question answering, or even creative text generation.
• While many deep learning algorithms exist, some play a particularly important role in various AI applications, including language models. Here are a few key ones: Recurrent Neural Networks (RNNs) | Long Short-Term Memory (LSTMs) networks | Transformers | Generative Adversarial Networks (GANs). This is not a complete list of algorithms, but it represents some of the most important foundational concepts for developers working in generative AI. Understanding these algorithms is very important.
• Now that we've explored core algorithms, the possibilities are endless! Popular Python libraries like PyTorch and scikit-learn empower you to implement language models tailored to your specific use case.? Imagine the applications you can build!
• Having explored language models,? we can now explore their different specializations. The next section dives into generative and classification models, two fundamental approaches within the language modeling world.

Generative vs. Classification Models: What's the Difference?

Language models can be classified into two main categories: generative and classification models. Let's look into the difference between them.

• Generative Models: These models are designed to generate new text data. They often leverage algorithms like RNNs or transformers to analyze vast amounts of text and learn the underlying patterns. Based on this knowledge, they can then create new text formats,? translate languages, write different kinds of creative text formats of content, or even? hold conversations that resemble human-written text.
• Classification Models: These models are designed to analyze vast amounts of text data and categorize it into predefined classes. They often leverage algorithms like CNNs or Naive Bayes to learn the underlying patterns within the text. Based on this knowledge, they can then classify the input content for various purposes. Some popular use cases include sentiment analysis (determining if a review is positive or negative) and spam filtering.

What are Large Language Models?

As we continue our exploration of language models, let's turn our focus to Large Language Models (LLMs). These models have been making waves in the NLP field, so let's understand what they are and how they differ from other language models.

• Large Language Models (LLMs) are a type of language model that's trained on massive amounts of text data. This vast training allows them to tackle complex tasks.? They often leverage deep learning algorithms, particularly transformers, to process information and generate human-like text.
• Different companies like Facebook, Google, and OpenAI release different LLMs trained on massive amounts of data. These LLMs are designed for various AI applications.
• Domain-specific data is crucial for many AI applications. While you can train your own LLM from scratch using your data, it's a complex and resource-intensive process.? An alternative approach is to leverage a pre-trained LLM from a vendor or open-source project. These models are already trained on vast amounts of general text data. You can then fine-tune this pre-trained LLM with your specific domain data to tailor it for your organization's unique use cases.
• Popular platforms like Hugging Face offer public repositories of pre-trained models. These repositories include both open-source and paid versions, catering to various NLP tasks. While some of these models are indeed Large Language Models (LLMs), the repository encompasses a broader range of NLP models. You can also use the other vendors like OpenAI, Google AI, Amazon SageMaker, to get access or use the pre-trained models for your custom needs.

Overall Architectural Flow

Now you might be wondering: how do these LLMs truly work? What goes on under the hood when a user feeds them text? In this section, we'll delve into the overall architectural flow of Large Language Models (LLMs). At the heart of this process lies a powerful concept called transformers, a type of deep learning architecture. We will explore how transformers work within LLMs to process user input, understand the context, and generate those remarkable responses.? While other algorithms like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) have played a role in NLP, transformers have become particularly effective at handling complex language tasks.

Let’s divide into 4 main layers : (Encoder-Decoder Architecture)

• Pre-Processing Layer - This stage prepares the input text for the model.
• Encoder Layer - The core processing layer that analyzes the input text using the transformer architecture.
• Decoder Layer - This layer utilizes the encoded information to generate the output text word by word.
• Output Layer -? Final layer of the LLM to predict and generate the content / text.

Pre-Processing Layer

The pre-processing layer plays a crucial role in transforming raw text into a format suitable for the model's processing capabilities. This stage involves tasks like tokenization and embedding, essentially bridging the gap between human language and the numerical world of vectors. Let’s see the details of ‘Tokenization’ and ‘Embedding’.

• Tokenization: ?In the world of LLMs, tokenization is the fundamental process of splitting raw text into smaller units called tokens. These tokens can be words, sub-words (like parts of a word), or characters depending on the chosen method. There are several tokenization methods used in LLMs:

Word-Level Tokenization | Sub-Word Tokenization | Character-Level Tokenization

Each method of Tokenization offers unique advantages and drawbacks. Word-level tokenization is easy to understand but may not handle unseen words well. Sub-word tokenization addresses this but leads to a larger vocabulary. Character-level tokenization can be achieved using basic string manipulation techniques. We'll explore these concepts further, along with practical code examples, in a future part. PyTorch offers built-in functionalities for tokenization through the torchtext library.

For Example: Word Level Tokenization

import torchtext

# Example for Word-Level Tokenization
text = "Example for Word Level Tokenization”
tokenizer = torchtext.data.get_tokenizer("basic_english")
tokens = tokenizer(text)

print(tokens)  # Output: ['Example ', 'for', 'Word', 'Level', 'Tokenization']
        

• Embeddings : ??After tokenization, we transform the tokens into numerical representations called embeddings. These embeddings reside in a high-dimensional vector space, where each dimension captures semantic aspects of the word and its relationships to other words. This allows the LLM model to efficiently process and understand the text. For instance, the embedding for "king" might be closer to the embedding for "queen" in this vector space compared to the embedding for "car" due to their semantic similarity. By utilizing these embeddings, the model can perform tasks like text generation or machine translation, where understanding the relationships between words is crucial.

o?? This link provides a deeper understanding of embeddings, including how they represent word relationships in multi-dimensional spaces and how we can measure the distance between words to quantify their similarity - https://www.cs.cmu.edu/~dst/WordEmbeddingDemo/tutorial.html

o?? Word2Vec is a popular algorithm to implement embeddings in multidimensional vector space. This is a specific algorithm designed to efficiently learn word embeddings from a large corpus (collection of data) of text. It analyzes the context in which words appear to capture semantic relationships and represent them as numerical vectors. While Word2Vec is a popular choice, it’s not the only option for creative embeddings. Other algorithms like GloVe or methods based on neural networks are also used.

o?? ?This blog clearly explains the Word2Vec algorithm in detail - https://jalammar.github.io/illustrated-word2vec/

Encoder Layer

Now that we have completed the pre-processing steps and have our word embeddings ready, we can feed them to the encoder layer. The encoder layer is responsible for processing the sequence of embeddings and extracting meaningful information that captures the context of the text. Depending on our specific use case (classification, generation, or general contextual representation), we will choose the appropriate encoder architecture (e.g., LSTMs for RNNs, Transformers) to process the embeddings effectively. Encoders use neural networks, a Deep Learning technique, to process text data. We choose the specific encoder design (e.g., LSTMs, Transformers) based on our NLP task.

• RNNs and Transformers use multiple encoder layers to process a sequence of word embeddings. As the embeddings pass through each layer, the network performs calculations using weights (parameters) to understand word relationships and context. These weights are adjusted during training based on the input embeddings and the desired task, allowing the model to learn effectively. The number of input embeddings determines the vocabulary size, while the number of parameters reflects the overall model complexity.
• Having more layers to process information is a significant factor in determining model complexity and the total number of parameters. This allows the model to learn more intricate relationships within the data and potentially achieve better performance on the NLP task.

• The encoder layer typically outputs a sequence of hidden vectors. The primary purpose of the encoder layer's output is to provide a more informative representation of the input sequence, capturing not just the meaning of individual words but also the relationships and context between them. This enriched representation is then used by subsequent layers in the model. ?We will cover more details about the RNN / Transformer architecture in future article.
• Hidden vectors can be seen as a compressed representation of the "knowledge" gained by the encoder layer while processing the sequence of word embeddings. This knowledge includes:
• Individual Word Meaning: The core meaning of the word itself is still present.
• Contextual Information: The encoder layer considers the word's position, relationship to surrounding words, and the overall context of the text. This contextual information enriches the hidden vector, providing a more nuanced understanding.
• Understanding of relationships between Weights / Parameters vs Hidden vectors : ?Weights (or parameters) are adjustable values associated with the connections between neurons in the encoder layer. During training, these values are adjusted based on the training data and the desired NLP task. This adjustment process allows the model to learn how to transform the input embeddings into a sequence of informative hidden vectors. These hidden vectors capture the context and relationships between words, making them crucial for the model to perform effectively.

Decoder Layer

Decoder layer uses the hidden vectors produced by the Encoder Layer to generate the output, such as a translated sentence or a summarized text, depending on the specific NLP task.

• Step-by-Step Processing: The decoder takes the sequence of hidden vectors as input and processes them one element at a time. At each step, it predicts the next element (word) in the target sequence based on its internal state and the information from the encoder hidden vectors.
• Architecture Dependency: The specific mechanism for processing these vectors depends on the type of neural network used in the decoder. There are two main architectures: RNNs (Recurrent Neural Networks) and Transformers.

Output Layer

Encoder-decoder models translate hidden vectors (think compressed meaning of source text) into human-readable text like translated sentences or summaries. The decoder works step-by-step, predicting the next word in the target sequence at each step. It assigns probabilities to each word in the vocabulary, indicating how likely it is the next element. The output layer focuses on the final probability distribution for the last word, essentially the most likely way to complete the sequence. It often picks the word with the highest probability, but some models might explore multiple high-probability options during decoding using techniques like beam search. This collaborative effort across layers helps convert numerical representations into human-readable text.

In this first part, we've explored the essential building blocks that power generative AI's ability to translate languages. We've seen how NLP techniques like tokenization, embeddings, and hidden vectors work together to prepare the data and capture the meaning within text.

Now that we have these foundational concepts in place, Part 2 will delve into the specifics of how a generative AI model translates text. We'll see how these building blocks come together to bridge the gap between languages, transforming the source text into a new, understandable form.