Multimodal Large Language Models (LLMs): From data management to training

Creation and Development of Multimodal LLMs

The creation of a Large Language Model (LLM), such as those used by ChatGPT or LLaMA, is an incredibly complex process requiring deep knowledge of several critical phases. Each phase contributes to the construction and training of a functional model. Below is a detailed description of each component, from managing input data to deployment, with links between various elements and numerous practical examples.

Overview of LLMs

Large Language Models (LLMs) are primarily designed to understand and generate natural language, which means their training and operation are heavily based on textual data. However, thanks to recent advancements in neural architectures, some LLMs have been extended to process not only text but also images, videos, and audio. This is possible by integrating advanced deep learning techniques that allow for the handling of multimodal inputs and making joint inferences across different modes.

These models use billions of parameters and are trained on massive datasets to perform a variety of tasks, including text completion, machine translation, and multimedia content generation based on descriptions.

For instance, ChatGPT can generate a complete article or answer questions based on an initial prompt, while models like DALL·E can transform textual descriptions into realistic images. These models are fundamental for applications like chatbots, virtual assistants, and recommendation systems.

Input Data Processing

LLMs can process different types of input data, including text, images, audio, and video. Each type of data requires specific preprocessing steps to be converted into a format compatible with the model's architecture.

For text, it's crucial to perform cleaning operations, such as removing special characters or typos, and normalizing the text. For example, "Il G@tto corrrrrre!!!" is transformed into "Il gatto corre." The text is then divided into fundamental units called tokens, a crucial step in the language processing workflow.

For images, operations include resizing the images to a standard size and normalizing pixel values, bringing them within a uniform range (e.g., between 0 and 1). Augmentation techniques may also be applied to create image variants, thus improving the model's robustness during training. This is important for models like DALL·E, which generate images based on textual descriptions.

In the case of audio, the process includes noise filtering and segmentation into logical parts, such as words or phrases. Audio can also be transformed into spectral representations to analyze frequencies over time. An audio file of a speech, for example, can be converted into a spectrogram for better analysis of sound components. In more recent models like DeepMind's Flamingo, audio is synchronized with images or video frames to ensure consistent multimodal understanding.

For videos, the process involves splitting them into individual frames, reducing the frame rate if necessary, and synchronizing audio and video to ensure that speech correctly matches lip movements. This is essential for applications such as generating realistic videos based on scripts.

Tokenization and Feature Extraction

Text

Text tokenization is the process of dividing text into fundamental units called tokens, which can be words, subwords, or individual characters. This process is crucial because it allows the model to transform a text sequence into a sequence of numerical representations (tokens) that can be processed.

Consider the sentence "Il gatto corre veloce" (The cat runs fast). Word-level tokenization would transform it into ["Il", "gatto", "corre", "veloce"]. However, in modern models, subword-level tokenization is often preferred to better handle uncommon or new words: the sentence could be tokenized as ["Il", "gat", "to", "cor", "re", "veloce"], where "gat" and "to" are subword segments. In models based on BERT or GPT, this technique helps reduce the vocabulary and improve the model's ability to handle new or rare words. Techniques like "byte pair encoding" (BPE) or "Unigram" tokenization are also frequently used in models like GPT-3 and T5.

Images

For images, the corresponding process to tokenization is called patching. Here, the image is divided into small sections called patches (e.g., 16x16 pixel blocks). Each patch is "flattened" into a feature vector and then converted into a numerical representation (embedding).

Consider a 224x224 pixel image. Using a Vision Transformer (ViT), the image is divided into 196 patches of 16x16 pixels each. Each patch is then transformed into an embedding vector, representing the visual features of that image section. This allows the model to analyze the image in detail, similarly to how text tokens are processed in language models.

Audio

For audio, feature extraction replaces tokenization. Audio is a continuous signal, so it is divided into small time segments called frames, from which specific features are extracted that represent the sound information in numerical form.

Suppose you have an audio file containing a speech. This audio is divided into 25-millisecond frames. From each frame, features like Mel-Frequency Cepstral Coefficients (MFCC) are extracted, which capture the spectral characteristics of sound, simulating how the human ear perceives sound frequencies. These MFCCs are then converted into numerical vectors that represent the audio in a form the model can process.

In more recent models like OpenAI's Whisper, audio is transformed into spectrograms, which are visual representations of the frequency distribution over time. These spectrograms are then treated like images, using patching techniques similar to those described for images.

Video

For videos, frame extraction and feature extraction are crucial. A video is divided into individual frames, which are static images captured at regular intervals. Each frame can be processed using the patching techniques described for images. Additionally, to capture temporal dynamics, features related to movement and frame order can be extracted.

A 60-second video at 30 fps contains 1800 frames. Each frame is treated as an image, divided into patches, and transformed into embeddings. Additionally, to capture temporal information, a temporal encoding is added, allowing the model to consider the order and sequence of frames, enabling the analysis of movement and actions in the video. In models like Timesformer, a Transformer specialized for video, frames are processed simultaneously, considering both spatial information (what happens in each frame) and temporal information (how frames are related over time), using "spatio-temporal attention" techniques for a deeper understanding.

Embedding Layer

The embedding layer is the phase where each token is transformed into a numerical vector through an embedding matrix. These vectors capture the semantic meaning of tokens and the relationships between them, allowing the model to understand how words or other tokens are related in the vector space.

For example, the words "cat" and "dog" will have similar vectors in the embedding space because they share semantic attributes, both being pets. These numerical vectors are used by the model to determine more complex relationships between tokens during training and inference. In multimodal models like CLIP, contrastive learning is used to improve embedding quality, aligning text and image vectors in the same vector space for a joint understanding of the modalities.

Positional Encoding

Positional encoding adds information about the position of tokens in the sequence, which is essential for Transformer-based models. These models do not have an intrinsic notion of order, unlike RNNs (Recurrent Neural Networks). Without a position indicator, a Transformer model would not be able to distinguish the order of tokens, which is crucial for understanding the meaning of the text.

For example, the position of "Il" in the sentence "Il gatto corre" (The cat runs) is different from the position of "corre", and this position information is added to the token embeddings through sinusoidal functions, ensuring that each token is uniquely encoded and that the model can understand the correct sequence. In modern multimodal models, positional encoding can also be learned rather than static, better adapting to non-sequential data like images and videos.

Transformer Blocks

The self-attention mechanism is the heart of the Transformer architecture and allows the model to determine the relative importance of each token concerning the others in the sequence. This mechanism enables the model to consider the global context, evaluating relationships between tokens even at a great distance.

For instance, in the sentence "Il gatto mangia il pesce" (The cat eats the fish), the self-attention mechanism allows the model to understand that "mangia" (eats) refers to "gatto" (cat) and "pesce" (fish), even if they are separated by other tokens. The model calculates an attention matrix to determine how much each token should "pay attention" to the others.

Multi-head attention allows the model to perform several self-attention calculations in parallel, capturing different aspects of the relationships between tokens. One head might focus on syntactic relationships (like subject-verb), while another might focus on semantic relationships (like an action and its object).

After self-attention, the data passes through a feed-forward network (FFN), a fully connected neural network that applies non-linear transformations to further enrich the token representation. For example, after self-attention has determined the relationships between "gatto" (cat), "mangia" (eats), and "pesce" (fish), the FFN processes this information to reinforce the model's understanding.

Residual connections and layer normalization are used to stabilize the training process. Residual connections add the original input of a layer to the output of the same layer, while layer normalization normalizes the output of each layer, preventing instabilities like vanishing gradients during training.

In large models, sparse attention techniques are often used to better manage very long sequences or multimodal attention, where attention weights are calculated between different types of data (text, image, etc.).

Layer Stacking

An LLM is composed of multiple stacked Transformer layers, each adding a new transformation to the data, improving the model's ability to understand complex linguistic structures. The initial layers focus on local and simple relationships, such as grammatical ones, while the deeper layers capture more abstract and global relationships.

For instance, in the initial layers, the model may understand that "gatto" (cat) is the subject and "mangia" (eats) is the verb. In the deeper layers, the model may understand that the sentence describes a specific action performed by a pet, and it can generate a more complex context, such as "The cat eats the fish while the dog watches."

In multimodal models, specialized layers are often used for each modality, followed by fusion layers that combine information from the different modalities, allowing for a more complete and integrated understanding.

Context Window

The context window represents the maximum number of tokens the model can consider simultaneously during processing. A wider context window allows the model to better understand the connections between words in a long text, maintaining coherence and relevance in the generated or processed language.

For instance, a model with a limited context window might "forget" parts of the conversation during a long conversation, negatively impacting the coherence of the responses. A model with a wider context window, on the other hand, can maintain the logical thread even after many interactions, remembering key details and connecting distant concepts in the text. For example, GPT-4, with a context window of thousands of tokens, can handle extended documents or long conversations without losing the initial context, ensuring relevant and coherent responses.

Output Layer

The output layer is the final stage of the model, where the processed internal representations are transformed into an interpretable format, such as a sequence of tokens (words) or other output forms (e.g., classification categories). This level is crucial because it determines the model's final output, which can be text, a classification decision, or other responses based on the processed context.

The process begins with projection into token space, where the outputs of the last Transformer layer are mapped onto a vocabulary representing all possible words or symbols the model can generate. For example, after processing the sentence "Il gatto corre" (The cat runs), the output layer calculates the probability that the next word might be "velocemente" (quickly), "nella" (in), "strada" (street), etc.

The Softmax function is then applied to transform these outputs into a probability distribution, indicating which token is most likely to be next in the sequence. For instance, if the model has determined that "corre" (runs) has a 70% chance of being followed by "velocemente" (quickly), this will be the word selected by the model.

In multimodal models, the output can also be conditioned by different modes. For example, an image might influence the choice of generated tokens, creating more coherent and contextually appropriate descriptions.

Training and Optimization

The training of an LLM is divided into two main phases: pre-training and fine-tuning.

During pre-training, the model is trained on enormous unlabeled datasets to learn basic language structures. The goal is for the model to acquire a general understanding of language that can be applied to a wide range of tasks. Common techniques include Masked Language Modeling (MLM), where parts of the text are masked, and the model must predict them, and Causal Language Modeling (CLM), where the model must predict the next token in a sequence. For example, in the sentence "Il ___ corre" (The ___ runs), the model might be trained to complete it with "gatto" (cat).

Fine-tuning occurs after pre-training and is aimed at optimizing the model for specific tasks using smaller, targeted datasets. This step is crucial for improving the model's performance in specific applications, such as sentiment classification or machine translation. For example, a model trained on general texts can be fine-tuned on labeled reviews to recognize the sentiment (positive or negative) of the reviews themselves.

Techniques like prompt tuning or adapter layers are also used, allowing for more efficient fine-tuning without having to modify all the model's parameters, quickly adapting it to new specific tasks.

Infrastructure and Computational Resources

Training an LLM requires powerful computational resources, such as high-end GPUs or TPUs (Tensor Processing Units), and efficient data parallelism management. The training process is distributed across multiple GPUs to accelerate the process and manage very large models. For example, training GPT-3, with 175 billion parameters, required an infrastructure composed of thousands of GPUs operating in parallel for several weeks.

Data management is equally critical. Datasets must be prepared, cleaned, and pre-processed to be efficiently used during training. Managing data storage and throughput is essential to avoid bottlenecks that could slow down model training.

Optimization techniques like mixed precision training are used to speed up training and reduce memory consumption, enabling the training of very large models on available hardware.

Security, Bias, and Ethical Considerations

LLMs can reflect and amplify biases present in the training data, so it is essential to apply techniques to reduce these biases. During the preprocessing phase, it is possible to identify and mitigate bias in input data. For example, if a dataset contains mostly texts written by a particular demographic group, the model may develop cultural biases. Monitoring and correcting the model is essential to prevent it from generating discriminatory or inappropriate responses.

Techniques like fairness-aware learning and debiasing during training are also used to address biases in multimodal models. These approaches aim to ensure that the model generates fair and representative outputs, avoiding perpetuating stereotypes or discrimination.

Security is another crucial aspect. Mechanisms must be implemented to prevent the model from being used for harmful purposes, such as generating disinformation or dangerous content. For example, a model used in the medical field must be rigorously controlled to prevent it from suggesting risky behaviors or providing unvalidated medical advice.

Monitoring and Evaluation

The model's performance must be continuously monitored using metrics such as accuracy, recall, and F1-score to ensure that the model functions as expected. This monitoring is critical to detect any issues or performance degradation, especially after deployment.

For instance, if a chatbot starts making errors on a particular type of query, additional retraining or fine-tuning may be necessary. The model must also be tested in real-world scenarios to ensure it functions correctly in all possible use cases. For example, a chatbot must be tested on thousands of possible conversations to ensure it responds consistently and usefully.

In multimodal models, it is essential to use specific metrics for each mode, such as mean average precision (mAP) for images or videos, to ensure that the model performs well in all input modes.

Continual Learning and Fine-Tuning

Continual learning allows the model to keep learning after deployment, updating its parameters with new data without forgetting what it has already learned. This is particularly useful in contexts where language and user needs are constantly evolving. For instance, a model used for customer support can continue to improve its responses as it acquires more data on user interactions.

Domain-specific fine-tuning allows further optimization of the model for specific applications or sectors. For example, a generic model can be fine-tuned for use in the medical field, learning specific terminology and response protocols.

Techniques like meta-learning are used to enable models to quickly adapt to new tasks with few data, making continual learning more effective and less resource-intensive.

Retrieval-Augmented Generation (RAG)

In addition to fine-tuning, an emerging and powerful technique is Retrieval-Augmented Generation (RAG), which combines text generation with the retrieval of relevant information from a large data corpus. In practice, when the model receives input, it first searches for relevant information in an external database and then generates a response based on both the input and the retrieved data.

This approach improves the accuracy and relevance of responses, as the model can access up-to-date and specific information, reducing the likelihood of providing incorrect or outdated answers. For instance, a RAG model could be used in the legal field to retrieve previous cases and generate analyses based on updated legal data or in the healthcare field to provide advice based on recent medical research.

Retrieval in a RAG system can also be conditioned by multimodal queries, where textual and visual context is used to improve information retrieval, making responses even more precise and contextually relevant.

Conclusion

Creating and training an LLM requires a deep understanding of every phase of the process, from initial data processing to final model optimization. By carefully connecting the various architectural and technical components, an LLM can be developed to tackle a wide range of tasks with high accuracy and reliability. Ensuring the model's security, fairness, and continuous evolution, including the integration of advanced techniques like RAG, is essential to guarantee that an LLM not only performs well but is also safe and useful in the long term.

If you enjoyed this article, you can read the previous one or continue with the next article!

Ing. Giovanni Masi

www.dhirubhai.net/in/giovanni-masi

Email: [email protected]