Artificial Intelligence - Part 7.2 - GENERATIVE AI - Transformer Models

What Are Transformer Models?

Transformer models are deep learning architectures that excel at handling sequential data, such as text, time-series data, and speech. Unlike traditional sequence-based models like recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), transformers process entire input sequences simultaneously. This capability is enabled by a mechanism called self-attention, which allows transformers to weigh the importance of each input element relative to others.

How Do Transformers Work?

Transformers consist of several key components, each playing a crucial role in data processing and transformation. Here's a detailed breakdown of their workings:

1. Input Representation

The input data, whether text, images, or other sequences, needs to be converted into a numerical form that the model can process.

• Textual Data: Words or tokens are represented as vectors using embeddings like Word2Vec, GloVe, or learned embeddings during training.
• Positional Encoding: Since transformers process data non-sequentially, positional encodings are added to the input embeddings to provide information about the order of tokens in a sequence. These encodings use sine and cosine functions to introduce position-related patterns.

Example: For the sentence "Transformers are amazing," the word "Transformers" might be mapped to a vector like [1.2, 0.3, -0.5], and a corresponding positional encoding [0.1, 0.05, 0.01] is added.

2. Self-Attention Mechanism

The self-attention mechanism is the heart of transformers. It allows the model to focus on different parts of the input sequence when processing each token.

• Query, Key, and Value (Q, K, V):
• Attention Score Calculation: Attention scores are computed by taking the dot product of the Query vector for one token with the Key vectors of all tokens. The scores are normalized using a softmax function.

Here, dk is the dimension of the Key vector.

Example: In the sentence "She saw the cat," when processing the word "she," the model might assign higher attention to "saw" and "cat" because they are contextually related.

3. Multi-Head Attention

Instead of computing attention once, transformers use multi-head attention, which allows the model to capture different types of relationships in the data. Multiple attention mechanisms (or heads) operate in parallel, and their outputs are concatenated and projected into a single representation.

Example: One attention head might focus on grammatical relationships, while another identifies semantic connections.

4. Feedforward Neural Networks

After the self-attention layer, the data passes through a fully connected feedforward neural network. This network adds non-linearity and helps in learning complex transformations.

5. Residual Connections and Layer Normalization

• Residual Connections: Shortcut connections skip layers, allowing the model to retain information from earlier layers.
• Layer Normalization: Ensures stable training by normalizing the inputs to each layer.

6. Encoder-Decoder Architecture

Transformers are composed of an encoder and a decoder.

• Encoder: Processes the input sequence and generates a context-rich representation.
• Decoder: Uses the encoder’s output and previously generated tokens to predict the next token in a sequence.

Example: In machine translation, the encoder processes the sentence "I love AI" and the decoder generates its French translation, "J'aime l'IA."

Training Transformers

Training transformers involves two main phases:

1. Pre-training

The model is trained on massive datasets to learn general language patterns or representations.

• Masked Language Modeling (MLM): Used in BERT, where some input tokens are masked, and the model predicts them based on the context.
• Causal Language Modeling (CLM): Used in GPT, where the model predicts the next token in a sequence.

Example: For the sentence "The [MASK] is blue," the model predicts the missing word, "sky."

2. Fine-tuning

The pre-trained model is adapted for specific tasks like sentiment analysis, summarization, or translation using domain-specific datasets.

Applications of Transformers

Transformers have revolutionised AI across multiple domains. Here are some notable applications:

1. Natural Language Processing (NLP)

• Machine Translation: Tools like Google Translate use transformers to translate text between languages.
• Text Summarization: Models like T5 generate concise summaries of lengthy documents.
• Chatbots and Assistants: GPT-based systems power virtual assistants like ChatGPT.

Example: A transformer model generates a response to the query, “What is the weather today?”

2. Computer Vision

• Image Classification: Vision Transformers (ViTs) classify images into categories like "dog" or "cat."
• Object Detection: Transformers identify objects and their locations in images.

Example: A Vision Transformer classifies an image as "a sunny beach with palm trees."

3. Speech Recognition

• Speech-to-Text: Tools like Whisper transcribe spoken words into text.
• Text-to-Speech: Transformers synthesize natural-sounding speech.

Example: A model converts “Can you play some music?” into text for a voice assistant.

4. Generative Models

• Image Generation: Models like DALL-E create images from textual descriptions.
• Text Generation: GPT models generate coherent and contextually relevant text.

Example: DALL-E generates an image of "a futuristic cityscape with flying cars."

5. Healthcare

• Drug Discovery: Transformers analyze molecular data to identify potential drug candidates.
• Medical Imaging: Transformers assist in diagnosing diseases from X-rays or MRIs.

Example: A transformer identifies signs of pneumonia in a chest X-ray.

Advantages of Transformers

1. Parallel Processing: Faster training compared to sequential models like RNNs.
2. Scalability: Can be scaled to handle billions of parameters, as seen in GPT-4.
3. Versatility: Adaptable to various tasks and modalities (text, images, audio).

Challenges of Transformers

1. High Computational Cost: Training large models requires significant resources.
2. Memory Usage: Storing attention weights and large embeddings can be demanding.
3. Data Dependency: Performance depends heavily on the quality and quantity of training data.

Implementation Example

Here’s a simplified implementation of a transformer encoder using Python and TensorFlow/Keras

Example 1

import tensorflow as tf
from tensorflow.keras import layers

class TransformerEncoder(layers.Layer):
    def __init__(self, num_heads, d_model, dff, rate=0.1):
        super().__init__()
        self.attention = layers.MultiHeadAttention(num_heads, d_model)
        self.ffn = tf.keras.Sequential([
            layers.Dense(dff, activation='relu'),
            layers.Dense(d_model)
        ])
        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = layers.Dropout(rate)
        self.dropout2 = layers.Dropout(rate)

    def call(self, x, training):
        attn_output = self.attention(x, x)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(x + attn_output)
        
        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)        

Example 2

import tensorflow as tf
from tensorflow.keras import layers

# Define the scaled dot-product attention mechanism
def scaled_dot_product_attention(query, key, value):
    matmul_qk = tf.matmul(query, key, transpose_b=True)
    dk = tf.cast(tf.shape(key)[-1], tf.float32)
    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)
    attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)
    output = tf.matmul(attention_weights, value)
    return output

# Define a multi-head attention layer
class MultiHeadAttention(layers.Layer):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_model = d_model

        self.depth = d_model // num_heads
        self.wq = layers.Dense(d_model)
        self.wk = layers.Dense(d_model)
        self.wv = layers.Dense(d_model)
        self.dense = layers.Dense(d_model)

    def split_heads(self, x, batch_size):
        x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))
        return tf.transpose(x, perm=[0, 2, 1, 3])

    def call(self, query, key, value):
        batch_size = tf.shape(query)[0]

        query = self.split_heads(self.wq(query), batch_size)
        key = self.split_heads(self.wk(key), batch_size)
        value = self.split_heads(self.wv(value), batch_size)

        attention = scaled_dot_product_attention(query, key, value)
        attention = tf.transpose(attention, perm=[0, 2, 1, 3])
        concat_attention = tf.reshape(attention, (batch_size, -1, self.d_model))
        output = self.dense(concat_attention)
        return output

# Transformer Encoder Block
class TransformerEncoderBlock(layers.Layer):
    def __init__(self, d_model, num_heads, dff, rate=0.1):
        super().__init__()
        self.mha = MultiHeadAttention(d_model, num_heads)
        self.ffn = tf.keras.Sequential([
            layers.Dense(dff, activation='relu'),
            layers.Dense(d_model)
        ])
        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = layers.Dropout(rate)
        self.dropout2 = layers.Dropout(rate)

    def call(self, x, training):
        attn_output = self.mha(x, x, x)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(x + attn_output)

        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        out2 = self.layernorm2(out1 + ffn_output)
        return out2        

Conclusion

Transformer models have redefined the boundaries of AI, enabling breakthroughs in fields ranging from NLP and computer vision to healthcare and beyond. With their unique architecture and self-attention mechanism, transformers offer unparalleled flexibility and performance. As research and innovation continue, the potential of transformers to address complex challenges in AI and real-world applications is virtually limitless.