How GPT Models Are Trained
Ever wondered how powerful AI models like ChatGPT are created? Let’s break down the GPT Training Process into simple steps with examples, Python code snippets, and clear explanations. Perfect for high school students curious about artificial intelligence! ???
1. Data Collection ??
What It Is:
We start by gathering a massive amount of text data from various sources like books, websites, and articles. This diverse data helps the model understand language in all its forms.
Example Input:
"The sun rises in the east."
"Birds chirp in the morning."
"Nature is beautiful."
Python Code:
# Example dataset
text_data = [
"The sun rises in the east.",
"Birds chirp in the morning.",
"Nature is beautiful."
]
print(text_data)
Output:
["The sun rises in the east.", "Birds chirp in the morning.", "Nature is beautiful."]
We collect and store sentences like these to teach the model about language.
2. Tokenization ??
What It Is:
Tokenization breaks down sentences into smaller pieces called tokens (like words or subwords). Each token is assigned a unique number (ID) that the model can understand.
Example Input:
Sentence: "The sun rises in the east."
Python Code:
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
# Tokenize example sentences
tokens = tokenizer(text_data, return_tensors="pt", padding=True, truncation=True)
print(tokens['input_ids'])
Output:
tensor([[464, 1859, 2425, 287, 262, 2019, 13],
[3499, 14106, 287, 262, 1626, 13, 0],
[8503, 318, 1966, 13, 0, 0, 0]])
Each number represents a unique token ID corresponding to words in the sentences.
3. Input Embedding ??
What It Is:
Embeddings convert token IDs into dense vectors (lists of numbers) that capture the meaning of each token. These vectors allow the model to understand relationships between words.
Example Input:
Token IDs: [464, 1859, 2425, 287]
Python Code:
import torch
# Create embedding layer
embedding_layer = torch.nn.Embedding(50257, 768) # GPT-2 vocab size and embedding dim
input_ids = torch.tensor([[464, 1859, 2425, 287]])
embeddings = embedding_layer(input_ids)
print(embeddings)
Output:
tensor([[[ 0.1234, -0.5678, 0.9101, ..., 0.1122, -0.3344, 0.5566],
[ 0.2233, -0.6677, 1.0102, ..., 0.2122, -0.4344, 0.6566],
[ 0.3232, -0.7676, 1.1103, ..., 0.3122, -0.5344, 0.7566],
[ 0.4231, -0.8675, 1.2104, ..., 0.4122, -0.6344, 0.8566]]],
grad_fn=<EmbeddingBackward0>)
Each token ID is now represented by a vector of numbers that capture its meaning.
4. Positional Encoding ???
What It Is:
Positional Encoding adds information about the position of each token in the sentence. This helps the model understand the order of words.
Example Input:
Embedding size: 4 tokens x 768 dimensions
Python Code:
import math
import torch
def positional_encoding(seq_len, d_model):
position = torch.arange(seq_len).unsqueeze(1).float()
div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
pe = torch.zeros(seq_len, d_model)
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
return pe
pos_encoding = positional_encoding(seq_len=4, d_model=768)
print(pos_encoding)
Output:
tensor([[ 0.0000, 1.0000, 0.0000, 1.0000, ..., 0.0001, 1.0000],
[ 0.8415, 0.5403, 0.8415, 0.5403, ..., 0.0002, 1.0000],
[ 0.9093, -0.4161, 0.9093, -0.4161, ..., 0.0003, 1.0000],
[ 0.1411, -0.9900, 0.1411, -0.9900, ..., 0.0004, 1.0000]])
Positional encoding vectors are added to the embeddings to provide information about each token's position in the sequence.
5. Transformer Block ?
What It Is:
The Transformer Block is the core of GPT models. It processes the embeddings to understand the context and relationships between words using:
Example Input:
Embeddings with positional encodings: tensor([...]) (as shown above)
领英推荐
Python Code:
transformer_layer = torch.nn.TransformerEncoderLayer(d_model=768, nhead=8)
output = transformer_layer(embeddings + pos_encoding.unsqueeze(0))
print(output)
Output:
tensor([[[ 0.5678, -1.2345, 0.6789, ..., 0.3456, -0.7890, 0.1234],
[ 0.6789, -1.3456, 0.7890, ..., 0.4567, -0.8901, 0.2345],
[ 0.7890, -1.4567, 0.8901, ..., 0.5678, -1.0012, 0.3456],
[ 0.8901, -1.5678, 0.9012, ..., 0.6789, -1.1123, 0.4567]]],
grad_fn=<AddBackward0>)
The transformer processes the embeddings and outputs updated vectors that capture contextual information.
6. Output Projection & Softmax ??
What It Is:
The model's final embeddings are transformed back into a large set of numbers corresponding to the vocabulary. Softmax then converts these numbers into probabilities, predicting the next word in the sequence.
Example Input:
Contextual embeddings: tensor([...]) (as shown above)
Python Code:
output_layer = torch.nn.Linear(768, 50257) # Vocabulary size
logits = output_layer(output) # Logits before softmax
# Apply softmax
softmax = torch.nn.Softmax(dim=-1)
probs = softmax(logits)
print(probs)
Output:
tensor([[[1.2e-05, 3.4e-04, 5.6e-03, ..., 2.3e-02, 4.5e-01, 1.1e-03],
[2.3e-05, 4.5e-04, 6.7e-03, ..., 3.4e-02, 5.6e-01, 2.2e-03],
[3.4e-05, 5.6e-04, 7.8e-03, ..., 4.5e-02, 6.7e-01, 3.3e-03],
[4.5e-05, 6.7e-04, 8.9e-03, ..., 5.6e-02, 7.8e-01, 4.4e-03]]],
grad_fn=<SoftmaxBackward0>)
Each number represents the probability of a specific word being the next word in the sequence.
7. Loss Calculation ??
What It Is:
Loss Calculation measures how well the model's predictions match the actual target tokens. We use Cross-Entropy Loss for this purpose.
Example Input:
Python Code:
from torch.nn import CrossEntropyLoss
predicted_logits = logits.view(-1, 50257) # Flatten logits
target_ids = torch.tensor([464, 1859, 2425, 287]) # Target tokens
loss_fn = CrossEntropyLoss()
loss = loss_fn(predicted_logits, target_ids)
print(f"Loss: {loss.item()}")
Output:
Loss: 10.34 # Example loss value
Lower loss values indicate better predictions.
8. Backpropagation & Optimization ??
What It Is:
Backpropagation calculates gradients (how much each weight contributes to the loss). Optimization uses these gradients to adjust the model's weights, improving future predictions. We typically use the Adam optimizer for this.
Python Code:
optimizer = torch.optim.Adam(transformer_layer.parameters(), lr=1e-4)
optimizer.zero_grad() # Clear previous gradients
loss.backward() # Backpropagate the loss
optimizer.step() # Update the weights
print("Weights updated!")
Output:
Weights updated!
The model's weights are now slightly adjusted to reduce future loss.
9. Inference (Text Generation) ??
What It Is:
Inference is when the trained model generates new text based on a given prompt. It uses the learned patterns to predict and create coherent sentences.
Example Input:
Prompt: "Once upon a time"
Python Code:
from transformers import GPT2LMHeadModel
# Load GPT-2 model
model = GPT2LMHeadModel.from_pretrained("gpt2")
# Generate text
input_ids = tokenizer.encode("Once upon a time", return_tensors="pt")
output = model.generate(input_ids, max_length=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Output:
"Once upon a time there was a young boy who dreamed of becoming an explorer."
The model continues the story based on the initial prompt.
?? Final Summary
Here’s how GPT models are trained, step by step:
This guide provides a complete overview of the GPT training process with clear examples, Python code, and outputs for each step. I hope this makes understanding GPT models easier and more accessible! ??