Day 22 — Gated Recurrent Units (GRU)

• Concept: Simplified LSTM.
• Implementation: Update gate.
• Evaluation: Performance, complexity.

CONCEPT

Gated Recurrent Units (GRUs) are a type of recurrent neural network (RNN) designed to handle the vanishing gradient problem that affects traditional RNNs. GRUs are similar to Long Short-Term Memory (LSTM) units but are simpler and have fewer parameters, making them computationally more efficient.

KEY FEATURES OF GRU

1. Update Gate: Decides how much of the previous memory to keep.
2. Reset Gate: Decides how much of the previous state to forget.
3. Memory Cell: Combines the current input with the previous memory, controlled by the update and reset gates.

KEY STEPS

1. Reset Gate: Determines how to combine the new input with the previous memory.
2. Update Gate: Determines the amount of previous memory to keep and combine with the new candidate state.
3. New State Calculation: Combines the previous state and the new candidate state based on the update gate.

IMPLEMENTATION

Let’s implement a GRU for a sequence prediction problem using Keras.

# Import necessary libraries

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GRU, Dense
from sklearn.preprocessing import MinMaxScaler

import warnings                                         
warnings.simplefilter(action = 'ignore')          

# Generate synthetic sequential data

data = np.sin(np.linspace(0, 100, 1000))        

# Prepare the dataset

def create_dataset(data, time_step=1):
    X, y = [], []
    for i in range(len(data) - time_step - 1):  # i is defined *inside* the loop
        a = data[i:(i + time_step)]
        X.append(a)
        y.append(data[i + time_step])
    return np.array(X), np.array(y)        

# Scale the data

scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data.reshape(-1, 1))  # Reshape for scaling        

# Create the dataset with timesteps

time_step = 10
X, y = create_dataset(data, time_step)        

# Reshape X for LSTM (if you're using one later) - Important!

X = X.reshape(X.shape[0], X.shape[1], 1)        

# Split the data into train and test sets

train_size = int(len(X) * 0.8)
X_train, X_test = X[:train_size], X[train_size:]
y_train, y_test = y[:train_size], y[train_size:]        

# Create the GRU model

model = Sequential([
    GRU(50, input_shape = (time_step, 1)),
    Dense(1)
])        

# Compile the model

model.compile(optimizer = 'adam', loss = 'mean_squared_error')        

# Train the model

model.fit(X_train, y_train, epochs = 50, batch_size = 1, verbose = 1)        

# Evaluate the model

loss = model.evaluate(X_test, y_test, verbose = 0)
print(f'Test Loss: {loss}')        

# Predict the next value in the sequence

last_sequence = X_test[-1].reshape(1, time_step, 1)
predicted_value = model.predict(last_sequence)
predicted_value = scaler.inverse_transform(predicted_value)
print(f'Predicted Value: {predicted_value[0][0]}')        

EXPLANATION OF THE CODE

1. Data Generation: We generate synthetic sequential data using a sine function.
2. Dataset Preparation: We create sequences of 10 time steps to predict the next value.
3. Data Scaling: Normalize the data to the range [0, 1] using MinMaxScaler.
4. Dataset Creation: Create the dataset with input sequences and corresponding labels.
5. Train-Test Split: Split the data into training and test sets.
6. Model Creation:

• GRU Layer: A GRU layer with 50 units.
• Dense Layer: A fully connected layer with a single output neuron for regression.

7. Model Compilation: We compile the model with the Adam optimizer and mean squared error loss function.

8. Model Training: Train the model for 50 epochs with a batch size of 1.

9. Model Evaluation: Evaluate the model on the test set and print the loss.

10. Prediction: Predict the next value in the sequence using the last sequence from the test set.

ADVANCED FEATURES OF GRUs

1. Bidirectional GRU: Processes the sequence in both forward and backward directions.
2. Stacked GRU: Uses multiple GRU layers to capture more complex patterns.
3. Attention Mechanisms: Allows the model to focus on important parts of the sequence.
4. Dropout Regularization: Prevents overfitting by randomly dropping units during training.
5. Batch Normalization: Normalizes the inputs to each layer, improving training speed and stability.

# Example with Stacked GRU and Dropout
from tensorflow.keras.layers import Dropout

# Create the stacked GRU model
model = Sequential([
    GRU(50, return_sequences=True, input_shape=(time_step, 1)),
    Dropout(0.2),
    GRU(50),
    Dense(1)
])

# Compile, train, and evaluate the model (same as before)
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_train, y_train, epochs=50, batch_size=1, verbose=1)
loss = model.evaluate(X_test, y_test, verbose=0)
print(f"Test Loss: {loss}")        

APPLICATIONS

GRUs are widely used in various fields such as:

• Natural Language Processing (NLP): Language modeling, machine translation, text generation.
• Time Series Analysis: Stock price prediction, weather forecasting, anomaly detection.
• Speech Recognition: Transcribing spoken language into text.
• Video Analysis: Activity recognition, video captioning.
• Music Generation: Composing music by predicting sequences of notes.

GRUs’ ability to capture long-term dependencies while being computationally efficient makes them a popular choice for sequential data tasks.

Download the Jupyter Notebook file for Day 22 here.