Autoencoders Dimensionality Reduction with Example

In the world of machine learning, dimensionality reduction is a critical process, especially in tasks involving high-dimensional data like images. Autoencoders, a type of neural network, are particularly useful for this purpose. They not only reduce data dimensions but also assist in learning efficient data codings in an unsupervised manner. This article explores how an autoencoder can compress an 8x8 image of the digit '4' into a 2x2 latent representation and the implications of this transformation.

This article is a continuation of the following article:

From Pixels to Features: The Compression Journey

An autoencoder consists of two main components: the encoder and the decoder. The encoder compresses the input and the decoder attempts to recreate the input from this compressed data. Here’s how it applies to an 8x8 image of the digit '4':

• Original Data Representation: Initially, the image of '4' is represented by an 8x8 matrix, equating to 64 individual pixel values. Each pixel represents a dimension in the space where the image resides.

• Reduced Data Representation: The autoencoder transforms this 64-dimensional input into a 4-dimensional (2x2) latent matrix. This reduction is achieved through a process where the network learns to keep only the most essential information, sufficient to allow the decoder to reconstruct the image.

• The Encoding Process: During training, the encoder adjusts itself to map the high-dimensional data into a low-dimensional space, the latent space. The values in this 2x2 latent matrix are features that the network has learned; they are not pixel values but rather represent higher-level characteristics of the image like shapes and edges.
• Decoding - The Reconstruction: The decoder uses the compressed 2x2 latent representation to generate an 8x8 output. The objective is for this output to be as close as possible to the original image. It interprets the abstract features encoded by the latent space to recreate the visual elements of the digit '4'.

Implications of Autoencoder-based Reduction

The latent space values generated by the encoder are a dense representation of the input image. They are crucial for the decoder but are also valuable in various other applications:

• Data Compression: Reducing the dimensionality of data significantly reduces storage requirements and speeds up processing, which is essential for large datasets.
• Denoising: Autoencoders can be trained to ignore "noise" in the input data, thereby learning to recover the original undistorted data in the output.
• Anomaly Detection: By learning the normal patterns of data, autoencoders can detect outliers or anomalies when they fail to reconstruct these patterns accurately.
• Feature Extraction and Learning: The process helps in understanding the underlying patterns in the data, which can be useful in more complex unsupervised or supervised learning tasks.

Conclusion

Autoencoders offer a robust framework for understanding and manipulating large datasets in a way that is computationally efficient and insightful. By transforming an 8x8 image of the digit '4' into a 2x2 latent representation, we see a practical example of how complex information can be efficiently summarized through learned features. This capability is not just a technical curiosity but a powerful tool in the modern data scientist's toolkit, facilitating deeper insights and more effective data processing.

Full Code:

import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras import layers, models

# Create  8x8 image of the digit 4
image_4 = np.array([
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 1, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 0, 0, 0, 0],
    [1, 0, 0, 1, 0, 0, 0, 0],
    [1, 1, 1, 1, 1, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0]
])

# Display the original 8x8 image
plt.imshow(image_4, cmap='gray')
plt.title("Original 8x8 Image of Digit '4'")
plt.show()

# Define the autoencoder model
def build_autoencoder():
    input_img = layers.Input(shape=(8, 8, 1))
    # Encoder
    x = layers.Flatten()(input_img)
    x = layers.Dense(16, activation='relu')(x)
    encoded = layers.Dense(4, activation='relu')(x)  # 2x2 encoded representation

    # Decoder
    x = layers.Dense(16, activation='relu')(encoded)
    x = layers.Dense(64, activation='sigmoid')(x)
    decoded = layers.Reshape((8, 8, 1))(x)

    autoencoder = models.Model(input_img, decoded)
    autoencoder.compile(optimizer='adam', loss='mse')
    return autoencoder

# Instantiate and train the autoencoder
autoencoder = build_autoencoder()
autoencoder.fit(image_4.reshape((1, 8, 8, 1)), image_4.reshape((1, 8, 8, 1)), epochs=1000, verbose=0)

# Use autoencoder to reconstruct the image
decoded_img = autoencoder.predict(image_4.reshape((1, 8, 8, 1)))

# Plot the reconstructed image
plt.imshow(decoded_img.reshape(8, 8), cmap='gray')
plt.title("Reconstructed 8x8 Image of Digit '4'")
plt.show()
        

import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras import layers, models

# Create a more accurate 8x8 image of the digit 4
image_4 = np.array([
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 1, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 0, 0, 0, 0],
    [1, 0, 0, 1, 0, 0, 0, 0],
    [1, 1, 1, 1, 1, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 0]
])

# Define the autoencoder model
def build_autoencoder():
    input_img = layers.Input(shape=(8, 8, 1))
    # Encoder
    x = layers.Flatten()(input_img)
    x = layers.Dense(16, activation='relu')(x)
    encoded = layers.Dense(4, activation='relu')(x)  # 2x2 encoded representation

    # Decoder
    x = layers.Dense(16, activation='relu')(encoded)
    x = layers.Dense(64, activation='sigmoid')(x)
    decoded = layers.Reshape((8, 8, 1))(x)

    autoencoder = models.Model(input_img, decoded)
    autoencoder.compile(optimizer='adam', loss='mse')
    
    encoder = models.Model(input_img, encoded)
    
    return autoencoder, encoder

# Instantiate and train the autoencoder
autoencoder, encoder = build_autoencoder()
autoencoder.fit(image_4.reshape((1, 8, 8, 1)), image_4.reshape((1, 8, 8, 1)), epochs=1000, verbose=0)

# Use encoder to get the latent space representation
latent_representation = encoder.predict(image_4.reshape((1, 8, 8, 1)))

# Plot the latent space representation
plt.imshow(latent_representation.reshape(2, 2), cmap='gray')
plt.title("Latent Space Representation (2x2) of Digit '4'")
plt.colorbar()
plt.show()