Quantization in the context of deep learning and neural networks

What is Quantization?

Quantization in the context of deep learning and neural networks refers to the process of reducing the precision of the numbers used to represent model parameters (like weights and biases) and computations (such as activations). This technique is primarily used to reduce the model size and speed up inference while attempting to maintain accuracy.

Quantization is a process of reducing model size so that it can run on edge devices

Benefits of Quantization

1. Reduced Model Size: Decreases the storage requirements by reducing parameter precision, often cutting model size by up to 4x.
2. Increased Inference Speed: Enhances computational speed, particularly on hardware optimized for low-precision integers, improving real-time application responsiveness.
3. Lower Power Consumption: Consumes less power, crucial for extending battery life in mobile and wearable devices.
4. Improved Utilization of Hardware Accelerators: Leverages specialized hardware accelerators designed for efficient low-precision computation, boosting performance.
5. Accessibility: Makes advanced AI technologies more accessible on devices with limited computational resources, expanding AI deployment capabilities

Two Ways to Perform Quantization

1. Post-Training Quantization (PTQ): This method involves quantizing a model after it has been fully trained using high-precision (e.g., 32-bit floating-point) data. PTQ is simpler and faster to implement as it does not require retraining the model. It converts weights and activations to lower precision formats, typically 8-bit integers, which can significantly reduce the model size and improve computational efficiency. but here accuracy might get affected.

2. Quantization Aware Training (QAT): Unlike PTQ, QAT incorporates quantization directly into the training process. This means the training simulates lower precision arithmetic, allowing the model to adapt to the quantization-induced changes in distribution of its internal representations. This generally helps in maintaining higher accuracy as the model learns to mitigate the effects of reduced precision during its training phase.

Post-Training Quantization

Post-Training Quantization involves several steps:

- Calibration: This is used to determine the appropriate scaling factors for converting floating-point numbers into integers. It typically involves running a subset of the training or validation data through the model and observing the distribution of activations.

- Conversion: The floating-point weights and activations are converted to integers using the scaling factors determined in the calibration step.

- Optimization: Optional additional steps might be taken to optimize the quantized model, such as fine-tuning certain parameters or applying specific hardware accelerations.

Quantization Aware Training

Quantization Aware Training integrates quantization into the training loop:

- Quantization Simulation: All numerical operations (forward and backward passes) are simulated at lower precision. This involves modifying the model graph to include quantization nodes that mimic the effect of lower precision.

- Parameter Update: Parameters are updated in a way that considers the quantization effects, which typically involves using fake quantization nodes during the training to approximate the effects of real quantization.

- Fine-Tuning: The model might require re-training or fine-tuning with quantization considerations to regain or retain accuracy lost due to reduced numerical precision.

Coding Example

Here's a simple example using TensorFlow to perform post-training quantization:

import tensorflow as tf

# Load a pre-trained model
model = tf.keras.applications.MobileNetV2(weights='imagenet', input_shape=(224, 224, 3))

# Convert the model to TensorFlow Lite format with quantization
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
quantized_tflite_model = converter.convert()

# Save the quantized model
with open('quantized_model.tflite', 'wb') as f:
    f.write(quantized_tflite_model)        

For Quantization Aware Training, TensorFlow provides specific APIs through the tf.quantization module, allowing for a more involved setup that simulates low-precision computation during training.

Follow this link to access the GitHub repository containing the complete code for our Quantization tutorial. This includes detailed implementations for both post-training quantization and quantization-aware training.

https://github.com/maryamsoftdev/Quantization-in-Machine-Learning