Number System - Quantization of LLMs, Part-1
Large Language Models (LLMs) have significantly advanced in recent years, becoming increasingly user-friendly and versatile in various applications. Nevertheless, as the intelligence and complexity of LLMs have expanded, so too has the number of parameters, including weights and activations, which determine their ability to learn from and analyze data.
Hence, the larger an LLM, the more memory it requires.
As the size of an LLM increases, so does the memory it demands. This necessitates running LLMs on high-spec hardware with the required number of GPUs, limiting deployment options and the ease of adopting LLM-based solutions. Fortunately, machine learning researchers are working on a range of solutions to tackle the challenge of growing model sizes, with quantization being a prominent one.
Why do we need Quantization?
Before we deep delve into the concept of Quantization. Let us first try to understand why do we need it in the first place.
Quantization aims to address the following challenges:
Challenge-1: Most contemporary deep neural networks consist of millions or even billions of parameters, which poses a significant challenge.
Consider the following examples:
Ex 1. The smallest LLaMA-2 model consists of 7 billion parameters. Assuming each parameter is 32 bits, we would require 28GB of storage space just to store these parameters on the disk
Ex 2. The smallest LLaMA-3 model consists of 8 billion parameters. Assuming each parameter is 32 bits, we would require 32GB of storage space just to store these parameters on the disk.
Ex 3. The current state-of-the-art GPT-4 has in excess of 1 trillion parameters. Rumor's claim that it has 1.76 trillion parameters. Assuming each parameter is 32 bits, we would require 7.04TB of storage space just to store these parameters on the disk.
Challenge-2: Consequently, larger models pose a challenge as they cannot be effortlessly loaded on a standard PC or a smart phone. When utilizing a CPU for inference, it is necessary to load it into the RAM. Conversely, when using a GPU, it should be loaded into the GPU's memory.
Challenge-3: Similar to humans, computers have a slower processing speed when it comes to performing floating-point operations in comparison to integer operations. Consider the calculation of 4 × 8 and compare it to 1.17 × 2.389. Which one can be computed more quickly?
Answer - 4 x 8
How to tackle these challenges?
To address these challenges, quantization provides the solution. Quantizing large language models (LLMs) is a crucial method for reducing their size and memory usage, all while preserving their quality.
So what exactly is Quantization?
Quantization
Quantization, in an abstract sense, is the process of constraining an input from a continuous or otherwise large set of values to a discrete set.
But, how does this relate to LLMs?
To see the relation between abstract quantization mechanism and LLMs. Let us first try to understand the following fundamental concepts of Number System.
Numeric Data Types
Let's examine the representation of numbers in hardware at either the CPU or GPU level. Computers utilize a set number of bits to represent various types of data, such as numbers, characters, or pixel colors. The fixed number of bits is consistently employed.
How is Numeric Data Represented in Modern Computing Systems?
Human beings utilize the decimal (base 10) and duodecimal (base 12) number systems to perform counting and measurements, likely due to our possession of 10 fingers and two prominent toes.
Conversely, computers rely on the binary (base 2) number system, as they consist of binary digital components, known as transistors, which function in two distinct states - on and off. If the current passes through the transistor then the computer reads “1” and if the current is absent from the transistor then it read “0”.
Decimal Number system (Base 10)
Decimal number system has ten symbols: 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, called digits. It uses positional notation. That is, the least-significant digit (right-most digit) is of the order of 10^0 (units or ones), the second right-most digit is of the order of 10^1 (tens), the third right-most digit is of the order of 10^2 (hundreds), and so on, where ^ denotes exponent.
For example,
Binary Number System (Base 2)
The Binary Number System is a numerical system that utilizes two symbols, "0" and "1", to represent different numbers. It uses positional notation. The term "binary" is derived from the word "bi," meaning two. As a result, this numerical system is referred to as the Binary Number System. A binary digit is called a bit.
What is the decimal equivalent of binary number 10110?
There are generally various types of number systems and among them the four major ones are,
As discussed, Computers utilize a set quantity of bits to symbolize various types of data, such as numbers, characters, or pixel colors. A bit sequence consisting of n bits (also called n-bit string or n-bit storage location) has the capability to represent a maximum of 2^n unique entities.
For example, a 3-bit memory location can hold one of these eight binary patterns: 000, 001, 010, 011, 100, 101, 110, or 111.
Hence, it can represent at most 8 distinct entities.
You could use them to represent numbers 0 to 7, numbers 8881 to 8888, characters 'A' to 'H', or up to 8 kinds of fruits like apple, orange, banana; or up to 8 kinds of animals like lion, tiger, etc.
Typically, numbers are represented in groups of 8 bits (byte), 16 bits (short), 32 bits (int), or 64 bits (long).
1. Integer representation in CPU (or GPU)
Integers are whole numbers or fixed-point numbers with the radix point fixed after the least-significant bit. Computers use a fixed number of bits to represent an integer. The commonly-used bit-lengths for integers are 8-bit, 16-bit, 32-bit or 64-bit.
In addition to bit-lengths, there exist two distinct representation schemes for integers.
Three representation schemes had been proposed for signed integers:
As a programmer, it is your responsibility to determine the bit-length and representation scheme for the integers based on the specific requirements of your application. In the case of needing a counter to track a small quantity ranging from 0 to 200, you could opt for the 8-bit unsigned integer scheme since it does not involve negative numbers.
Let us try to understand these representations in detail.
1.1 Unsigned Integer (n-bit)
Unsigned integers have the ability to represent zero and positive integers, excluding negative integers. The interpretation of an unsigned integer's value is based on "the magnitude of its underlying binary pattern".
Example 1: Suppose that n=8 and the binary pattern is 01000001, the value of this unsigned integer is 65.
Example 2: Suppose that n=16 and the binary pattern is 0000000000000000, the value of this unsigned integer is 0.
1.2 Signed Integers
Signed integers can represent zero, positive integers, as well as negative integers. Three representation schemes are available for signed integers:
In each of the aforementioned three schemes, the sign bit, also known as the most-significant bit (msb), is utilized to indicate the sign of the integer. A value of 0 represents a positive integer, while a value of 1 represents a negative integer. Nevertheless, the interpretation of the integer's magnitude varies across the different schemes.
领英推荐
1. Sign-Magnitude Representation
In sign-magnitude representation:
Example 1: Suppose that n=8 and the binary representation is 01000001.?
Sign bit is 0 ? positive
Absolute value of remaining (7-bits) is 1000001 = 65.???Hence, the integer is +65.
Example 2: Suppose that n=8 and the binary representation is 00000000.
Sign bit is 0 ? positive.
Absolute value is 0000000 = 0???Hence, the integer is +0. Note the + sign here.
Example 3: Suppose that n=8 and the binary representation is 10000000.
Sign bit is 1 ? negative??
Absolute value is 0000000 = 0.??Hence, the integer is -0. Note the - sign here.
So from example-2 and example-3 we can infer that in sign magnitude representation, binary numbers 00000000 and 10000000 have same value.
Range of signed-magnitude n-bit integers:
The drawbacks of sign-magnitude representation are:
2. 1's Compliment Representation
In 1's complement representation:
Example 1: Suppose that n=8 and the binary representation 01000001.
Example 2: Suppose that n=8 and the binary representation 10000001.
Example 3: Suppose that n=8 and the binary representation 0 000 0000.
Example 4: Suppose that n=8 and the binary representation 1 111 1111.
The following figure illustrates the visual working of Example-3 and Example-4.
Range of 1's compliment representation for n-bit integer:
Let us visualize the range of 1's compliment representation for n=8.
Once more, the disadvantages are:
3. 2's Compliment Representation
In 2's complement representation:
Example 1: Suppose that n=8 and the binary representation 01000001.
Example 2: Suppose that n=8 and the binary representation 1 000 0001.
Example 3: Suppose that n=8 and the binary representation 00000000.
Example 4: Suppose that n=8 and the binary representation 11111111.
Let us visualize the range of 2's compliment representation for n=8.
Range of 2's compliment representation for n-bit integer:
Modern Computing System use 2's Complement Representation for Signed Integers
Computers use 2's complement in representing signed integers. This is because:
To be continued... in Part-2