What is Vectorization in GenAI? Explained!!

Vectorization is a way of organizing data that allows computers to perform operations on multiple pieces of data at once, making computations faster and more efficient.

Imagine you have a list of 10 numbers, and you want to multiply each number by 2. Instead of doing it one by one, you can use vectorization to perform the operation on the entire list simultaneously. This is like having a conveyor belt where all the numbers are processed at once, rather than individually.

In a real-life example for a class 10th student, think of vectorization as solving multiple math problems at once. For instance, if you have a set of equations with multiple variables, instead of solving each equation separately, you can use vectorization to solve them all together, saving time and effort.

Explain with example.

Let's say you have two lists of numbers:

List 1: [1, 2, 3, 4, 5]

List 2: [6, 7, 8, 9, 10]

And you want to add each corresponding pair of numbers together. Without vectorization, you'd have to do it like this:

Result: [1+6, 2+7, 3+8, 4+9, 5+10] = [7, 9, 11, 13, 15]

But with vectorization, you can do it in one step by treating these lists as vectors:

Result: [1, 2, 3, 4, 5] + [6, 7, 8, 9, 10] = [1+6, 2+7, 3+8, 4+9, 5+10] = [7, 9, 11, 13, 15]

So, instead of doing each addition separately, you perform the operation on the entire list at once, which is faster and more efficient. That's the power of vectorization!

Write Down Python Code.

Here's the Python code for the example using vectorization:

import numpy as np        

# Define the lists        

list1 = [1, 2, 3, 4, 5]        

list2 = [6, 7, 8, 9, 10]        

# Convert the lists to numpy arrays        

array1 = np.array(list1)        

array2 = np.array(list2)        

# Perform vectorized addition        

result = array1 + array2        

# Print the result        

print(result)        

This code uses the NumPy library to perform vectorized addition on the arrays array1 and array2, which are created from the lists list1 and list2. Finally, it prints the result [ 7 9 11 13 15].

How vectorization code work internally for above example?

Internally, when you perform vectorized operations using libraries like NumPy, the code leverages highly optimized routines written in lower-level languages like C or Fortran.

Here's a simplified explanation of how it works:

1. Conversion to Arrays: The lists are converted into NumPy arrays (`array1` and array2).

2. Memory Allocation: NumPy allocates memory for the result array (`result`) which will store the output of the addition operation.

3. Vectorized Operation: NumPy's underlying C or Fortran code performs the addition operation element-wise. This means it adds corresponding elements from array1 and array2 together directly without the need for explicit loops.

4. Efficient Execution: The operation is executed efficiently at the machine level, taking advantage of hardware features like SIMD (Single Instruction, Multiple Data) instructions on modern CPUs, which allow for parallel processing of data.

5. Return Result: Once the operation is completed, the result is returned as a NumPy array.

Overall, vectorization allows for faster and more efficient computation by taking advantage of optimized low-level routines and hardware capabilities, making it a powerful tool for numerical computing tasks in Python.

How it is different from normal addition internally?

In normal addition (without vectorization), you typically use loops to iterate through each element of the lists and perform the addition operation one by one. Here's how it works internally:

1. Loop Initialization: You initialize variables to store the result and control the iteration through the lists.

2. Iterative Addition: You iterate through the lists using a loop (e.g., for loop) and perform the addition operation on each pair of elements.

3. Memory Allocation: You may allocate memory for the result array or list if needed.

4. Element-wise Addition: At each iteration, you add the corresponding elements from the two lists together.

5. Update Result: You update the result array or list with the computed values.

6. Loop Termination: The loop continues until all elements have been processed.

7. Return Result: Once the loop finishes, you return the result array or list.

Internally, this process involves repeated instructions to load, add, and store data, which can be less efficient compared to vectorized operations, especially for large datasets. Vectorization, on the other hand, performs the addition operation on entire arrays at once, leveraging optimized routines and hardware capabilities for faster execution.

How it is useful in GenAI coding?

In GenAI coding, which typically involves working with large datasets and complex mathematical operations, vectorization plays a crucial role in improving the efficiency and speed of computations. Let's consider an example where we want to calculate the cosine similarity between two sets of vectors using vectorization.

Without vectorization, you might write code like this:

import numpy as np        

# Generate random vectors        

vector1 = np.random.rand(1000, 100)  # 1000 vectors of dimension 100        

vector2 = np.random.rand(1000, 100)        

# Compute cosine similarity without vectorization        

similarities = []        

for i in range(len(vector1)):        

    similarity = np.dot(vector1[i], vector2[i]) / (np.linalg.norm(vector1[i]) * np.linalg.norm(vector2[i]))        

    similarities.append(similarity)        

This code iterates through each pair of vectors and calculates the cosine similarity one by one.

Now, let's see how vectorization can improve this:

import numpy as np        

# Generate random vectors        

vector1 = np.random.rand(1000, 100)  # 1000 vectors of dimension 100        

vector2 = np.random.rand(1000, 100)        

# Compute cosine similarity with vectorization        

similarities = np.sum(vector1  vector2, axis=1) / (np.linalg.norm(vector1, axis=1)  np.linalg.norm(vector2, axis=1))        

In this vectorized code, we perform element-wise multiplication of the two arrays vector1 and vector2 directly, and then sum along the specified axis (`axis=1` for summing along rows). We also compute the norms of the vectors along the same axis. This approach eliminates the need for explicit loops and leverages NumPy's optimized routines for faster computation.

By using vectorization, we can significantly improve the performance of our code, especially for large datasets, as it allows computations to be performed in parallel and takes advantage of optimized low-level routines. This makes GenAI coding more efficient and scalable.