Model Optimization in Machine Learning: Random vs. Grid?Search

What is a Machine Learning?model?

A machine learning model is a mathematical formula with several parameters that need to be learned from data. This process is known as training. In other words, training with data is nothing more than adjusting parameters.

Machine Learning algorithms rely on another type of parameter, called Hyperparameter. Hyperparameters are defined before the training process, therefore, they cannot be learned.

Now let’s reflect on some examples of Hyperparameters of the classification algorithm called Random Forest:

• n_estimators: Specifies the total number of Random Forest trees.
• max_depth?: Specifies the maximum depth of each tree.
• min_samples_Split: specifies the minimum number of samples required to split an internal leaf node.
• min_samples_leaf: specifies the minimum number of samples required to be in a leaf node.

The problem:

Given that hyperparameter values must be specified before training, there are several situations in which it is not possible to adequately determine the best values for each hyperparameter.

Search Space

The image bellow describing the search space of an optimization algorithm with local minima and maxima, as well as global minima and maxima, would be a visual representation to understand how these algorithms navigate a set of possible solutions to find the best (optimal) result.?.

Faced with this problem, several researchers have presented intelligent optimization algorithms for optimizing hyperparameters (and parameters).

Grid Search

Grid Search is a technique used to find the best combination of parameters for a given model in machine learning. This technique involves defining a “grid” of parameters, which are all the possible combinations of different parameter values that you want to test for your model.

Advantages of Grid Search

• Exhaustiveness: Grid Search is exhaustive in the search for the best hyperparameters, as it tests all possible combinations. This ensures that within the defined set of parameter values, the best combination will be found.
• Simplicity: It is an easy method to understand and implement. The process is straightforward, without the need for complex adjustments or advanced knowledge about the model’s behavior.
• Reproducibility: Because Grid Search tests fixed combinations of parameters, the results are reproducible. This means that other researchers or developers can recreate the same experiment and get the same results.

Disadvantages of Grid Search

• Computational Cost: The main drawback of Grid Search is its high computational cost. Testing all possible combinations of parameters can be very time-consuming, especially for large data sets and complex models.
• Grid Search: Grid Search may miss the “best” match if it is not included in the defined grid. The quality of the results depends on how well the grid is defined.
• Decreased Efficiency in High-Dimensional Spaces: As the number of hyperparameters increases, Grid Search becomes less efficient due to the “curse of dimensionality”. This means that the number of combinations grows exponentially, making the process impractical for many hyperparameters.

Random Search

The Random Search algorithm, is a hyperparameter optimization method used in machine learning, which randomly selects combinations of parameters from a distribution specified for each hyperparameter. Unlike Grid Search, which tests all possible combinations of a predefined grid of parameter values, Random Search explores the hyperparameter space randomly. This can be more efficient, especially in high-dimensional spaces, as it allows for a more comprehensive search that is less restricted to a specific grid, and often finds a good solution with far fewer iterations and computation time compared to Grid Search.

Advantages of Random?Search

1. Efficiency in High-Dimensional Spaces: Random Search is often more efficient for high-dimensional hyperparameter spaces because it doesn’t need to explore all combinations within a fixed grid. This means it can find good solutions with fewer iterations.
2. Flexibility in Parameter Distribution: It allows hyperparameters to be sampled from non-uniform distributions, which is useful when some hyperparameters have a greater impact on model performance than others.
3. Potential to Discover Better Solutions: As Random Search explores the hyperparameter space randomly, there’s a chance of discovering parameter combinations that might be missed by Grid Search, especially if the ‘best’ set of hyperparameters lies outside the predefined grid in Grid Search.

Disadvantages of Random?Search

1. No Guarantee of the Best Solution: Given that Random Search relies on randomness, there is no assurance that it will find the optimal combination of hyperparameters, especially if the number of iterations is limited.
2. Dependence on Number of Iterations and Luck: The effectiveness of Random Search can heavily depend on the number of iterations and luck, as some random combinations might be significantly better than others.
3. Inefficiency in Smaller Spaces: For small and well-defined parameter spaces, Grid Search might be more efficient, as Random Search could waste time exploring irrelevant parameter combinations.

Grid Search x Random Search

Let’s to Benchmark Results

First, we need to import tle libraries:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np
import time        

For this benchmarking experiment, we will utilize the Iris Dataset

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)        

Model Definition

model = SVC()        

Grid Search Application

# Parameters for Grid Search
param_grid = {'C': [0.1, 1, 10, 100], 'gamma': [1, 0.1, 0.01, 0.001], 'kernel': ['rbf', 'poly', 'sigmoid']}

# Start the timer for Grid Search
start_time = time.time()

# Execute Grid Search
grid_search = GridSearchCV(model, param_grid, refit=True, verbose=0)
grid_search.fit(X_train, y_train)

# Calculate the execution time of Grid Search
grid_search_time = time.time() - start_time

print("Best score of Grid Search: ", grid_search.best_score_)
print("Processing time of Grid Search: {:.2f} seconds".format(grid_search_time))        

Best score of Grid Search: 0.971

Processing time of Grid Search: 0.96 seconds

Random Search Application

# Parameters for Random Search
param_distributions = {'C': np.logspace(-2, 2, 100), 'gamma': np.logspace(-3, 1, 100), 'kernel': ['rbf', 'poly', 'sigmoid']}

# Start the timer for Random Search
start_time = time.time()        

# Execute Random Search
random_search = RandomizedSearchCV(model, param_distributions, n_iter=50, refit=True, verbose=0, random_state=42)
random_search.fit(X_train, y_train)

# Calculate the execution time of Random Search
random_search_time = time.time() - start_time

print("Best score of Random Search: ", random_search.best_score_)
print("Processing time of Random Search: {:.2f} seconds".format(random_search_time))        

Best score of Random Search: 0.96

Processing time of Random Search: 0.55 seconds

Results and Discussion

The benchmarking experiment results demonstrate key differences between the Grid Search and Random Search algorithms in terms of performance and efficiency.

Firstly, the Grid Search algorithm achieved a slightly higher score of 0.97 compared to the Random Search’s 0.96. This difference, in percentage terms, is approximately 1.19%. While this indicates a marginally better performance by the Grid Search in accuracy, the difference is relatively small.

However, when considering processing time, the distinction becomes more significant. The Grid Search took 0.96 seconds, whereas the Random Search completed in just 0.55 seconds. This means that Random Search was approximately 42.7% faster than Grid Search. In scenarios where processing speed is crucial, such as in real-time applications, this difference in time efficiency can be highly impactful.

While both algorithms performed well, Random Search stands out as the more efficient option in this experiment. Its ability to achieve nearly similar accuracy to Grid Search but in significantly less processing time underlines its suitability for situations where quick decision-making is essential. This advantage can be particularly valuable in real-world applications where computational resources and time are critical factors.