Mastering XGBoost: From Basics to Advanced Techniques with a Complete Use Case

In today's world, data is everywhere, and Machine Learning (ML) has become an essential tool to make sense of it. One of the most popular and powerful ML algorithms is XGBoost, short for eXtreme Gradient Boosting. It is an efficient, flexible, and scalable implementation of gradient boosting, designed to solve a wide range of problems. In this comprehensive article, we will explore the basics of XGBoost, dive into advanced techniques, and provide a complete use case with code to help you master XGBoost.

Table of Contents

1. What is XGBoost?
2. Installing XGBoost
3. Preparing the Data
4. Basic XGBoost Model
5. Tuning Hyperparameters
6. Advanced Techniques
7. What kind of problems can XGBoost solve?
8. Complete Use Case: Predicting House Prices
9. Conclusion

1. What is XGBoost?

XGBoost is an open-source software library that provides a gradient boosting framework for various programming languages, including Python, R, and Java. It is built on the principles of Gradient Boosting Machines (GBM) and aims to be extremely efficient, flexible, and portable. XGBoost has gained immense popularity in the Machine Learning community due to its excellent performance on various tasks, especially in structured data and tabular datasets.

2. Installing XGBoost

To get started, you'll need to install the XGBoost library. For Python, you can use pip to install XGBoost:

pip install xgboost        

3. Preparing the Data

In order to use XGBoost effectively, it's essential to preprocess the data. Here are the general steps for data preparation:

1. Load the dataset.
2. Clean the data: Remove missing values, outliers, and duplicate rows.
3. Encode categorical variables: Convert categorical variables into numeric form using techniques like one-hot encoding or label encoding.
4. Normalize/standardize the data: Scale the numeric features to be on the same scale.
5. Split the data: Divide the dataset into training and testing sets.

4. Basic XGBoost Model

Once the data is prepared, you can create a basic XGBoost model using the following steps:

1. Import the necessary libraries.
2. Create an instance of the XGBRegressor or XGBClassifier class, depending on your problem.
3. Fit the model on the training data using the fit() method.
4. Make predictions using the predict() method.
5. Evaluate the model's performance using appropriate metrics.

import xgboost as xgb
from sklearn.metrics import mean_squared_error

# Create an instance of XGBRegressor
xgb_model = xgb.XGBRegressor()

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)        

5. Tuning Hyperparameters

XGBoost provides many hyperparameters that can be tuned to improve the model's performance. Some important hyperparameters are:

1. n_estimators: The number of boosting rounds.
2. max_depth: The maximum depth of each tree.
3. learning_rate: The step size used for weight updates.
4. subsample: The fraction of samples to be used for each boosting round.
5. colsample_bytree: The fraction of features to be used for each tree.

To find the best combination of hyperparameters, you can use techniques like grid search, random search, or Bayesian optimization. Below is an example of tuning hyperparameters using grid search with the help of GridSearchCV from scikit-learn:

from sklearn.model_selection import GridSearchCV

# Define the hyperparameter grid
param_grid = {
? ? 'n_estimators': [100, 200, 300],
? ? 'max_depth': [3, 4, 5],
? ? 'learning_rate': [0.01, 0.1, 0.2],
? ? 'subsample': [0.5, 0.8, 1],
? ? 'colsample_bytree': [0.5, 0.8, 1]
}

# Create an instance of XGBRegressor
xgb_model = xgb.XGBRegressor()

# Create the grid search object
grid_search = GridSearchCV(estimator=xgb_model, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error')

# Fit the grid search object to the data
grid_search.fit(X_train, y_train)

# Get the best combination of hyperparameters
best_params = grid_search.best_params_
print("Best hyperparameters:", best_params)

# Train the model with the best hyperparameters
best_xgb_model = xgb.XGBRegressor(**best_params)
best_xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = best_xgb_model.predict(X_test)

# Evaluate the model's performance
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)        

6. Advanced Techniques

To further improve the performance of your XGBoost model, you can use advanced techniques such as:

1. Early Stopping: Stop training when the validation error doesn't improve for a certain number of rounds.

# Split the training data into train and validation sets
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

# Fit the model with early stopping
xgb_model = xgb.XGBRegressor()
xgb_model.fit(X_train, y_train, early_stopping_rounds=10, eval_set=[(X_val, y_val)])        

2. Custom Loss Function: Define your own objective function based on your problem's requirements.

# Define custom objective function (example: Huber Loss)
def huber_approx_obj(y_true, y_pred):
? ? d = y_pred - y_true
? ? scale = 1 + (d / 2)
? ? squared_loss = np.square(d) / 2
? ? linear_loss = np.abs(d) - 0.5
? ? return np.where(np.abs(d) < 1, squared_loss, linear_loss) * scale

# Fit the model with custom objective function
xgb_model = xgb.XGBRegressor(obj=huber_approx_obj)
xgb_model.fit(X_train, y_train)        

3. Feature Importance: Identify and select the most important features for your model.

# Fit the model
xgb_model = xgb.XGBRegressor()
xgb_model.fit(X_train, y_train)

# Get feature importances
feature_importances = xgb_model.feature_importances_

# Display feature importances
for i, importance in enumerate(feature_importances):
? ? print(f"Feature {i}: {importance}")

# Select the most important features (example: top 5)
top_n = 5
most_important_indices = np.argsort(feature_importances)[-top_n:]
X_train_selected = X_train[:, most_important_indices]
X_test_selected = X_test[:, most_important_indices]        

4. Regularization: Control the complexity of the model using L1 and L2 regularization.

# Fit the model with L1 and L2 regularization (alpha and lambda)
xgb_model = xgb.XGBRegressor(alpha=1, lambda=1)
xgb_model.fit(X_train, y_train)        

5. Stacking: Combine XGBoost with other models to create an ensemble for better performance.

from sklearn.ensemble import RandomForestRegressor, StackingRegressor
from sklearn.linear_model import Ridge

# Create estimators for stacking
estimators = [
? ? ('xgb', xgb.XGBRegressor()),
? ? ('rf', RandomForestRegressor())
]

# Create the stacking regressor
stacking_regressor = StackingRegressor(estimators=estimators, final_estimator=Ridge())

# Fit the stacking regressor
stacking_regressor.fit(X_train, y_train)

# Make predictions on the test data
y_pred = stacking_regressor.predict(X_test)

# Evaluate the model's performance
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)        

These examples should help you understand how to implement each advanced technique using XGBoost. Note that these techniques can be combined and further refined to achieve the best performance for your specific problem.

7. What kind of problems can XGBoost solve?

XGBoost is a versatile machine learning algorithm that can be used to solve a wide range of problems. Some of the most common types of problems that XGBoost can handle effectively are:

1. Regression: XGBoost can be used to predict continuous target variables. For example, it can be used to predict house prices, sales forecasting, or stock price prediction.

Here's an example of using XGBoost for a regression task, predicting house prices using the Boston Housing dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import xgboost as xgb

# Load the dataset
boston = load_boston()
X = boston.data
y = boston.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an instance of XGBRegressor
xgb_model = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Mean Squared Error:", mse)
print("R^2 Score:", r2)        

This code demonstrates how to use XGBoost for regression tasks. The dataset is loaded, split into training and testing sets, and an XGBRegressor instance is created with specific hyperparameters. The model is then fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using mean squared error and R^2 score.

2. Binary classification: XGBoost can be used to classify instances into two classes, such as spam or not spam, fraud or not fraud, or positive or negative sentiment.

Here's an example of using XGBoost for a binary classification task, classifying whether a tumor is malignant or benign using the Breast Cancer Wisconsin dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import xgboost as xgb

# Load the dataset
breast_cancer = load_breast_cancer()
X = breast_cancer.data
y = breast_cancer.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an instance of XGBClassifier
xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
accuracy = accuracy_score(y_test, y_pred)
confusion = confusion_matrix(y_test, y_pred)
classification_rep = classification_report(y_test, y_pred)

print("Accuracy:", accuracy)
print("Confusion Matrix:\n", confusion)
print("Classification Report:\n", classification_rep)        

This code demonstrates how to use XGBoost for binary classification tasks. The dataset is loaded, split into training and testing sets, and an XGBClassifier instance is created with specific hyperparameters. The model is then fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using accuracy, confusion matrix, and classification report.

3. Multiclass classification: XGBoost can be used to classify instances into multiple classes, such as image classification, text categorization, or medical diagnosis.

Here's an example of using XGBoost for a multiclass classification task, classifying the species of Iris flowers using the Iris dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import xgboost as xgb

# Load the dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an instance of XGBClassifier
xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
accuracy = accuracy_score(y_test, y_pred)
confusion = confusion_matrix(y_test, y_pred)
classification_rep = classification_report(y_test, y_pred)

print("Accuracy:", accuracy)
print("Confusion Matrix:\n", confusion)
print("Classification Report:\n", classification_rep)        

This code demonstrates how to use XGBoost for multiclass classification tasks. The dataset is loaded, split into training and testing sets, and an XGBClassifier instance is created with specific hyperparameters. The model is then fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using accuracy, confusion matrix, and classification report.

4. Ranking: XGBoost can be used for learning-to-rank tasks, where the goal is to rank items according to their relevance or importance. This is particularly useful in search engines, recommendation systems, and information retrieval applications.

Here's an example of using XGBoost for a learning-to-rank task, ranking a set of documents based on their relevance to a query. We will use the 'MQ2008' dataset from the LETOR 4.0 collection for this example.

import numpy as np
import xgboost as xgb
from sklearn.datasets import load_svmlight_file
from sklearn.metrics import ndcg_score

# Load the dataset
train_data = load_svmlight_file('mq2008.train')
X_train, y_train, query_train = train_data[0], train_data[1], train_data[2]

test_data = load_svmlight_file('mq2008.test')
X_test, y_test, query_test = test_data[0], test_data[1], test_data[2]

# Convert query ids to group sizes
def query_ids_to_groups(query_ids):
? ? _, group_counts = np.unique(query_ids, return_counts=True)
? ? return group_counts

train_groups = query_ids_to_groups(query_train)
test_groups = query_ids_to_groups(query_test)

# Create an instance of XGBRanker
xgb_ranker = xgb.XGBRanker(objective='rank:pairwise', n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_ranker.fit(X_train, y_train, group=train_groups)

# Make predictions on the test data
y_pred = xgb_ranker.predict(X_test)

# Evaluate the model's performance
ndcg = ndcg_score(test_groups, y_test, y_pred)
print("Normalized Discounted Cumulative Gain (NDCG):", ndcg)        

This code demonstrates how to use XGBoost for learning-to-rank tasks. The dataset is loaded, split into training and testing sets, and an XGBRanker instance is created with specific hyperparameters. The model is then fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using the Normalized Discounted Cumulative Gain (NDCG) metric.

Note: You need to download the MQ2008 dataset from the LETOR 4.0 collection (https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/) and place the 'mq2008.train' and 'mq2008.test' files in your working directory for this code to work.

5. Feature selection: XGBoost can help identify the most important features in a dataset, enabling you to select a subset of features for improved model performance and reduced training time.

Here's an example of using XGBoost for feature selection, identifying the most important features in the Breast Cancer Wisconsin dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import xgboost as xgb

# Load the dataset
breast_cancer = load_breast_cancer()
X = breast_cancer.data
y = breast_cancer.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an instance of XGBClassifier
xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Get feature importances
feature_importances = xgb_model.feature_importances_

# Display feature importances
for i, importance in enumerate(feature_importances):
? ? print(f"Feature {i}: {importance}")

# Select the most important features (example: top 5)
top_n = 5
most_important_indices = np.argsort(feature_importances)[-top_n:]
X_train_selected = X_train[:, most_important_indices]
X_test_selected = X_test[:, most_important_indices]        

This code demonstrates how to use XGBoost for feature selection. The dataset is loaded, split into training and testing sets, and an XGBClassifier instance is created with specific hyperparameters. The model is then fitted on the training data, feature importances are extracted, and the most important features are selected for further analysis or modeling.

6. Imbalanced datasets: XGBoost can handle imbalanced datasets effectively by using the scale_pos_weight parameter, which helps balance the positive and negative classes in binary classification problems.

Here's an example of using XGBoost to handle imbalanced datasets in a binary classification task, using a synthetic dataset generated by the make_classification function from scikit-learn:

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import xgboost as xgb

# Generate a synthetic imbalanced dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=2,
? ? ? ? ? ? ? ? ? ? ? ? ? ?n_redundant=10, n_clusters_per_class=1,
? ? ? ? ? ? ? ? ? ? ? ? ? ?weights=[0.99], flip_y=0, random_state=42)

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Calculate the scale_pos_weight
positive_class = np.sum(y_train == 1)
negative_class = np.sum(y_train == 0)
scale_pos_weight = negative_class / positive_class

# Create an instance of XGBClassifier with scale_pos_weight
xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=3, learning_rate=0.1, scale_pos_weight=scale_pos_weight)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
accuracy = accuracy_score(y_test, y_pred)
confusion = confusion_matrix(y_test, y_pred)
classification_rep = classification_report(y_test, y_pred)

print("Accuracy:", accuracy)
print("Confusion Matrix:\n", confusion)
print("Classification Report:\n", classification_rep)        

This code demonstrates how to use XGBoost with the scale_pos_weight parameter to handle imbalanced datasets. The dataset is generated, split into training and testing sets, and an XGBClassifier instance is created with the scale_pos_weight parameter set to the ratio of negative to positive class instances. The model is then fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using accuracy, confusion matrix, and classification report.

7. Ensemble learning: XGBoost can be combined with other machine learning algorithms to create ensembles, resulting in improved model performance and generalization

Here's an example of using XGBoost along with other machine learning algorithms in an ensemble for a binary classification task, classifying whether a tumor is malignant or benign using the Breast Cancer Wisconsin dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
import xgboost as xgb

# Load the dataset
breast_cancer = load_breast_cancer()
X = breast_cancer.data
y = breast_cancer.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create instances of the classifiers
xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=3, learning_rate=0.1)
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
logistic_model = LogisticRegression(solver='lbfgs', max_iter=1000, random_state=42)

# Create an ensemble using VotingClassifier
ensemble_model = VotingClassifier(estimators=[
? ? ('xgb', xgb_model),
? ? ('rf', rf_model),
? ? ('logistic', logistic_model)],
? ? voting='soft')

# Fit the ensemble model on the training data
ensemble_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = ensemble_model.predict(X_test)

# Evaluate the ensemble model's performance
accuracy = accuracy_score(y_test, y_pred)
confusion = confusion_matrix(y_test, y_pred)
classification_rep = classification_report(y_test, y_pred)

print("Accuracy:", accuracy)
print("Confusion Matrix:\n", confusion)
print("Classification Report:\n", classification_rep)        

This code demonstrates how to use XGBoost in an ensemble with other machine learning algorithms. The dataset is loaded, split into training and testing sets, and classifier instances are created for XGBoost, RandomForest, and Logistic Regression. These classifiers are then combined using scikit-learn's VotingClassifier, which creates an ensemble by training each classifier separately and combining their predictions. The ensemble model is fitted on the training data, predictions are made on the test data, and the model's performance is evaluated using accuracy, confusion matrix, and classification report.

8. Time series forecasting: Although XGBoost is not specifically designed for time series problems, it can still be used for time series forecasting with appropriate feature engineering, such as lag features, rolling window statistics, and seasonal decomposition.

Time series forecasting: Although XGBoost is not specifically designed for time series problems, it can still be used for time series forecasting with appropriate feature engineering, such as lag features, rolling window statistics, and seasonal decomposition.

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Load the dataset
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv"
data = pd.read_csv(url)
data['Month'] = pd.to_datetime(data['Month'])

# Create lag features
def create_lag_features(df, n_lags):
? ? for i in range(1, n_lags + 1):
? ? ? ? df[f'lag_{i}'] = df['Passengers'].shift(i)
? ? return df

n_lags = 3
data = create_lag_features(data, n_lags)

# Drop rows with NaN values
data = data.dropna()

# Split the dataset into training and testing sets
train_data = data[data['Month'] < '1958-01-01']
test_data = data[data['Month'] >= '1958-01-01']

X_train = train_data.drop(['Month', 'Passengers'], axis=1)
y_train = train_data['Passengers']

X_test = test_data.drop(['Month', 'Passengers'], axis=1)
y_test = test_data['Passengers']

# Create an instance of XGBRegressor
xgb_model = xgb.XGBRegressor(n_estimators=100, max_depth=3, learning_rate=0.1)

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print("RMSE:", rmse)

# Plot the actual vs. predicted values
plt.plot(test_data['Month'], y_test, label="Actual")
plt.plot(test_data['Month'], y_pred, label="Predicted")
plt.xlabel("Month")
plt.ylabel("Passengers")
plt.legend()
plt.show()        

This code demonstrates how to use XGBoost for time series forecasting with appropriate feature engineering. The Airline Passengers dataset is loaded, and lag features are created using a helper function. The dataset is then split into training and testing sets based on a specific date. An XGBRegressor instance is created, fitted on the training data, and used to make predictions on the test data. The model's performance is evaluated using the root mean squared error (RMSE) metric, and the actual vs. predicted values are plotted.

In summary, XGBoost is a powerful and flexible algorithm that can be used to tackle a variety of machine learning problems across different domains. Its efficiency, scalability, and ability to handle structured and tabular data make it an excellent choice for many real-world applications.

8. Complete Use Case: Predicting House Prices

In this use case, we will use the Boston Housing dataset to demonstrate the process of creating and tuning an XGBoost model for predicting house prices. The dataset is available in the scikit-learn library.

# Import necessary libraries
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Load the dataset
boston = load_boston()
X = boston.data
y = boston.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an instance of XGBRegressor
xgb_model = xgb.XGBRegressor()

# Fit the model on the training data
xgb_model.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb_model.predict(X_test)

# Evaluate the model's performance
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)        

Follow the hyperparameter tuning steps mentioned in section 5 to further improve the model's performance.

9. Conclusion

XGBoost is a powerful and versatile machine learning algorithm that can be used for a wide range of problems. By understanding its basics, tuning hyperparameters, and employing advanced techniques, you can harness the full potential of XGBoost to solve complex real-world challenges. In this comprehensive guide, we covered the installation process, data preparation, basic model building, hyperparameter tuning, advanced techniques, and provided a complete use case of predicting house prices using the Boston Housing dataset. With this knowledge, you can now confidently apply XGBoost to your own projects and continue exploring its capabilities to further enhance your machine learning expertise.

As you continue your journey with XGBoost, consider delving deeper into the following topics:

1. Cross-validation: Use techniques like k-fold cross-validation to obtain more reliable performance estimates.
2. Handling imbalanced datasets: Learn how to use XGBoost's scale_pos_weight parameter to handle imbalanced classification problems effectively.
3. Feature engineering: Improve the model's performance by creating new features based on domain knowledge.
4. Integrating XGBoost with other libraries: Utilize XGBoost with other popular ML libraries like scikit-learn, TensorFlow, and PyTorch.

By mastering XGBoost and its advanced techniques, you'll be well-equipped to tackle various machine learning problems, enhance your data-driven decision-making capabilities, and ultimately make a significant impact in your domain.

References:

1. XGBoost Documentation: https://xgboost.readthedocs.io/en/latest/
2. Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). https://arxiv.org/abs/1603.02754
3. scikit-learn Documentation: https://scikit-learn.org/stable/index.html
4. Breast Cancer Wisconsin (Diagnostic) Data Set: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29
5. LETOR 4.0 collection: https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/
6. Airline Passengers dataset: https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv
7. Brownlee, J. (2018). How to Develop an Ensemble Learning System for Time Series Forecasting. Machine Learning Mastery. https://machinelearningmastery.com/ensemble-machine-learning-algorithms-python-scikit-learn/
8. Brownlee, J. (2018). How to Develop a Time Series Forecasting Model with XGBoost. Machine Learning Mastery. https://machinelearningmastery.com/xgboost-for-time-series-forecasting/

#ViewsMyOwn #MachineLearning #ML #XGBoost