Demystifying XGBoost with a Real-World Example

In the dynamic field of machine learning, one algorithm has risen to prominence for its outstanding performance and versatility: XGBoost (eXtreme Gradient Boosting). Revered for its speed and efficiency, XGBoost has carved a niche in both competitive machine learning and in practical applications across various industries. From predicting consumer behavior to aiding in medical diagnoses, XGBoost's applications are as diverse as they are impactful.

This article aims to shed light on the inner workings of XGBoost, not through abstract theory but via a practical, hands-on example. We will dive deep into the Breast Cancer Wisconsin (Diagnostic) dataset, a classic in the domain of binary classification problems. By walking through this example, readers will gain a tangible understanding of how XGBoost functions and why it is such a powerful tool in the machine-learning arsenal.

We will begin by introducing the necessary Python libraries and the dataset, followed by a step-by-step guide through the data preprocessing, model training, and evaluation stages. In addition, we will delve into the interpretation of the model's results, focusing on aspects like accuracy, confusion matrices, and feature importance.

Our objective is not just to familiarize you with XGBoost as a tool, but to provide you with the skills and understanding necessary to apply it to your datasets. Whether you are a student stepping into the world of data science, a seasoned professional looking to refine your toolkit, or just a curious mind eager to understand the mechanics behind one of today's leading machine-learning algorithms, this article is for you. Let's embark on this journey of discovery and learn how to harness the power of XGBoost in real-world scenarios.

Note: This article is part of the following article:

Step 1: Downloading the Dataset

We will be using the Kaggle dataset from https://www.kaggle.com/datasets/uciml/breast-cancer-wisconsin-data

Make sure to download it locally to the same directory of the Notebook or Google Colab, as explained in this article.

Step 2: Loading the Dataset

Assuming you've downloaded the dataset from Kaggle and it's saved as data.csv, we'll load it into a DataFrame:

import pandas as pd
df = pd.read_csv("data.csv")

df.head()        

Step 3: Understand the Dataset

Before we start processing the data and EDA, we need to understand the dataset. First read through the explanation on the site. Let's get the info as follows:

df.info()        

The Breast Cancer Wisconsin (Diagnostic) dataset from Kaggle is a widely used data set in machine learning, particularly for binary classification problems. It contains features computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. The data is used to predict whether the cancer is benign or malignant. Let's break down the key aspects of this dataset:

1. Dataset Overview:The dataset consists of 569 instances. There are 32 attributes (or features) in total, including the ID number and diagnosis.
2. Attributes:ID Number: A unique identifier for each patient. Diagnosis (Y): The diagnosis of breast tissues where 'M' stands for malignant and 'B' for benign. [This is the label (Y) Column]
3. Features:The dataset contains ten real-valued features computed for each cell nucleus: Radius (mean of distances from the center to points on the perimeter). Texture (standard deviation of gray-scale values). Perimeter.Area.Smoothness (local variation in radius lengths).Compactness (perimeter^2 / area - 1.0). Concavity (severity of concave portions of the contour).Concave points (number of concave portions of the contour).Symmetry.Fractal dimension ("coastline approximation" - 1).
4. Feature Columns:Each of these ten features has been summarized in three ways in the dataset: Mean (average of these features for each image). Standard Error (standard error for these features). Worst (worst or largest value of these features found in the same image).
5. Missing Values:The dataset, as provided, typically has no missing attribute values.
6. Use for Classification:The primary goal with this dataset is to classify a tumor as malignant ('M') or benign ('B') using the features provided. This is a binary classification problem, and the algorithms like logistic regression, decision trees, random forest, and notably XGBoost can be applied to solve it.
7. Applications:The dataset is crucial for developing and testing algorithms for cancer detection and is widely used in academic and educational settings.
8. Preprocessing:Users of this dataset might need to perform certain preprocessing steps such as normalization or standardization, especially since some machine learning algorithms are sensitive to the scale of data.

This dataset is not only a valuable resource for practicing data preprocessing, feature extraction, and classification models but also serves as a foundation for exploring more advanced machine learning techniques and concepts.

Step 4: Data Preprocessing

Before training the model, we need to preprocess the data:

• Handle missing values (if any). To check for columns with missing values, you can use

df.columns[df.isnull().any()].tolist()        

As you can see, only the 'Unnamed: 32' has missing values. We can safely drop this column.

• Convert categorical variables to numerical (if required).
• Separate the features and the target variable.

# Dropping an unnecessary column, if there's any
df.drop(['Unnamed: 32'], axis=1, inplace=True)

# Separating features and target
X = df.drop('diagnosis', axis=1)
y = df['diagnosis']

# Converting categorical variables to numerical, if there are any
y = y.map({'M':1, 'B':0})        

Step 5: Splitting the Dataset

Split the data into training and test sets:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)        

Step 6: XGBoost Model Training

Now, we'll train an XGBoost classifier:

!pip install xgboost        

import xgboost as xgb

model = xgb.XGBClassifier()
model.fit(X_train, y_train)
        

Step 7: Model Evaluation

Evaluate the model's performance:

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import matplotlib.pyplot as plt
import seaborn as sns

# Predictions
y_pred = model.predict(X_test)


# Accuracy
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)

# Confusion Matrix
conf_matrix = confusion_matrix(y_test, y_pred)
sns.heatmap(conf_matrix, annot=True, fmt='g')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

# Classification Report
print(classification_report(y_test, y_pred))
        

The results above show the evaluation of a XGBoost model. It includes accuracy, a confusion matrix, and a classification report, each of which provides information on the model's performance:

The Accuracy is 96%

1. Confusion Matrix:The confusion matrix is a table often used to describe the performance of a classification model on a set of test data for which the true values are known. In your matrix, there are two classes: 0 (likely representing the negative class or benign tumors) and 1 (likely representing the positive class or malignant tumors). The matrix shows the following counts: True Negative (TN): 69 instances were correctly predicted as class 0.False Positive (FP): 2 instances were incorrectly predicted as class 1 when they are actually class 0.False Negative (FN): 3 instances were incorrectly predicted as class 0 when they are actually class 1.True Positive (TP): 40 instances were correctly predicted as class 1.Ideally, you want the numbers in the diagonal (TP and TN) to be as high as possible, indicating correct predictions.
2. Classification Report:The classification report provides key metrics on the performance of the classifier: Precision (for each class) is the ratio TP / (TP + FP). For class 0, it is 0.96, meaning the model is 96% precise in predicting class 0. For class 1, it is 0.95. Recall (for each class) is the ratio TP / (TP + FN). For class 0, it is 0.97, and for class 1, it is 0.93. This means the model is slightly better at identifying all relevant instances of class 0 than class 1.F1-score is the harmonic mean of precision and recall, a balance between the two. For class 0, it is 0.97, and for class 1, it is 0.94, which is quite high for both classes, indicating a good balance between precision and recall. Support is the actual number of occurrences of each class in the provided dataset. For class 0, it is 71, and for class 1, it is 43.The report also provides averages for these metrics:Accuracy is the ratio of correctly predicted instances to the total instances, which is 0.9561, or about 95.61%.Macro average calculates metrics for each label and finds their unweighted mean. This does not take label imbalance into account.Weighted average calculates metrics for each label, and finds their average, weighted by the number of true instances for each label. This alters 'macro' to account for label imbalance.

Overall, these results suggest that the XGBoost model has performed quite well on this dataset, with high values for accuracy, precision, recall, and F1-score, which indicate a reliable classification model. However, the slight imbalance between the classes (more instances of class 0 than class 1) should be taken into account when considering these metrics, especially since the weighted average takes this into account and still shows high performance.

Step 8: Feature Importance

Plotting the feature importance:

xgb.plot_importance(model)
plt.rcParams['figure.figsize'] = [12, 9]
plt.show()
        

Feature importance scores help to understand which features have the most influence on the predictions of a model. Here’s what the chart illustrates:

1. Features:The y-axis lists the features used by the model. Each feature corresponds to a specific characteristic from the dataset. For instance, texture_worst and concave points_mean are likely to be some of the calculated metrics for the cell nuclei characteristics in the breast cancer dataset.
2. F Score:The x-axis shows the 'F score', which is a metric that quantifies the importance of each feature. The F score in the context of XGBoost is derived from the number of times a feature is used to split the data across all trees within the model.
3. Bar Length:Each bar's length represents the importance of that feature. The longer the bar, the more important the feature is considered by the model.
4. Interpretation:According to the chart, texture_worst is the most important feature, with the highest F score of around 32. This means that during the model building, texture_worst was the most useful feature for making splits in the decision trees.The feature id appears to be quite important, which is unusual as the 'id' column is typically a unique identifier for each sample and should not have predictive power. This may suggest a need to re-evaluate the feature inclusion process, as including the 'id' in the model training can lead to overfitting.Other significant features include concave points_mean, compactness_se, area_se, and concavity_worst, which also have high F scores, indicating their strong influence on the model's decisions.

It's important to note that feature importance should be interpreted with caution. High importance does not necessarily mean that the feature is a good predictor. For example, the id column might appear to be important due to overfitting or data leakage.

Additionally, this chart is useful for feature selection and understanding the model's behavior. It can guide the data scientist in improving the model by focusing on the most relevant features and potentially discarding or reevaluating less important ones.

=========

Each step is crucial for understanding the workflow of using XGBoost in a machine-learning task:

• Importing libraries sets up our environment.
• Loading the dataset is about getting our data into a workable format.
• Data preprocessing ensures our model receives the right type of input without any irrelevant or missing data.
• Splitting the dataset is necessary to train the model and then test its performance on unseen data.
• Model training is where XGBoost learns from the training data.
• Model evaluation helps us understand how well our model is performing.
• Feature importance gives insights into which features are most influential in the model's decisions.

The full Notebook can be found here: