Classification Measures in Machine Learning

In classification problems, it’s crucial to have effective measures to evaluate how well our model is performing. Unlike regression, where measures like R2 and mean squared error help assess model performance, classification tasks rely on other metrics such as Accuracy, Precision, Recall, and F1 Score due to the categorical nature of predictions.

When we’re just getting started with machine learning, evaluating our model’s performance might seem confusing, especially with terms like precision, recall, F1 score, and accuracy. But fear not! In this article, we’ll walk through these important concepts step by step, starting with the confusion matrix and going all the way to understanding F1 score and the Dice coefficient.

Confusion Matrix

Before diving into classification measures, we need to understand the confusion matrix. It’s the foundation upon which accuracy, precision, recall, and other metrics are built. Simply put, the confusion matrix is a table that helps visualize how well a classification model is performing.

• True Positives (TP): When the model correctly predicts a positive class.
• True Negatives (TN): When the model correctly predicts a negative class.
• False Positives (FP): When the model incorrectly predicts a positive class (also known as a Type I error).
• False Negatives (FN): When the model incorrectly predicts a negative class (also known as a Type II error).

Example: Let’s imagine we’re building a model to predict whether an email is spam (positive class) or not spam (negative class). If the model predicts 50 emails correctly as spam and 40 correctly as not spam, but mistakenly labels 10 non-spam emails as spam and misses 5 actual spam emails, this will look like:

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

iris = datasets.load_iris()
x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target, 
                                                    random_state = 1)
clf = LogisticRegression(solver = 'liblinear')
# fit the training data
clf.fit(x_train, y_train)
# Do prediction on training data
y_train_pred = clf.predict(x_train)
# Do prediction on testing data
y_test_pred = clf.predict(x_test)
# find the confusion matrix for train data
confusion_matrix(y_train, y_train_pred)
# find the confusion matrix for test data
confusion_matrix(y_test, y_test_pred)
# get the full classification report for training data
print(classification_report(y_train, y_train_pred))
# get the full classification report for testing data
print(classification_report(y_test, y_test_pred))        

With the confusion matrix in hand, we can now calculate key metrics like accuracy, precision, recall, and F1 score.

Accuracy

Accuracy tells us the percentage of predictions the model got right. It is calculated as:

from sklearn.metrics import accuracy_score
y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
print("Score :", accuracy_score(y_true, y_pred))        

Accuracy is simple and intuitive, but it doesn’t always tell the full story, especially with imbalanced datasets (when one class significantly outweighs the other). That’s why we also look at precision and recall.

Precision

Precision focuses on how many of the predicted positive cases were actually correct. It’s important when false positives are costly (like predicting fraud when there’s none).

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]
from sklearn.metrics import precision_score
precision_score(y_true, y_pred)        

Recall

Recall (or sensitivity) tells us how many actual positive cases were correctly identified. It’s critical when missing a positive case (false negative) is costly, such as in medical diagnoses.

F1 Score

The F1 score balances both precision and recall. It’s particularly useful when we need to account for both false positives and false negatives.

Classification Report

The classification report is a summary that includes precision, recall, F1 score, and support (the number of occurrences of each class). In Python’s sklearn, we can generate this easily:

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))        

This gives us a complete view of how our model performs on each class.

Bringing It All?Together

To summarize, classification metrics help us understand how well our model is performing beyond just accuracy.

• Accuracy: Best for balanced datasets.
• Precision: Important when false positives are costly.
• Recall: Crucial when false negatives are costly.
• F1 Score: Useful when we need a balance between precision and recall.