Benchmarking PD Models

When evaluating various scoring functions for the Probability of Default (PD) modeling, the most commonly assessed performance metric is the Gini score. The Gini score measures the rank-ordering power of the model, which helps determine how effectively the model can identify default observations based on the order of its predictions. A Gini score of 1 indicates perfect discrimination ("crystal ball" performance), while a score of 0 implies no discriminatory power. However, in certain scenarios, the Gini coefficient can be a misleading metric and could lead to a selection of a less optimal model.

Imagine a situation where two scoring functions, WOE Logistic Regression (WOE LR) and XGBoost, perform similarly in terms of the Gini score (Gini = 2 * AUC - 1):

In this scenario, a common method of model evaluation beyond the Gini coefficient is to establish a classification rule based on a probability model and assess the outcomes using a cost-benefit analysis. This analysis involves assigning fixed costs and benefits based on the costs of misclassification at a predetermined cut-off.

Given that this approach requires setting a specific cut-off point for classifying a prediction as being correct (e.g., 4% or 50%), it's not guaranteed that a model selected using this method would perform similarly on the new data when the underlying distribution of the target variable is different from the training dataset.

Log Loss

One metric for assessing PD model performance that is not so often discussed is a logarithmic scoring rule, more known as Log Loss:

Log Loss measures the proximity of predictions to the true labels and is a common loss function in model estimation methods such as Logistic Regression and Gradient Boosted Decision Trees. Additionally, this metric is employed to assess the calibration of forecasts.

Log Loss assigns more weight to predictions that are "correct". When predictions deviate significantly, for example, when the predicted probability is 0.99 for a "good" customer (default flag is 0) or 0.01 for a "bad" customer (default flag is 1), we observe large loss values, indicating lower accuracy of our predictions:

Alternatively, if the predicted probabilities are close to the true labels of default event, the loss will be close to 0 indicating high accuracy of our predictions:

Below Loss curves for the two scoring functions with similar Gini scores are shown:

We can observe that Loss curves for our models are quite similar, but there are differences in the lower probability range for the defaulted cases (left chart) and in the upper probability range of the non-defaulted cases (right chart) where loss values are high indicating inaccuracies. Naive loss corresponds to a forecast of a sample average default rate (50%) for each observation.

Looking at these results it is difficult to draw a conclusion whether one model is preferable over the other.

Loss Uplift

There is a way how we can compare two Loss metrics to benchmark the two models using a Loss uplift metric:

The Loss uplift metric is conceptually similar to Normalized Cross Entropy from He X. et al. 2014 paper, but here we benchmark our challenger model against a baseline model and not a "random guess" or naive prediction model (e.g., assigning 50% probability for every observation).

Essentially, Loss uplift an arithmetic average of the absolute differences between Log Loss values of the two models for each prediction. Negative values indicate better performance compared to the baseline and vice versa.

In our example, we will benchmark WOE LR (challenger) against XGBoost (baseline).

Loss uplift can be calculated for each class separately and on the overall level, the results are reported below:

• Uplift goods (class = 0): -0.20%
• Uplift bads (class = 1): -0.02%
• Uplift: -0.10%

The metrics above show that WOE LR has a lower error and has an improved performance for both classes.

Business metric

To visualize the Loss uplift, we will create a business metric based on a cost-benefit analysis with the following logic in mind:

• We assign +€100 when our challenger model predicted a higher probability for Class 1 vs the baseline and -€100 otherwise
• We assign +€50 when our challenger model predicted lower probability for Class 0 vs the baseline and -€50 otherwise

This approach allows us to give a graphical representation in the form of Payout curves for the Loss uplift metric:

What we observe in the chart above that most of the payout is coming from the improvements in prediction accuracy for defaulters in the probability range > 70%. Due to a higher weight (€100) used for defaulted observations, we can see that our choice in selecting WOE LR would be warranted from a business perspective.

Total payout of using the challenger vs the baseline amounts to +€20,300.

Concluding remarks

Benchmarking PD models with similar rank-ordering power based on alternative metrics can help in identifying most promising models with higher payouts of model use. Utilizing a combination of scoring rules and business metrics enables risk practitioners to look beyond Gini scores in search of the best scoring model.

--

I hope you have enjoyed reading this post!???

The technical appendix with the code can be found in this notebook.

All views expressed are my own.

References

He, Xinran, et al. "Practical Lessons from Predicting Clicks on Ads at Facebook." Proceedings of the Eighth International Workshop on Data Mining for Online Advertising. 2014. https://research.facebook.com/publications/practical-lessons-from-predicting-clicks-on-ads-at-facebook/