How to use Machine Learning Models for Credit Risk: Best Practices
Machine Learning has the potential to be extensively used in managing #creditrisk, especially in this age of instant credit decisions. The use cases could be as diverse as origination, pricing, credit scoring, fraud prevention, and collection.
While #machinelearning based algorithms could be a boon in certain situations, such as where patterns are hard to detect through traditional data analysis techniques, there is also a flip side.
#Machinelearning based models could suffer from serious flaws such as instability, bias, and explainability, if the models are not constructed, used, and governed properly. There could also be other reasons, that influence the reliability of the models, such as:
For these reasons, despite its potential widespread usage, #Machinelearning models have to be used along with appropriate guardrails. In this respect, Model Risk Management of #machinelearning models is a very deep topic by itself. Various regulators such as the OCC and the European Central Bank have come out with guidelines on the usage and governance of machine learning models.
In this article, we examine key best practices pertaining to the usage and governance of #machinelearning models, especially with regard to credit.
Defining Problem Statement
Defining a problem statement, as precisely as possible, is one of the perquisites of applying #Machine Learning techniques for managing #Credit Risk.
For example, the identification of good customers for origination (new customers) and the identification of good customers for cross-selling (selling new products to existing customers) are two different problem statements. In the former case, the prospect is New-To-Book (hence there is availability of limited behavioral characteristics), while in the latter case, he/she has matured from a prospect to a customer (availability of limited behavioral characteristics); necessitating different approaches to solving the problem.
The illustrative construct of problem statements where #Machine Learning could help in managing #Credit Risk could be:
This process can be drilled down to the next level, where we can ask a much more precise question such as "Identification of good prospects for origination for a Personal (unsecured) Loan product for self-employed."
This step is essential from the point of view of selecting and applying the appropriate technique for solving the problem. For example, Classification Models may be more suited to problem statements involving origination and credit score; whereas Unsupervised learning techniques may be suited for use cases related to fraud detection.
Understanding Data
Data needs to be understood from various angles such as:
Selection of Technique
Multiple #machinelearning techniques have to be evaluated based on factors such as
Approach Finalization
Until the finalization state, alternative #Machine Learning approaches have to be considered. There is not one approach that is devoid of limitations and can be claimed as perfect. During the finalization stage, the pros and cons of the approaches have to be discussed and at least two approaches can be selected for pilot testing.
It is best if such deliberations are decided after consulting multiple stakeholders such as business, risk, compliance, and finance. This is very important as the different stakeholders' inputs, feedback, and concerns could be considered in selecting the alternatives. For example, #riskmanagement or # compliance could provide input on the acceptability of the approach from a regulatory acceptability standpoint.
Based on the selected approaches, models have to be developed as per standard model development frameworks.
领英推荐
Pilot Testing
Pilot testing has to be done with at least two alternative approaches. This is distinct from various testing (such as out-of-time/out-of-sample testing) which is done as part of the modeling process.
Pilot testing of #machinelearning models needs to be done on real-life data; For example, it could be in the form of limited roll-outs for certain customer segments.
Pilot testing of alternative #machinelearning models gives valuable insights into the robustness of the models, given the complexities associated with real-life situations.
An evaluation window needs to be fixed in advance and the results of the pilot testing need to be monitored over the evaluation window.
At the end of the evaluation window, the results need to be deliberated in a multi-disciplinary group and a way forward to be arrived at.
Challenger Model
Challenger models are very important aspects of monitoring model performance over time. One of the alternative #machinelearning models considered during the pilot testing can be regarded as the Challenger Model. Both the primary and challenger models need to run in parallel and results need to be compared for substantial deviation in performance.
In fact, if the Challenger Model proves to be better than the primary models over a period of time, the primary model can be replaced with the Challenger model.
Back Testing
Backtesting is the process of validating the models, by looking backwards wherein actual results are compared against predicted results. #Machinelearning models can be scored against various parameters such as Predictability, Stability, and Explainability. The models can be scored against each of these parameters (based on the validator's expert judgment or a policy-based scoring matrix) and a composite performance evaluation can be undertaken by a multi-disciplinary committee, on a periodic basis.
Independent Validation
It is important to have #machinelearning models validated frequently by external parties. This helps not only bring to notice any gaps that the internal teams may have overlooked but also brings in external perspectives and best practices.
Limitations
Every model has its own limitations and these must be clearly documented. Such model limitations must also be communicated to the model users. Limitations could be in the form of:
Documentation
Documentation is a very important but often overlooked aspect of #machinelearning models. This is very important from the perspective of audibility, continuance, and transparency.
Conclusion
#Machinelearning models have to be used with sufficient guardrails; failing which the fallout from the improper usage and governance of such models could overshadow the much sought-after benefits.
Further Reading
'Machine Learning Explainability in Finance: An application to default risk analysis' by Philippe Bracke, Anupam Datta, Carsten Jung and Shayak Sen [Bank of England]
The potential of Machine Learning in managing #creditrisk is substantial, encompassing origination, pricing, credit scoring, fraud prevention, and collection. However, challenges like instability, bias, and explainability require careful construction, usage, and governance of ML models. Factors such as misfit algorithms, data limitations, biased models, 'black box' issues, and real-life reliability underscore the need for appropriate guardrails. Recognizing this, regulatory guidelines from entities like the OCC and the European Central Bank emphasize the importance of robust Model Risk Management for #machinelearning models.