Model Validation: The Key to Sound Financial Modelling and Risk Management

Introduction to Model Validation

Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps ensure that models are sound. It also identifies potential limitations and assumptions and assesses their possible impact. As with other aspects of effective challenge, model validation should be performed by staff with appropriate incentives, competence, and influence.

Comprehensive Approach to Model Components

All model components, including input, processing, and reporting, should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants. The rigor and sophistication of validation should be commensurate with the bank’s overall use of models, the complexity and materiality of its models, and the size and complexity of the bank’s operations.

Validation Reports: Ensuring Model Integrity and Responsiveness to Risk

Validation reports should articulate model aspects that were reviewed, highlighting potential deficiencies over a range of financial and economic conditions and determining whether adjustments or other compensating controls are warranted. Effective validation reports include clear executive summaries, with a statement of model purpose, and an accessible synopsis of model and validation results, including major limitations and key assumptions.

The range and rigor of validation activities conducted prior to the first use of a model should be in line with the potential risk presented by the use of the model. If significant deficiencies are noted as a result of the validation process, the use of the model should not be allowed or should be permitted only under very tight constraints until those issues are resolved. If the deficiencies are too severe to be addressed within the model’s framework, the model should be rejected. If it is not feasible to conduct necessary validation activities prior to model use because of data paucity or other limitations, that fact should be documented and communicated in reports to users, senior management, and other relevant parties. In such cases, the uncertainty about the results that the model produces should be mitigated by other compensating controls. This is particularly applicable to new models and to the use of existing models in new applications.

Validation activities should continue on an ongoing basis after a model goes into use, to track known model limitations and to identify any new ones. Validation is an important check on model use during periods of benign economic and financial conditions, when estimates of risk and potential loss can become overly optimistic, and when the data at hand may not fully reflect more stressed conditions. Ongoing validation activities help to ensure that changes in markets, products, exposures, activities, clients, or business practices do not create new model limitations. For example, if credit risk models do not incorporate underwriting changes in a timely manner, flawed and costly business decisions could be made before deterioration in model performance becomes apparent.

Banks should conduct a periodic review—at least annually but more frequently if warranted—of each model to determine whether it is working as intended and if the existing validation activities are sufficient. Such a determination could simply affirm previous validation work, suggest updates to previous validation activities, or call for additional validation activities. Material changes to models likely warrant validation. It is generally good practice for banks to ensure that all models undergo the full validation process at some fixed interval, including updated documentation of all activities.

Model Validation Outcomes: Assessing Reliability and Managing Model Risk

Effective model validation helps reduce model risk by identifying model errors, corrective actions, and appropriate use. It also provides an assessment of the reliability of a given model, based on its underlying assumptions, theory, and methods. In this way, it provides information about the source and extent of model risk. Validation also can reveal deterioration in model performance over time and can set thresholds for acceptable levels of error, through analysis of the distribution of outcomes around expected or predicted values. If outcomes fall consistently outside this acceptable range, then the models should be redeveloped.

An effective validation framework should include three core elements:

·?????? Evaluation of conceptual soundness, including developmental evidence.

·?????? Ongoing monitoring, including process verification and benchmarking.

·?????? Outcomes analysis, including back-testing.

A model should be validated before it is put into use. The rigor of validation before implementation should be commensurate with the potential risk presented by the use of the model. If the bank has not fully validated models before implementation, examiners should assess the bank’s compensating controls and other measures to mitigate risks. Model reviews and validations (in whole or in part) are generally performed using a risk-based approach, and with a frequency appropriate for, or when, there are changes to a bank’s risk profile.

Material changes to models may warrant validation. Appropriate validation reports generally include the review of the conceptual soundness of a model for its intended purpose and the results of ongoing monitoring, process verification, benchmarking, and outcomes analysis.

A sound validation process generally includes:

·?????? Defined purpose and goals.

·?????? Scope, validation approach, schedule, resources, and types and extent of validation activities and tasks.

·?????? Specific actions that must be taken to complete individual validation activities and tasks.

·?????? Detailed and sufficient documentation to demonstrate that all validation procedures are appropriately completed.

While control staff may grant exceptions to typical procedures of model validation on a temporary basis, that authority should be subject to other control mechanisms, such as timelines for completing validation work and limits on model use.

The Imperative of Independence and Expertise in Model Validation

Validation involves a degree of independence from model development and use. Generally, validation should be done by people who are not responsible for development or use and do not have a stake in whether a model is determined to be valid. Independence is not an end in itself but rather helps ensure that incentives are aligned with the goals of model validation. While independence may be supported by separation of reporting lines, it should be judged by actions and outcomes, since there may be additional ways to ensure objectivity and prevent bias. As a practical matter, some validation work may be most effectively done by model developers and users; it is essential, however, that such validation work be subject to critical review by an independent party, who should conduct additional activities to ensure proper validation. Overall, the quality of the process is judged by the manner in which models are subject to critical review. This could be determined by evaluating the extent and clarity of documentation, the issues identified by objective parties, and the actions taken by management to address model issues.

In addition to independence, banks can support appropriate incentives in validation through compensation practices and performance evaluation standards that are tied directly to the quality of model validations and the degree of critical, unbiased review. In addition, corporate culture plays a role if it establishes support for objective thinking and encourages questioning and challenging of decisions.

Staff doing validation should have the requisite knowledge, skills, and expertise. A high level of technical expertise may be needed because of the complexity of many models, both in structure and in application. These staff also should have a significant degree of familiarity with the line of business using the model and the model’s intended use. A model’s developer is an important source of information but cannot be relied on as an objective or sole source on which to base an assessment of model quality.

Staff conducting validation work should have explicit authority to challenge developers and users and to elevate their findings, including issues and deficiencies. The individual or unit to whom those staff report should have sufficient influence or stature within the bank to ensure that any issues and deficiencies are appropriately addressed in a timely and substantive manner. Such influence can be reflected in reporting lines, title, rank, or designated responsibilities. Influence may be demonstrated by a pattern of actual instances in which models, or the use of models, have been appropriately changed as a result of validation.

Independent validation may be performed in-house, by a third party, or a combination thereof. In large or complex banks, model validation is typically conducted by IRM (e.g., model risk management function in a second line of defense) or by an independent third party. Sometimes, particularly for small or noncomplex banks, some validation work may be most effectively done by model developers or users. In such cases, an independent party with appropriate technical knowledge typically provides critical review and effective challenge and conducts additional activities to confirm proper validation.

Addressing Model Limitations: The Role of Outcomes Analysis and Rigorous Validation

Outcomes analysis and the other elements of the validation process may reveal significant errors or inaccuracies in model development or outcomes that consistently fall outside the bank’s predetermined thresholds of acceptability. In such cases, model adjustment, recalibration, or redevelopment is warranted. Adjustments and recalibration should be governed by the principle of conservatism and should undergo independent review.

Material changes in model structure or technique, and all model redevelopment, should be subject to validation activities of appropriate range and rigor before implementation. At times banks may have a limited ability to use key model validation tools like back-testing or sensitivity analysis for various reasons, such as lack of data or of price observability. In those cases, even more attention should be paid to the model’s limitations when considering the appropriateness of model usage and senior management should be fully informed of those limitations when using the models for decision making. Such scrutiny should be applied to individual models and models in the aggregate.

Examiners should determine if bank management has appropriate processes in place to validate models. In assessing the effectiveness of model validation processes, examiners generally evaluate the extent and clarity of documentation, issues identified by the validation, and the actions bank management takes to address such issues.

Evaluation of Conceptual Soundness in Model Risk Management

This element of validation involves assessing the quality of the model design and construction. It entails review of documentation and empirical evidence supporting the methods used and variables selected for the model. Documentation and testing should convey an understanding of model limitations and assumptions. Validation should ensure that judgment exercised in model design and construction is well-informed, carefully considered, and consistent with published research and with sound industry practice. Developmental evidence should be reviewed before a model goes into use and also as part of the ongoing validation process, in particular whenever there is a material change in the model.

A sound development process will produce documented evidence in support of all model choices, including the overall theoretical construction, key assumptions, data, and specific mathematical calculations. As part of model validation, those model aspects should be subjected to critical analysis by both evaluating the quality and extent of developmental evidence and conducting additional analysis and testing as necessary. Comparison to alternative theories and approaches should be included. Key assumptions and the choice of variables should be assessed, with analysis of their impact on model outputs and particular focus on any potential limitations. The relevance of the data used to build the model should be evaluated to ensure that it is reasonably representative of the bank’s portfolio or market conditions, depending on the type of model. This is an especially important exercise when a bank uses external data or the model is used for new products or activities.

Evaluation of conceptual soundness generally includes such activities as the following, as appropriate:

·?????? Evaluating the quality and extent of developmental evidence and conducting additional testing as necessary.

·?????? Assessing whether the model achieves the intended purpose.

·?????? Comparing alternative model theories and approaches.

·?????? Justifying the choice of a particular model theory and approach.

Assessing key assumptions and variables, with analysis of their impact on model outputs and particular focus on any potential limitations, including model transparency and explainability for AI approaches.

Evaluating the relevance of the data used to build the model to validate that data are reasonably representative of the model’s inputs, such as the bank’s portfolio, account activity, or market conditions, depending on the type of model. This is particularly important when a bank uses external data or the model is used for new activities.

Sensitivity analysis and stress testing.

An evaluation of conceptual soundness may be difficult for some complex models (e.g., those that use AI approaches) because the underlying theory and logic may not be transparent. Transparency and explainability are key considerations that are typically evaluated as part of effective risk management regarding the use of complex models. The appropriate level of explainability of a model outcome depends on the specific use and level of risk associated with that use. Models applied to significant operations or decisions (e.g., credit underwriting decisions) should be supported by a thorough understanding of how the model arrived at its conclusions and validation that it is operating as intended. There may be challenges with explaining some models based on complexity or, in some cases, limited documentation provided for third-party models. Examiners should discuss with bank management the bank’s process for exploring various approaches to determine whether bank personnel have an understanding of how models function and make decisions, including identifying any limitations and use of compensating controls.

Conclusion

Model validation, as a dynamic and integral part of model risk management, adapts to changes in the bank's operations, market conditions, and innovations in modelling. This comprehensive approach ensures the reliability and effectiveness of models in the banking sector.

Reference

Comptroller’s Handbook on Model Risk Management, pages 36-41.