Model Risk Guidelines-Bank of England-SS1/23

The use of models is pervasive in the financial domain. However, it can be argued that model risk framework is still catching up in several countries. Recently, Bank of England has recently released it model risk framework which will be effective from May 2024 for all the U.K-incorporated banks. While the new supervisory statement SS1/23 released by EBA aligns with the prevalent SR11 guidelines in the U.S., I find the new guidance provides clarity on some granular aspects. The following is a summary of the guidelines from the model validation perspective. In this article, I will limit the scope to the top three areas of validation: Conceptual Soundness, Ongoing Monitoring, and Governance:

Conceptual Soundness

• Conceptual soundness can be supported by published research where available or industry practice where applicable.
• Choice of variables included and the parameters should be in line with the design objectives.

Data and Inputs:

• Model development process should ensure there is no bias in the data used and data aligns with the privacy guidelines
• Input data used to develop is a representative of firm’s products/portfolios. When unrepresentative data had to be used, it should be addressed through a limitation.
• Model inventory should include adjustments made to the input data and a record of interconnected data sources

Ongoing Monitoring and outcomes analysis

• ?Use of benchmarking models to support the model output is advised.
• Guidelines prescribes four broad outcomes analysis techniques

1. Benchmarking – comparing model estimates with comparable but alternative estimates;
2. Sensitivity testing – reaffirming the robustness of the model
3. Analysis of overrides – evaluate and analyze the performance of model post adjustments
4. Parallel outcomes analysis – assessing whether new data should be included in model calibration

• ?Performance monitoring should be assessed from two perspectives: (i) backward-looking and (ii) Forward-looking. The use of actual observations to assess performance is backward-looking. The use of sensitivity analysis to assess the extent to which economic conditions can deteriorate is forward-looking. ?
• ?Sensitivity analysis can be viewed as a performance monitoring tool as it can be used to establish model’s operating boundaries. Within the boundaries the model performance is expected to acceptable and beyond which model risk increases.

?Model governance

• ?Model validation includes independent review of the model, performance monitoring of the model, and process verification.
• Model validation should include the review of model risk mitigants (overlays) that are applied when model is not performing as expected.
• The design objectives of a model should be pre-determined. These represent discriminatory power for rating systems, and accuracy for provisioning or pricing models, and may represent a degree of conservatism for liquidity and capital models
• ? Model validation activities should include a review of complexity and materiality of the model. ?
• Model materiality is a function of two factors: quantitative and qualitative. Quantitative factors such as book size, market value, or number of customers modeled. The qualitative factors include relative importance of the model for the business decisions.
• Complex models are those that satisfy one of the three criteria. Models that are difficult to explain in non-technical technical terms. Models for which it is difficult to anticipate the model output given model input. Models that leverage interconnected or unstructured data as input.
• Model risk mitigants typically include overlays, restrictions, and how model validation exceptions are escalated.

No model validation activity is complete without a review of the model documentation. Often, documentation-related findings is the most common issues in a model validation. Fortunately, guidance sheds some light on the documentation aspect as well.

Minimum documentation expectations

1. Model Development Document: The document should be detailed enough for independent reader with the relevant expertise would be able to follow the model. With the keyword used being "Relevant expertise" reduces a lot of ambiguity.
2. Model adjustment documentation (PMA): The documentation should at least cover (i) clear justification for applying PMAs; (ii) the criteria to determine how PMAs should be calculated and how to determine when PMAs should be reduced or removed; (iii) triggers for prolonged use of PMAs to activate validation and remediation

In conclusion, the supervisory statement from BoE is a welcome move as it provides regulatory guidance at a slightly more granular level, reducing the interpretation issues.

References:

(1) Bank of England Model Risk Guidelines

https://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-management-principles-for-banks-ss