Agile Model Validation: threat or opportunity?

As a consequence of the standard requirements for regulatory models, like internal rating based models for credit risk capital, finance professionals often equate model risk management with model validation. With the exploding range and scope of models and algorithms that are in the remit of model governance, it is worth remembering that model risk mitigation can be done through a variety of approaches, including model validation, effective challenge through continuous monitoring and volumetric limits on the amounts of business being supported by the model under consideration.

Even in the world of classical risk model development, but definitely in the realm of data science modelling, the agile way of working is gaining prominence as the standard. Pressure is therefore building on model validation teams to smoothly insert themselves in the agile development process. In practice this could mean that the end-to-end validation just prior to industrialization of the algorithm, is replaced by intermediate validations e.g. on the data set, on the methodology, on the outcome, ...each with validation opinions.

A priori such an agile validation approach could have several benefits.

• For example, validation at intermediate stage gates requires that documentation of completed steps is available. This likely improves overall documentation quality.
• Moreover, having to opine on data adequacy or methodology appropriateness without knowing the actual outcome, will eliminate any mental anchoring or survivor bias in the mind of the validators, caused by knowledge of the outcome. This could, in fact, actually lead to better effective challenge, the ultimate objective of validation. In the paper “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results” by R. Silberzahn and many others, the benefits of such an approach were actually tested.
• Thirdly, material findings by validation can be identified (and mitigated) on a more timely basis. No need to wait for the next model upgrade in three years or so.
• Last, but not least, it is likely that with the increase of different modeling and algorithm approaches in scope, specialization is needed, with bespoke validation experts for data adequacy and data privacy, for assessing the algorithms and modeling approaches, for challenging the robustness of the industrialization in an ML Ops framework, etc. The times when one validator with a PhD in a relevant field could cover it all, are long gone.

Of course, a potential risk is that the incorporation of the intermediate feedback in the final model, harms the independence of the model validation team. Also, it is well known from the psychology literature, that progressively received information can anchor you in a wrong first hypothesis. This can be easily remedied by having another validator do the end2end validation, building on the intermediate reports, but coming with a fresh perspective. Also, internal audit could opine on the appropriate maintenance of independence. And, there is also a bit of an absurdity when this logic is pushed to its extreme: any adoption of a model validation recommendation or suggestion would jeopardize independence if this point of view was maintained.

As this is a discussion that is likely ongoing at many institutions, I would love to see your comments and thoughts.