Insurers expect audit to evolve to accommodate AI reserving systems

In an era where Artificial Intelligence (AI) is reshaping the insurance landscape, traditional audit methods are facing a new challenge: black-box reserving models. These sophisticated systems promise efficiency and precision but present a realm of new complexities.

Gary Stakem , Actuarial Director in Mazars in Ireland , outlines ten steps on how the audit approach must adapt to meet emerging industry needs, whilst maintaining accuracy, compliance, and trust in financial reporting.

1 - Upskilling audit specialist teams

Insurance audit teams have traditionally relied upon actuarial specialists to assess complex reserving models. Actuarial teams bring a deep understanding of the commercial and regulatory environments in which insurers operate. However, actuarial audit teams must now upskill and embrace machine learning analysis techniques, ensuring algorithms are explainable, transparent, and compliant with regulatory and accounting standards.

2 - Enhanced risk assessments

Risk assessments are a critical step in the audit planning process, helping actuaries to identify significant risks within reserves and design appropriate audit procedures.

With the advent of AI, risk assessments must go beyond financial risks. Model bias, overfitting, data quality, and model transparency must be considered as significant in-scope risks.

3 - Data governance expectations

Traditional actuarial models are inherently interpretable - for example, audit teams can easily conduct a visual inspection of non-life claim development patterns. In life insurance, plots of mortality or lapse experience often uncover discrepancies in underlying data.

The shift towards less transparent black boxes will limit the ability to deploy these ‘in-model’ controls. This greatly increases the importance of pre-model data governance and hence audit procedures will need to respond accordingly. Manual interventions to cleanse or transform data are potential sources of management bias. Feature engineering, a step whereby raw data is transformed into features that enable more effective machine learning, is likely to become a major audit focus.

4 - Benchmarking against traditional methods

Traditional actuarial reserving techniques have long been part of the auditor's toolkit and this well-understood approach is unlikely to fully disappear anytime soon – it will provide an established benchmark for assessing black-box model results.

But what happens when black-box models claim superior results that cannot be validated by traditional methods alone? Actuarial audit teams will need to exercise expert judgment and professional skepticism, knowing when to bring more sophisticated techniques into the audit process.

5 - Black-box vs black-box

The notion of using one AI machine to challenge another may sound like a scenario somewhat reminiscent of a sci-fi film, yet it could become commonplace in the actuarial audit domain.

This independent approach could prove critical in identifying potential user bias in the selection and design of AI models. Such a technique underscores the need for actuarial audit teams to enhance their coding skills to develop credible AI algorithms independently.

6 - Model architecture and hyperparameter assessment

AI models have gained popularity based on their ability to self-learn, refining their own model parameters to optimise accuracy.? Nevertheless, initial human oversight is required to design an appropriate architecture and to set external model parameters that are not machine adjustable – often referred to as ‘hyperparameters’. Designing a suitable architecture is a subjective art and introduces a new source of potential management bias. Partition of actuarial data into subsets, for training and testing of AI models, will also be subject to audit attention.

Benchmarking and sensitivity testing of key assumptions is no stranger to actuarial audits. This practice now looks set to evolve with assessing variations of hyperparameters becoming a new norm.

7 - Explainable AI (XAI) testing

Explainable AI, or ‘XAI’, is fundamental to understanding, trusting, and effectively communicating the behaviour of black-box reserving models. The modern audit process will include both global and local XAI techniques.

Global techniques examine a model’s overall behaviour providing insights into the relationship between black-box inputs and predicted outputs. Use of surrogate models is an example of one such common technique whereby simplified, more interpretable, models are fit to broadly replicate an insurer’s more sophisticated black-box model.

Local XAI techniques focus on explaining individual policy projections. Sampling of individual policies is not an unfamiliar concept to insurance audit, but new techniques such as ‘LIME’ and ‘SHAP’ will better equip actuarial teams seeking to understand individual results.

Deploying a combination of global and local XAI techniques can increase credibility and confidence in black-box models.

8 - Expert judgments and model overrides

Challenging the actuarial function’s expert judgment is already central to the audit process.? Reserving models often need to be adjusted in response to emerging risks not reflected in historic data sets. ?Covid-19, judicial developments, and high inflation rates have been prime examples in recent years. But with sophisticated AI models, it will no longer be straightforward to ‘go-inside’ models and tweak the parameters.

Use of synthetic or augmented datasets could become one workaround solution. Alternatively, post-processing algorithmic intervention could be utilised. Evidencing the appropriateness of these bespoke adjustments will become more challenging if underlying models are less transparent.

Auditors are accustomed to challenging insurers’ actuaries on expert judgements regarding the commercial and external environment – this assessment must now also extend to machine learning judgements. The scarcity of this specific blend of skills and domain knowledge could see insurers place ever increasing reliance on a small group of management experts. This will call for audit specialists with equally nuanced skills, capable of balancing precision and pragmatism.

9 - Model reliability, stability and reproducibility

Consistency and reliability are the bedrock of actuarial reserving models, qualities that are non-negotiable even as business landscapes evolve. Auditors must adapt and employ rigorous tests to ensure that AI models can withstand the scrutiny of varied datasets. Models should be capable of producing reasonable results even on obscure scenarios it was not trained upon.

While AI models, such as neural networks, cannot be identically replicated, audit teams should at least be capable of reproducing reasonable imitations. This will require insurers to maintain robust documentation on code management, model logs, and deployment procedures.

10 - Stakeholder collaboration

Deployment of AI in actuarial reserving is still a field in its infancy with development of commonly accepted AI audit standards even less mature. Audit firms must stay engaged with actuarial and accountancy professional bodies, keeping abreast of emerging standards, legislation, and regulatory expectations.

Other stakeholders, such as boards and audit committees, will expect regular and transparent communication through periods of change.

Conclusion

The transition towards integrating AI into actuarial practices demands a proactive approach from both insurers and auditors alike, emphasising the need for transparency, collaboration, and ongoing education to maintain confidence in financial disclosures. It is sound advice for insurers to initiate a conversation with their auditor right from the outset of their journey.