Are you testing your ML models to the same extent as your other software?

Unit; Integration; Functional; End-to-End; Acceptance; Performance; Smoke; A/B Testing... the list could probably go on for a while.

Chances are if you are building software (and these days who isn't) you are doing most of, if not all, of the above tests as a standard part of your SDLC (software development lifecycle). You are confident in your test coverage and keep working at improving it so that you can have as good a change failure rate DORA score as possible. Great! This is where you either are now, or are looking to get to with aspirations of continuous development. Automate all the things and make your devs lives that much easier, catching and correcting problems before they get to production. Hopefully, you've shifted all of the testing left sufficiently that when your developers merge into the repo, they get all the information they need to know if what they built was good enough - before it needs to go to anyone in the ops or QA teams. (*cough* including security testing! *cough* )

The devs are happy because they have this quality gate to ensure they are doing their best possible work with some guard rails in case anything gets borked in the process when they merge back into main. The portfolio managers are happy because they get some nice dashboards and reports to show them how the whole of the portfolio is performing in terms of quality. Engineering managers likewise get to know how well each code base is performing and perhaps where they need to go back and look at a sprint for paying back some technical debt.

In other words, if you are building software, you probably have your QA process well defined and mature. It may well be that your CI pipelines can make your devs the first part of your QA process.

So, if you are doing all of this testing for your SDLC, are you also doing the same for your machine learning development?

If you are, great! We would love to hear more about what you are doing and how it is going for you.

If not, let's talk. TruEra (the company I work for) have built a model testing suite. It's based on our own academic research into the explainability of AI/ML models and is all about helping you improve the quality and trust in the work that your DS teams are doing. We're still young - only a couple of years old - but are seeing a large uptick in the interest of teams wanting to improve their model validation speed, improve collaboration and explainability for non-DS stakeholders around the business and make sure that they comply with any current or future regulations (hint: EU AI regs will likely become law in the next 24 months; NYC already has them for HR hiring; Singapore has produced guidance; the UK is going to spin out its own flavour etc etc etc).

So what do you think? Whether you are on the DS or Dev sides of the house, let me know (a) if this is just a false analogy - you can't assume that the model and software dev processes are the same with regards to testing requirements; (b) if this is bang on and something you hadn't considered before; (c) if I should leave the writing to the professionals in the future... ;)