When should you release a model in production?

What’s the general process for buying a burger from a restaurant? You look at a menu, you order, and the waiter or waitress returns to your table, with a burger, ready for you to eat. They might offer some additional condiments, and then you can enjoy your burger. Next time you go to the restaurant, perhaps they’ve improved the burger, and it tastes even better, closer to what a “perfect” burger tastes like.

Alternatively, they you could order and wait several months till they’ve “improved” the burger, even if it tastes better. That’s not exactly practical. If you go to a restaurant you kind of want a burger then, because you’re hungry, not to have one in several months time.

When you create a financial model, it’s kind of similar. It’s a question I’ve encountered often in my career, whether it’s at?Turnleaf Analytics, which Alexander Denev and I co-founded to forecast economic variables like inflation using machine learning or in the past, when I’ve been assessing whether or not to deploy a systematic trading strategy in production, whether it was a bank or on my personal account. Indeed, it’s a subject Alexander Denev and I discussed at length in?The Book of Alternative Data.

The first point to note, is that you’ll never be able to create a “perfect” financial model, with 100% accuracy, and indeed in finance, your level of accuracy (or measure of success) can be a lot lower, and yet the model can still be monetised. If you’re better 50% of the time (depending on the return skew), that can be a good result. You will instead have the best model you can build at that time. Also the longer it takes to put something into production the more time you don’t benefit from the output of any model.

You of course need to judge whether the model is good enough to be released into production. There are a few checklist questions you can ask...

To read the rest of the blog, please click here!