How companies deploy machine learning models to production today

The real transformational benefits of machine learning and data science can only be realized when models are constantly being used in a production environment. Even more value gets generated when there is a feedback loop from production data that is used to retrain a model at high frequency (some companies do daily retrains). However, let's stay focused on just getting models in production.

At Orchestra, we are obsessed with understanding how machine learning can be more widely adopted and deployed into production environments. Our experience has been that it takes an incredibly long time from selecting the right model to having the model embedded in production environments. We have since embarked on a journey to understand some of the challenges that companies faced.

Numerous ways to companies need to deploy machine learning

Having spoken to 5 data scientists at various companies around the world, we've since discovered that there are numerous ways that companies can/need to deploy their machine learning models which are dictated my organizational constraints. We will continue our research and once we've finalized it, we will share our findings here.

Most common way to deploy a model is as an API

No surprise that the most common way to deploy machine learning is to expose the model as an API service. APIs can easily be connected to production applications given that there is an engineering need for it.

Docker and Flask is the most common technology stack for Python models

With the data scientists we spoke with, the majority of them developed their models in Python and served it as an API service using Docker and Flask. At Orchestra, we've also adopted containerization technology as our means for serving our API service as it provides an isolated environment for models to run, handles model dependencies well and provides a layer of security and risk management away from the rest of the engineering stack.

Close collaboration with software engineering is key

In all cases, close collaboration with the data science and software engineering is key to managing model deployment. The infrastructure could be own by either team depending on resource availability but both teams rely on each other heavily, especially if the core value proposition is an AI-driven "something".

Not all companies are ready to deploy although they are thinking about it

Of the companies we spoke with, there were a few who are actually not ready to deploy their models in a production environment. However, this is due to the sheer complexity of their production environment with legacy systems and inability to use cloud environments.

Orchestra is a managed cloud-based machine learning model deployment platform that lets data scientists serve their models as APIs within minutes using 3 lines of code. We save your data scientists and data engineers time and headaches from managing infrastructure, enabling them to be more productive.