All the ways to deploy an ML model

You should learn to deploy your Machine Learning models! The way to deploy is dictated by the business requirements. You should not start any ML development before you know how you are going to deploy the resulting model. There are 4 main ways to deploy ML models:

• Batch deployment - The predictions are computed at defined frequencies (for example, on a daily basis), and the resulting predictions are stored in a database and can easily be retrieved when needed. However, we cannot use more recent data, and the predictions can very quickly become outdated. Look at this article on how AirBnB progressively moved from batch to real-time deployments: “Machine Learning-Powered Search Ranking of Airbnb Experiences“. ?

• Real-time - the "real-time" label describes the synchronous process where a user requests a prediction, and the request is pushed to a backend service through HTTP API calls that, in turn, will push it to an ML service. It is great if you need personalized predictions that utilize recent contextual information, such as the time of the day or recent searches by the user. The problem is that until the user receives its prediction, the backend and the ML services are stuck waiting for the prediction to come back. To handle additional parallel requests from other users, you need to count on multi-threaded processes and vertical scaling by adding additional servers. Here are simple tutorials on real-time deployments in Flask and Django: “How to Easily Deploy Machine Learning Models Using Flask“, “Machine Learning with Django“.
• Streaming deployment - This allows for a more asynchronous process. An event can trigger the start of the inference process. For example, as soon as you get on the Facebook page, the ads ranking process can be triggered, and by the time you scroll, the ad will be ready to be presented. The process is queued in a message broker such as Kafka, and the ML model handles the request when it is ready. This frees up the backend service and saves a lot of computation power by an efficient queueing process. The resulting predictions can also be queued and consumed by backend services when needed. Here is a tutorial in Kafka: “A Streaming ML Model Deployment“.

• Edge deployment - This is when the model is directly deployed on the client, such as the web browser, a mobile phone, or IoT products. This results in the fastest inferences and can also be predicted offline (disconnected from the internet), but the models usually need to be pretty small to fit on smaller hardware. For example, here is a tutorial on deploying YOLO on IOS: “How To Build a YOLOv5 Object Detection App on iOS“. ? ?