Machine learning systems are not static. They need to be constantly monitored and evaluated after deployment to ensure that they are performing well and meeting the business objectives. How do you evaluate your Machine Learning system after deployment? In this article, we will discuss some of the key aspects and methods of evaluating a Machine Learning system in production.
The first step of evaluation is to define what metrics and goals you want to use to measure the performance and impact of your Machine Learning system. These metrics and goals should be aligned with the business problem and the value proposition of your system. For example, if your system is a recommendation engine, you might want to use metrics such as click-through rate, conversion rate, revenue, customer satisfaction, etc. You should also set realistic and achievable targets for these metrics and track them over time.
In my experience , i defined functional and non-functional or operational goals of a model in production. A fuctional metrics could be FLINQ score for machine Translation or BLUE score or something else depnding on what use case we are solving. The non-functional or operational metrics could be the SLA like latency,availability, throttling, autoscaling etc
Applied Scientist @ Amazon | Large Language Models, Recommender Systems, Gen AI, MLops
For most of the Machine Learning models there are two types of metric Algorithmic and Business metric. Algorithmic metric are metric which tells how good a machine learning model is as compared to baseline in an algorithmic sense. Example For classification accuracy. precision, recall, for regression MSE etc. Business metric are those which are relevant to business like revenue impact, CTR etc. Metrics can be measured 1. offline - measured on historical data (mostly Algorithmic) and 2. online metric - measure post model is deployed (Both Algorithmic and Business) . When designing a Machine learning model one has to define Algorithmic and Business metric. Once it is defined one can defined success criteria or goal based on these metrics.
The second step of evaluation is to collect and analyze data from your Machine Learning system and its users. You need to have a reliable and secure data pipeline that can capture and store relevant data from your system, such as inputs, outputs, predictions, feedback, errors, etc. You also need to have a data analysis framework that can help you visualize, explore, and understand the data and the patterns in it. You should look for any anomalies, outliers, biases, drifts, or changes in the data that might affect the performance or behavior of your system.
In my experience, I deployed a pre-trained image classification model for floods as an Amazon SageMaker real-time endpoint to make inference. I then fine-tuned the model by bringing in my custom data that is saved as inputs in a S3 bucket. I configured my deployment by specifying the instance type for training. I then provide a model name and choose a default output location for the S3 bucket and update the hyperparameters such as the epochs, batch size and learning rate. I inspect the training model artifacts and then I deploy the fine-tuned model by specifying the output data path to save the model artifact and deploy my custom model. I then use a jupyter notebook to make predictions on test data that the probability is e.g. not damaged.
If we have to do continuous evalution of a Model in production, the first step would be to log the functional and operational metrics like P99 or P95 latency or % availability of the model for inference. Additionally if you decide to use A-B testing or caranaries to evaluate model performance then we should build capability to collect and analye this data. I also built dashboards to track model history and performance for benchmarking and comparision.
Applied Scientist @ Amazon | Large Language Models, Recommender Systems, Gen AI, MLops
This is the most crucial part of developing a machine learning model. Designing of data pipeline needs considerable planning and understanding of data sources, evaluating data quality issues and understanding data distribution. There are different analysis tools which can help to quickly analyze data like quicksight, powerbi and querying tools like Athena and Redshift. Also one can connect to data in s3 or any database to sagemaker or notebook environment on Spark cluster to analyze the data. Missing data, data drift, understanding outliers are among the key checks one should keep in mind. Also data is closely related to business context, so one must seek business understanding of insights obtained during analysis.
The third step of evaluation is to test and validate your Machine Learning system against various scenarios and conditions. You need to have a testing strategy that can cover different aspects of your system, such as functionality, accuracy, robustness, reliability, scalability, security, etc. You should also have a validation process that can verify that your system meets the requirements and expectations of the stakeholders and the users. You should use different methods and tools to test and validate your system, such as unit tests, integration tests, end-to-end tests, A/B tests, user tests, etc.
A/B testing is required for model quality evaluation and there could be different strategies for this - like is the CPU/GPU configurations going to change during inference based on availability or not etc. Integration test cases should be comprehensive to protect against any feature loss in existing functionality. Proactive alarming for latency, availability, errors and failures would drastically improve model performance.
A/B testing is great if you have different variants of the model for comparing their performance. I would like to add Regression tests, for checking whether the model still performs well after changes or updates which happens all the time.
The fourth step of evaluation is to monitor and update your Machine Learning system regularly and proactively. You need to have a monitoring system that can alert you of any issues or problems with your system, such as failures, errors, downtimes, performance degradation, etc. You should also have an update mechanism that can help you improve and optimize your system, such as retraining, fine-tuning, debugging, fixing, etc. You should monitor and update your system based on the feedback and the data you collect and analyze.
In my experience monitoring and update is extremely important. As the data grows the complexity grows and data patterns evolve over a period of time. Keeping up with these changes to retain the performance and accuracy measures continuous monitoring and updating needed. Because of the complexity and size of the data detecting these changes manually is very hard. Hence I have been using anomaly detection along with data drift detection techniques to identify those unusual patterns and outliers in the data which is hard to be found manually. This helps with automated monitoring and taking actions as needed to improve and update.
The fifth step of evaluation is to learn and iterate from your Machine Learning system and its outcomes. You need to have a learning culture that can encourage you to reflect on your system and its results, learn from your successes and failures, and identify new opportunities and challenges. You should also have an iterative approach that can help you adapt and evolve your system based on the changing needs and expectations of the business and the users. You should learn and iterate from your system in a continuous and agile way.
In my experience , machine Learning models iterate regularly to even maintain its performance and SLAs, the reason being data drift and bias. Building data and model pipeline is key to make this interation seamless and cost effective. Define strategies to have regular training data feeds, performance evaluation using A/B testing , quality checks using regression and integration tests, customer feedbacks on quality , code piplines to upgrade software library dependencies.
In my experience the cost of inference in production is a key operational metrics. If planned and tracked properly it can save significant infrastruture cost of operating a ML service. A detoriation in cost would mean there is a need to optimize model performance or to bring in engineering practices like request batching , caching , parallelization etc.