Machine Learning- ML workflow explained

If you are starting out in Machine Learning(ML) just like me, you will agree with me that machine learning workflow is the best starting point before trying to solve an ML problem. ML is a powerful tool that allows computers to learn from data without being explicitly programmed. As a result, it has revolutionized the way we interact with technology, from pricing predictions for sales to nutritional value predictions and medical appointment predictions. However, building an effective ML model can be a challenging task, requiring careful consideration at each step of the process.

In this article, I will try to break down and briefly explain the five key steps of the ML Workflow, from research to deployment. By following these steps, you can create an efficient and effective ML workflow that maximizes your model's value contribution. Whether you're building with resources on-premises or outsourced in the cloud, this article is a great starting point for an ML project.

The first step is to research existing literature. This step involves reading up on existing approaches, datasets, and resources. It helps inform design choices for ML algorithms, datasets, cloud services, and more. The literature review can help identify available resources to reduce workload.

The second step is Data Gathering and Processing. This step involves getting data and processing it into a state ready for the ML model. The data's size, compute power, available resources, data cleanliness, data source, and more are factors that influence this step. Cloud platforms such as GCP can help mitigate resource constraints.

The third step is Model Training, Experimentation, and Evaluation. This step involves training the ML model using the processed data. The model is then evaluated based on a specific set of metrics, such as accuracy or AUC-ROC. Hyperparameters and feature engineering are key components that influence model performance.

The fourth step is Deployment. This step involves deploying the model in a production environment. It involves containerizing the model using tools like Docker and deploying it using services like Vertex AI API endpoints for inferencing. Monitoring and maintenance are critical to ensuring the model remains effective.

Finally, to maximize the ML model's value contribution, the approach to each step in the workflow must consider resource constraints such as time, people, storage, and computing. This can involve optimizing queries, using cloud platforms like BigQuery ML, Vertex AI, and AutoML, or finding pre-trained models that you can directly use for your needs. Careful consideration of available resources will result in an efficient and effective ML workflow.

By following these five steps, you can create a successful ML workflow that maximizes your model's value contribution. Whether it's a small or large ML project, this workflow is a great starting point for your next ML project.

Thank you for taking the time to read my article on machine learning workflow. As I continue to learn and grow in this field, I am excited to publish more content sharing my experiences and insights.

If you or your organization have any opportunities related to machine learning, please do not hesitate to reach out. I am actively seeking junior-level roles, internships, and apprenticeships where I can contribute to data cleaning, data preparation, modeling, and regression tasks using tools such as Vertex AI platform, BigQueryML, AutoML, and with a little bit of Tensorflow and sci-kit learn.

If you would like to read this as a short Twitter thread, check out my ML Workflow tweets