MLOps: Integration of Machine Learning with DevOps

This article is about how we can integrate machine learning with the concept of devops to train the machine learning models to achieve the required accuracy. This is probably the solution for the problem faced during the training of the model. So many of the projects does not come into existence in real life due to this problem. Here the solution is that we have to give the threshold of the accuracy we want & rest is done by the machine itself , you don't need to worry. It will also send you mail when the accuracy is achieved.

Let's get started with the process of achieving such a great automation !

Problem Statement Overview:

1.Create container image that has python3, keras, numpy etc installed using dockerfile required that are reqquired for a cnn model .

2.When we launch this image, it should automatically starts train the model in the container.

3.Create a job chain of job1, job2, job3, job4, job5 & job6 using the build pipeline plugin in jenkins.

4. Job1: Pull the Github repo automatically when some developers push the repo to github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created.

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.

Solution steps:

Step1: Create a dockerfile that will install all the libraries for the CNN model.

Here i have created a directory in my root folder using mkdir & create a dockerfile there

Now use the docker build command to create the image like sudo docker build -t dl:v1 /root/mlops/ .

Step2: Create a job thatep will fetch the files from the github when developer make some changes in it & save it in a folder.

Step3: Job2 - This will do the code detection. i.e if the code is of cnn then it will launch cnn environment or else sklearn environment. Also it would train the model as soon as it run the container.

Step4: Job3 This job will check the accuracy of the model. If the model accuracy is <90% it would trigger the job 4(tweak) else it would trigger job 5(mail).

Step5: Job4 This will tweak the model and again retrain the model to get the better accuracy.

NOTE: The code of my model and the tweak code is given in my github link. Do follow me there!

Step6: Job5 This job will send the mail to the developer that the model accuracy is good.

Step7: Job6 This will keep on monitoring the system. If the Job2 is failed due to any reason, the container is terminated so this will trigger the job2 to again build environment.

My Jobs pipeline view :

Github link :

Thank you everyone for reading...