Machine Learning model with integration with Jenkins

In the era of AI, we require machine learning projects to be implemented to machines for more rapid growth in this AI world but we have known with the fact that 90% of the Machine Learning models never make it to the production. There are numerous obstacles in the path. One such obstacle is that the data science guys and the IT guys are not getting an opportunity to work together.

A solution to the above problem: MLOPS (A combination of ML and DevOps) that gives both the IT team as well as the Data Science team to work together in order to push the ML model into the production. With the help of this latest technology, both the teams could work together, deploy the model, monitor it, and manage the model in production. 

So, here is a task which was given by Vimal Daga sir in which I have tried to integrate ML with DevOps. I have used some of the most demanding technologies in this project such as Git, Github, Jenkins, Docker, and Machine Learning. Finally, I have created a delivery pipeline by using the build pipeline plugin in Jenkins that automates the production of ML model without the interference of any human being. 

The task was basically about automating the process of increasing accuracy of a CNN model by adding layers and epochs.

Detailed Description of the task :

1. Create a container image that has Python3 and Keras or numpy installed using dockerfile.

2. When we launch this image, it should automatically start training the model in the container.

3. Create a job chain of job1, job2, job3, job4,job5 and job6 using build pipeline plugin in Jenkins 

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter inside image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the CNN processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 95% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created.

9. Create One extra job job6 for monitor : If container where app is running, fails due to any reason then this job should automatically start the container again.

Concepts Required to accomplish the task:

• Machine Learning: Creating a CNN model
• Creating customized docker image using Dockerfile
• Pushing code to Github
• Configuring Jenkins Jobs
• Writing script of email notification
• Python Code to tweak the model if its accuracy is less than desired accuracy

Process followed by me and job created by me:

step 1:

The first step is to create a DockerFile that installs all the required libraries of python and builds an image from the file.

After successfully creating the Dockerfile, it is the time to create a docker image using this Dockerfile. The code to do so is:

And hence, the image has been successfully created. 

Now let us move towards the Jobs in Jenkins.

JOBS in Jenkins:

• The first job is to pull the code from GitHub. For automatic pulling of code from GitHub, I have used Poll SCM trigger in the first job of Jenkins and then the pulled code will be copied to the root directory of the Redhat (whichever OS/environment you are using).

JOB1:

Now, this Job will pull the code from the GitHub repository and then copy all the files from the workspace of Jenkins to the root directory of my RedHat8. 

• Now, here comes the Job2 where a docker container is launched and the copied python code is run. Then, the accuracy of the model is saved in a file named accuracy.txt. 

JOB2:

job2 will run only when job 1 is successfull.

This is the code in Job2 that will launch the container and run the python code in order to create a model. 

• The Machine Learning code that is pulled from the GitHub is:

I have used the concept of the loop in which changing the value of “filt” variable will change the number of filters and changing the value of “parameter” variable will increase the number of layers in the model. 

git hub link for my ml code :

• The console output of the second Job is:

The accuracy has been retrieved from the model and has been stored in a text file in the RedHat8. 

• Now let us move towards the third job. This is the most important part where the accuracy of the model will be compared with some value and if this accuracy is less than that expected value, then the model will change the hyperparameters (such as filters, the number of layers, etc.) itself until and unless the accuracy matches the expected one. 

• In Job 3, I have used the “sed” concept of Linux using which I have done changes in the python code in which the model is being trained. 

JOB3:

my accuracy output will store in show.html file of redhat8 and from there it will be copied to my html web page folder to deploy the output in browser.

JOB 4:

as the job 3 will build then only this job will run.

This job will check whether the accuracy of the model is 95% or not. And if it is less than the job will do necessary changes (increasing the number of layers and the number of filters) in the code. The value of the parameter variable keeps on increasing and every time the loop runs, the number of layers will be doubled and the number of filters will also be increased than the previous one. 

Console output of job4

JOB 5:

mail will be sent to developer as soon as the desired accuracy will be obtained.

content of email which in my file accuracy_mail.y it is having extension py so that in that python code is there

Email received by developer:

JOB 6:

Monitoring the container

• Hence, all the jobs have been completed. And here comes the role of the fifth Job that will keep monitoring the container in which the Machine Learning code is running. If the container is running properly, there is no problem at all!!
• However, at any moment, if the container fails, Job 6 will restart the container and on the other hand, it will also send an email to the authority regarding the issue.

log obtained for JOB 6:

showing that container is working in good condition.

For testing purpose, I manually stopped the container to check whether the second part of the code is working or not. It worked!!

The container was restarted by Jenkins and the received email is:

ALL MY JOB WHICH I HAVE CREATED AND THEIR WORKING CONDITION:

Overall summary of my model is been shown by build pipeline:

MY project CI/CD Pipeline

my github url:

https://github.com/manmohan2823/mL_devops_project.git