MLOps: Integrating Deep Learning with Computer Vision and DevOps

MLOps mini-project to integrate Machine Learning/Deep Learning + Jenkins + Docker

Task Description:

1. Create a container image that has Python3, Keras, NumPy, etc libraries installed using dockerfile 

2. When we launch this image, it should automatically start to train the model in the container.

3. Create a job chain of job1, job2, job3 and job4 using build pipeline plugin in Jenkins 

4. Job1: Pull the Github repo automatically when some developers push the repo to Github.

5. Job2: By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing). Then Train your model and predict accuracy or metrics.

7. Job3: if metrics accuracy is less than 80%, then tweak the machine learning model architecture.

8. Job4: Retrain the model and notify that the best model is being created

Technologies Used: Deep Learning, Computer Vision, Linux, Python, Docker, Jenkins

First Step: First we have to create a Dockerfile to build a container that will have all the required libraries like TensorFlow, NumPy, pandas, etc.

We can build this Dockerfile using the following command.

$ sudo docker build -t mlproject:v1 /path

This command will create a docker image name mlproject:v1

:- For this, I'll be using my MNIST DataSet.

In this particular code we are using only one epoch so that our accuracy will be low and some hidden layers are also missing to that we can change it later on using our automation.

Second Step: We will create 4 jobs in our Jenkins to automate the entire process.

As soon as the Developer commits his/her code .git/hooks/post-commit will be triggered which will push our code to GitHub repository and then the next curl command will trigger our build remotely i.e. job1 will be triggered inside our Jenkins as Jenkins will know that something has been pushed to Github repo.

Job1: job1 will pull all the code from Github repo and copy all that code to the folder /root/mlopsproject

After Job1 completes job2 will be automatically triggered.

Job2: Job2 will check whether the code is of CNN, sklearn or some NN_with .csv code once it checks the code then it will automatically run the particular container and as soon as the container starts then in the container our model will be automatically trained.

Explanation: Here we can see a little example of developer and operations guy working together as the developer will put some code_word in his programFile.py which will describe for what the code is ex: CNN, sklearn, etc. this way it will be a lot easy to check

which container to run which processes to execute etc.

After job2 completes our model is trained according to the code created by the developer. Here we are getting an accuracy of around 95% so we have to important.

This accuracy will be stored in a file named prediction.txt

Job3: job3 will trigger immediately after job2 gets a stable build. Job3 will go and check the accuracy inside our prediction.txt file and if the accuracy is less than 98% then it will do changes in our model(programFile.txt).

Here we are using stream editor(sed Unix-like operating system command) to make changes in our model(programFile.txt)

Job4: job4 will we triggered after job3 completes and job4 will go inside my workplace and it will push all the code that is changed back to the GitHub repo.

once the code is pushed to the Github repo with changes then we use build trigger to again trigger job1 which will repeat the same process again will run job1, job2 and job3 again.

After we found the best accuracy then it will exit the job and a mail will be stored inside the location /var/spool/mail/root because we are using here "mailx" library to mail the outcome

NOTE: We can also use email plugin inside jenkins

Finally our model with best accuracy is be trained We can see the build pipeline

After running all 4 jobs first time when the second time after changing the model when it reaches to job3 and find that accuracy is greater than 97% then it will give exit code 1 which will mark the build as failure and hence stop the jobs.