Machine Learning Integration with Devops

I m going to demonstrate the power of the Automation DevOps integration with Machine Learning .

This task is about automatically build the model of Machine Learning by checking the accuracy of the model if accuracy is less then 95% it changes in the source code and increase layers , epoches and neurons to increase the model accuracy.

Let's check the task :

• Create container image that’s has Python3 and Keras or Numpy installed using dockerfile
• When we launch this image, it should automatically starts train the model in the container.
• Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins
• Job1 : Pull the Github repository automatically when some developers push the source code to repository of Github. By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( e.g. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).
• Job2 :Train your model
• Job3 : Predict accuracy or metrics. If metrics accuracy is less than 80%
• Job4 : Then tweak the machine learning model architecture.
• Job5: Retrain the model or notify that the best model is being created

Let’s start the task :

Here creating a Docker File for image

Build This Docker file using CMD

           $ docker build -f nn:1.0 .

similarly Build one more file with name ann:1.0 for sklearn and build the required packages.

Now from here the Things Starts .

for creating the webhook in this github so that whenever the changes occur it will trigger to jenkins and then jenkins will pull so first i used the ngrok to create the webhook.

Job1 : In this i have used the github link to push the data automatically by webhooks and then made this data copy to a folder in base os ie /root/mlauto/ml/ . this job will check whether our os is running or not if not it will run and if already running it is not running start new.

Job2 : This job will run after the succesful build of job1 so this will run after the run of os in this we are executing the needed program python program start running and this will give us the accuracy and store the accuracy in a folder.

Job3 : This job will Run after the sucessful build of job2 and if not then it will notify Model Accuracy to the mail of admin . in this job3 it will check the accuracy an if accuracy is less than 95% it will trigger to job5(tweek) other wise if accuracy is greater than 95% then it will trigger to job4 and it will notify the success full build to the Mail also.

Job 4 : This job rus after the sucessfull build of the job 3 this will send the sucessfull build notification to the mail of the admin.

Job 5 (tweek) : This job will run of triggered by the job3 when the accuracy needs to be improve so this will increase the epoch and also the number of layers and start the job 2 again to train model again by the changes.

we can also denote this all building with visual pipelines i.e. By build pipeline. It will also show us the dynamically completion of task and jobs .

Thanks for giving precious time and attention.

• Github : https://github.com/NikhilSuryawanshi/task3_auto_ml.git
• Email : [email protected]