Integration of Machine Learning and Devops

Integration of Machine Learning and Devops

What is MLOps?

ML and DevOps when together used forms a powerful combination which is called MLOps.

Task description

  1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile
  2. When we launch this image, it should automatically starts train the model in the container.
  3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins
  4. Job1 : Pull the Github repo automatically when some developers push repo to Github.
  5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).
  6. Job3 : Train your model and predict accuracy or metrics.
  7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.
  8. Job5: Retrain the model or notify that the best model is being created
  9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

Tools used to Create-

No alt text provided for this image

1. Github- for hosting our repository.

2.Jenkins-is use to automate various jobs.

3. Redhat8-use as a base OS for running the services like httpd,ngrok,jenkins.

4.Docker- to run container image in python model.


Creating a dockerfile :

No alt text provided for this image


In RHEL8 fisrt make a directory that will store all the data or the program for our machine learning model.

mkdir Code

Now the jenkins will automatically copy the files in this folder. Download a centos:7 image in docker using:

*docker pull centos:7 docker run -it --name os centos:7

Now install al the requirements for the Machine learning model. After the requirements are fulfilled use commit command to make your own image and we will use this image in our Dockerfile.

*docker commit os myimage:v2

Now type:

*vim Dockerfile

And write the following code:

FROM: myimage:v2

RUN mkdir /root/my_model

VOLUME /root/my_model

COPY ./Code/. ./root/my_model/

WORKDIR /root/my_model CMD ["python3","code_file2.py"]

Save this docker file and now build two different images for different environments:

docker build -t deep:v1 /root

docker build -t neural_net:v1 /root

Create job1-

Job1 : Pull the Github repo automatically when some developers push repo to Github. job1 Congiguration.

1)Enter a Job Name, select “Freestyle project” and hit “OK” button.

2)Source Code Management: The section contents the source code options select GIT.

3)Build Triggers: The section contents trigger settings that trigger the build based on the specific condition match.


No alt text provided for this image

4)Build: The section contents the build steps that can be performed by adding Batch or shell command.

5) Post-build Actions: The section contents the build steps that can be performed after the build action done.

No alt text provided for this image

Creating job 2-

Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).Run docker container from docker image-

No alt text provided for this image

code see checking.py from here -checking.py

No alt text provided for this image

Configure Email notification in jenkins :

Click the ‘Manage Jenkins’ menu option displayed at the right side of the screen. You will be redirected to the ‘Manage Jenkins’ page, where you need to select the ‘Manage Plugin’ option.

Click the ‘Available’ tab then Manage Plugin page.

Start typing ‘Notification’ in the ‘Filter’ as ‘Manage Plugin’ page. Click the checkbox next to the ‘Email-ext plugin’ option. Click the ‘Install without restart’ button.

Now, click the checkbox next to the ‘Email-ext Template Plugin’ option. Click the ‘Install without restart’ button.

Go to the Jenkins home page and click the ‘Manage Jenkins’ menu option. Then, select the ‘Configure System’ option.

Enter the SMTP server name under ‘Email Notification’. Click the ‘Advanced’ button and then click the checkbox next to the ‘Use SMTP Authentication’ option. Now, set the following fields.

SMTP server name : smtp.gmail.com

User name: [email protected]

Password: abcd

Use SSL : Checked

SMTP Port: 456

Creating Job3-

JOB3 has a task to copy the [show_output.html] file from Parent Directory to /var/www/html which is the default directory for webpages in Apache Webserver.

No alt text provided for this image

Configuration of job 3

Systemctl start httpd

No alt text provided for this image

Creating Job 4-

Job4 : Train your model and predict accuracy or metrics.

Job5 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

No alt text provided for this image

Creating JOB5 in Jenkins-

  1. After getting the desired accuracy it will run. This job will run model_success_mail.py and send the notification of project completion to developer.
No alt text provided for this image
No alt text provided for this image


This project is yet not complete. It still need much of enhancement and needed to be more dynamic. I will try to make it more enhanced and dynamic.I will show in github link.

Thanks Vimal Daga sir for sharing us knowledge.

Meera Agrawal

Student at Jaipur Engineering College and Research Centre, Jaipur, Rajasthan

4 年

Good job vilsi..????

Prashant Saini

Lead LlmOps Engineer at Zeblok Computational Inc.

4 年

nice project

Yogya Chhatwani

React native developer at Metacube Software Pvt Ltd

4 年

Great work vilsi

Yashika Khandelwal

Assosciate Software Engineer at Accenture

4 年

Great work

Abhishek Chouhan

DevOps Engineer at Toorak Capital

4 年

Great work Vilsi Jain

要查看或添加评论,请登录

Vilsi Jain的更多文章

  • Task 3 Flutter Mobile App

    Task 3 Flutter Mobile App

    Linux and data store in firebase Linux Command store in Firebase 1. Create an app that can run any linux command using…

  • Animation in Mobile App

    Animation in Mobile App

    What is Flutter? Flutter is a free and open-source mobile UI framework . It allows you to create a native mobile…

    3 条评论
  • Face Recognition through Transfer Learning

    Face Recognition through Transfer Learning

    Face Recognition using transfer learning this model recognize the face through powerful pre- trained model Problem…

    3 条评论
  • MLOps Task-1

    MLOps Task-1

    Integration of Jenkins,Git,Docker Tools used to make- 1)Jenkins 2)Git 3)Github 4)Docker TASK Description- JOB -1 If…

  • Task 2-Devops Training

    Task 2-Devops Training

    Objective: to create container image that’s has Jenkins installed using dockerfile .As we launch the…

    4 条评论

社区洞察

其他会员也浏览了