Automation of Machine Learning model |Git | Jenkins | Docker

In this task, automation of machine learning model is achieved. We automated the task of tuning the model to achieve high accuracy. First let's look at the Machine Learning & Deep Learning programs provided to us by the client. We will be providing the codes one by one First for the machine learning model and its dataset then the same for the deep learning model. Let say these codes and dataset were pushed to the GitHub by the developer. You can find all the codes and dataset on my GitHub repository. https://github.com/khushi20218/projecttask2

Let's start by creating our first job which would pull the codes and dataset from the Github. My job would be using the method of post-commit files which would automatically push codes once you commit them, you could also use the method of webhooks as well.

This job copies the code as well as dataset files to the client folder. Now we would create 2 dockerfiles for creating 2 images, one for machine learning code whereas the other for the deep learning code.

This is a Dockerfile created for the image used to launch the container used to train the deep learning model and would require the following libraries for python.

And similarly this Dockerfile is for the image for the machine learning model.

So after building the images from these Dockerfile, jenkins will look at 1 code named “client.py” and check if it is a Machine Learning code or a Deep Learning code and accordingly launch a container from any 1 of the 2 images. This is done by our job2.

At first we provide a deep learning code in the client.py file. The console output shows that the file “client.py” according to Jenkins is a Deep Learning file, so it launched a container from the dlimage:v1 image.Then created an “accuracy.txt” file and also save the model with the name "ann.h5”.

To verify whether all the files were created in the client folder , we go tpo our RHEL and check and find all the folders present. Also now we create a python file ie. tweeking.py which would perform the next task.

Our next task is checking whether the accuracy is less than the required accuracy. In the code for machine learning I used various techniques like Feature Engineering which involves imputation, creating Dummy Variables. We can also do Feature Scaling to make the predictions more accurate.

For tweeking a DL code new layers can be added , number of epochs can be increased etcetra . For the DL model if the accuracy appears to be less than the required ie. 80% then we tweek the code to achieve the required accuracy. We achieve this by creating the python code which adds the CRP layers to our model.

This is the code written in tweeking python file:

import os


acc = os.popen("cat /client/accuracy.txt")
acc1 = acc.read()
print(acc1)
acc2 = acc1.rstrip()
print(acc2)
acc3 = float(acc2)




if acc3<85:
    x = os.popen("cat /client/train.py | grep model.add | wc -l")
    x1 = x.read()
    x2 = x1.rstrip()
    x3 = int(x2)
    print(x3)
    if x3==2:
        y = 'model.add(Dense(units=32, activation=\"relu\"))'
    elif x3==3:
        y = 'model.add(Dense(units=16, activation=\"relu\"))'
    elif x3==4:
        y = 'model.add(Dense(units=8, activation=\"relu\"))'
    else:
        print("ACCURACY ABOVE 85!!")
        exit()
    os.system("sed -i '/softmax/ i {}' /code/train.py".format(y))
    os.system("curl -u admin:khushi https://192.168.99.101:8080/job/mljob1/build?token=daksh")
    acc = os.popen("cat /client/accuracy.txt")
    acc1 = acc.read()
    print(acc1)
    acc2 = acc1.rstrip()
    print(acc2)
    acc3 = float(acc2)
else:
    print("ACCURACY ABOVE 85")

Now our job3 runs whose task is to run this python code. Also we set our downstream project as job2 so that when the tweeking is done it checks the accuracy again and keeps tweeking till we reach the required accuracy.

Since our accuracy is less than 85% it triggers our job2 and again after tweeking finds the accuracy.

Our accuracy reaches 88% !!!! Our task is done .

The email notification has been sent !!

Do leave your valuable feedbacks . For any queries or correction feel free to contact