MLOPS TASK-3 INTEGRATION OF MACHINE LEARNING WITH DEVOPS

??Task-3??

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile 

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins 

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

Now let's start the task soln

STEP-1

First we would make a Dockerfile in which we write down all the python library that we use to make a docker image that contains CNN environment ready. so in this, we can install all the library of CNN so when we launch docker with this particular image it will launch us a CNN env ready and also make one other simple Dockerfile in which we write only simple library of python if CNN not require than this image launch container for us.

Liberary are follow

after writing this Dockerfile then I build a image of this docker file with command

docker build -t ml .

this command will build the CNN env image for us it will take time and it also consume lots of space nearly it will take 4gb container image because in this we launch many library of python.we can see the ml image as follow:

after this, we are able to launch our docker with this container image command as follow

docker run -dit ml ml:latest

After completion of this docker container image we can start our task that is start from fetching of data from the github repository.

In the code we use tensorflow and keras library for CNN and we also use mnist dataset to train and test our model that will give us a better accuracy. that we need as we can automate it with devops so we automatically restart our process with devops to get high accuracy in advance part we can also change the parameters also dynamically with the help pf the devops.

my GitHub URL for MNIST code and dataset: https://github.com/Gaurav630314/mlops3

mow we can start our

Job- 1

in this, we fetch the GitHub repo automatically when the developer upload the code to the GitHub

in this, we provide the GitHub repo URL link where we have to download the code that the developer uploads to it after this

we can cop it to the certain folder as a storage.

Output of job 1

After completion of job-1

Job-2

By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

Output of job 2

our docker is also launched

Job-3

Train your model and predict accuracy or metrics.

Output of job3

in this we receive problem in fetch mnist dataset from the internet first time so one time pl try Manually then it will work it is not any problem this is only due to the link.

After completion of this we get accuracy that is about 99% so we don't have to need to resatrt our epoch and train our model again because it give us good accuracy that is 99%so there is no need to twick our program again.

Job-4

if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

Output

At last our pipeline is running

so as our accuracy is 99 we don't need to restart again and our delivery pipeline maintain our all jobs