TASK 3- Machine Learning Integration With DevOps (to select best Hyperparameter for dataset)

This is the most important task from the industry point of view because almost 60% of Machine Learning Projects are never implemented.

1. Create a container image that has Python3 and Keras or NumPy installed using dockerfile 2. When we launch this image, it should automatically start to train the model in the container.

3. Create a job chain of job1, job2, job3, job4, and job5 using the build pipeline plugin in Jenkins

4. Job1: Pull the Github repo automatically when some developers push the repo to Github.

5. Job2: By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing).

6. Job3: In this first, it checks the accuracy then take action accordingly if accuracy is greater than expected then it send mail otherwise it launch the container for tweaking

8. Job4: send mail if expected accuracy achieved

Solution -: I am using NN architecture for the training model using mnist dataset. I give solution in steps -

Step1: Create docker file that have python libraries installed

docker build -t pythonimg:v4  #creating docker image 

Step2: Pull the Github repo automatically when some developers push the repo to Github and copy the downloaded

Step3: By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing).

In this it first checks the code type and starts container accordingly. This is my code and for tweaking, I put a hidden layer in the loop and initialize i=0 and if the accuracy is less than expected then I value is incremented by 1 it is incremented till we don't get the expected accuracy i.e more than 90%

{{ nn.py}}

from keras.datasets import mnist

dataset=mnist.load_data('mymnist.db')

len(dataset)

train , test = dataset
X_train , y_train = train

X_train.shape

X_test, y_test = test

X_test.shape
img1_label=y_train[0]
img1_label
img1d=img1.reshape(28*28)
img1d.shape
X_train.shape
X_train_1d = X_train.reshape(-1 , 28*28)
X_train = X_train_1d.astype('float32')
y_train.shape
from keras.utils.np_utils import to_categorical
y_train_cat=to_categorical(y_train)
y_train_cat
from keras.models import Sequential
from keras.layers import Dense
model =Sequential()
model.add(Dense(units=512, input_dim = 28*28, activation= 'relu'))
model.add(Dense(units=256, activation='relu'))
i=0
for i in range(i):
    model.add(Dense(units=128,activation='relu'))
    
model.add(Dense(units=10, activation='softmax'))
model.summary()
from keras.optimizers import RMSprop
model.compile(optimizer= RMSprop(),loss='categorical_crossentropy', metrics=['accuracy'])
h = model.fit(X_train, y_train_cat,epochs=8)
X_test_1d=X_test.reshape(-1, 28*28)
X_test= X_train_1d.astype('float32')
y_test_cat=to_categorical(y_test)
model.predict(X_test)
p=h.history['accuracy']
h.history['accuracy'][7]
with open('file.txt', 'w') as f:
    f.write(str(p[7]))
model.save('mymodel.h1')

and in this code i write code that will save the accuracy in file.txt

Step4: In this first, it checks the accuracy then take action accordingly if accuracy is greater than expected then it send mail otherwise it launch the container for tweaking

Step 5: send mail if expected accuracy achieved

Pipeline for the Jobs