Integration of Docker, Machine Learning, Terraform & AWS EC2

Summary of this project

In this project, the main idea was to run machine learning jobs on top of a docker container.

STEPS

1. Write a terraform script which will setup the infrastructure. For this i have used an AWS EC2 instance on top of which i will be running my docker container
2. to install docker, i have put the commands to install docker in the userdata for the terraform script
3. Now we have pulled a Centos latest image on top of which we will run our workloads
4. Inside the docker container, we first install the required libraries (SKLEARN, Pandas etc)
5. Since our code was in github, we also installed git on the container so that we can pull the latest code from github
6. Finally we executed our code on top of docker.

TECHNOLOGIES USED:

1. Terraform (Infrastructure as a code) - Terraform is an open-source infrastructure as code software tool created by HashiCorp. Users define and provide data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language, or optionally JSON.
2. AWS EC2 - Amazon Elastic Compute Cloud is a part of Amazon.com's cloud-computing platform, Amazon Web Services, that allows users to rent virtual computers on which to run their own computer applications.
3. Docker - Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels.
4. Machine Learning - Machine learning is the study of computer algorithms that improve automatically through experience and by the use of data.
5. Github - GitHub, Inc. is a provider of Internet hosting for software development and version control using Git. It offers the distributed version control and source code management functionality of Git, plus its own features

CODE FOR TERRAFORM

aws.tf

resource "aws_instance" "dockeros" {
    ami =                  lookup(var.image, var.selectami)
    instance_type =        "${var.instancetype}"
    security_groups =      "${var.SG_ids}"
    tags = {
        Name = "Dockeros1"
    }
    key_name =               "${var.keyname}"
    subnet_id =              "${var.subnetid}"


    connection {
        type =        "ssh"
        user =        "ec2-user"
        private_key =  file("/Users/ankit/Downloads/virginia.pem")
        host =        aws_instance.dockeros.public_ip
    }
    provisioner "remote-exec" {
        inline = [
            "sudo yum install docker -y",
            "sudo systemctl start docker",
            "sudo systemctl enable docker",
            "sudo docker pull centos:latest"
        ]
    }
}


output "Instance_ip" {
    value = aws_instance.dockeros.public_ip
}

providers.tf

provider "aws" {
    profile = "default"
    region = "us-east-1"

}

variables.tf

variable "SG_ids" {
    type = list
    default = ["sg-0cc9f3f371c74ed31"]
}


variable "selectami" {}


variable "image" {
    type = map
    default = {
        ami2 = "ami-0d5eff06f840b45e9"
        redhat = "ami-06178cf087598769c"
        ubuntu = "ami-0194c3e07668a7e36"
        windows = "ami-0ae15c1544cd06ac8" 
    }
}


variable "instancetype" {
    type = string
    default = "t2.large"
}


variable "subnetid" {
    type = string
    default = "subnet-cae95595"
}


variable "keyname" {
    type = string
    default = "virginia"

}

Execution of terraform

Now we ssh into our instance and further pull the centos image from dockerhub

Next, inside the container we install python3 & git

Now after python3 got installed, we will install the required libraries

Next part comes our machine learning code, the link to which can be found in my github repo (given below). We pull and execute the code

And we get our result

Thank you for reading this article.