Building Automated Machine Images

Hashicorp Packer is a tool for building machine images by automating the process of Virtual Machine image creation on Cloud Services or On-Premise environments. Packer works out of the box with support to build images for Amazon EC2, CloudStack, DigitalOcean, Docker, GCP, Azure, AWS, Virtual Box and more. It is done by declaring the state of the VM using Packer JSON or HCL2 (Hashicorp Configuration Language) templates.

Packer is generaly used for Golden Image Creation and fits perfectly well in an IaaC (Infrastructure as Code) initiative. Can also be added in a CI/CD pipeline under specific use cases.

I won't describe the installation steps because it is quite straightforward, but here is an official link to the Hashicorp Packer webiste for reference - https://learn.hashicorp.com/tutorials/packer/getting-started-install

For the purpose of this example, I decided to build an AWS AMI using a JSON template, from my point of view is a language by far much more mature than HCL2.

A Packer json template consists of the following blocks:

{
    // Variables can be used in a separate json file as I did
    "variables": {


    },
    "builders": [


    ],
    "provisioners": [


    ],
    "post-processors": [


  ]

}

Let's take a look at each block in deep:

Variables:

As I mentioned before, I decided to put them all in a separate json file (aws-ami-variables.json). They could be perfectly added in a variables section in the main Packer json file. This file contains all the variables used in the template:

{
        "aws_access_key": "XXXXXXXXXXXXXXXXXXXX",
        "aws_secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "aws_subnet_id":  "subnet-09b0256884253cb29",
        "aws_security_group": "sg-0369c1444201957e9",
        "aws_region": "us-east-1",
        "virtualization_type": "hvm",
        "root_device_type": "ebs",
        "owners": "XXXXXXXXXXXX"

}

aws_access_key and aws_secret_key, both could be set up in the aws credentials file when configuring the aws cli

aws_subnet_id, the AWS subnet where the instance will be created

aws_security_group, the AWS security group assigned to the AWS instance

aws_region, is the AWS region where the instance and the AMI will be created

virtualization_type, is the AWS virtualization type, it could be PV (paravirtual) or HVM (hardware virtual machine)

root_device_type, the root device volume type. I will be using Amazon EBS basically because they launch faster and use persistent storage

owners, the owner ID which the base ami that I will be using to start the instance was published by

Builders:

A builder defines the platform that will be used and it's configuration.

    "builders": [
        {
            "type": "amazon-ebs",
            "region": "{{user `aws_region`}}",
            "access_key": "{{user `aws_access_key`}}",
            "secret_key": "{{user `aws_secret_key`}}",
            "subnet_id": "{{user `aws_subnet_id`}}",
            "security_group_id": "{{user `aws_security_group`}}",
            "source_ami_filter": {
                "filters": {
                    "virtualization-type": "{{user `virtualization_type`}}",
                    "name": "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*",
                    "root-device-type": "{{user `root_device_type`}}"
                },
                "owners": ["{{user `owners`}}"],
                "most_recent": true
            },
            "ssh_username": "ubuntu",
            "instance_type": "t2.micro",
            "ami_name": "amazon-ami-test_{{timestamp}}",
            "ami_description": "AMI Generated by Packer",
            "tags": {
                "Builder": "ldipaola",
                "Role": "Ubuntu Test",
                "Team": "DevOps"
            }
        }

    

    ],

type, the builder type that we will be using, in my case, amazon-ebs

source_ami_filter, contain filters for public Amazon Machine Images that I can use to select the base image that I will be using for the instance creation. In my case, I decided to use an Ubuntu 20.04 server

instance_type, the AWS instance type, in my case a t2.micro is enough

ssh_username, ubuntu is the default user to authenticate to an Ubuntu Official instance

ami_name, the AMI name. I decided to add a timestamp variable because I am not forcing a deregistration of an existing AMI

ami_description, any significant description to be able to easily identify my AMI

tags, it is always a good practice to tag any AWS resource created, so I decided to add some 

The values for the following keys: region, acces_key, secret_key, subnet_id, security_group_id, virtualization_type, root-device-type and owners, are going to be be replaced by the ones declared in the variables json file

Provisioners:

Provision and perform actions on the instance/image using either a shell or any other configuration management platform like Puppet, Chef, Ansible, Saltstack.

    "provisioners": [
        {
            "type": "shell",
            "inline": ["sudo apt update -y && sudo apt upgrade -y"]
        },
        {
            "type": "shell",
            "scripts": "init.sh"
        },
        {
            "type": "shell",
            "inline": [
                "sudo apt update -y && sudo apt upgrade -y",
                "sudo hostnamectl set-hostname packer",
                "wget https://apt.puppetlabs.com/puppet6-release-focal.deb",
                "sudo dpkg -i puppet6-release-focal.deb",
                "sudo apt update -y",
                "sudo apt install puppet-agent -y",
                "echo 'Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin' | sudo tee -a /etc/sudoers.d/extra",
                "bash"
            ]
        }

    

    ],

I am using 3 shell types. The first one with an inline array which performs an apt update and an apt upgrade. The second one calls an script (init.sh) which installs docker-ce, and the third one installs a puppet serverless and the puppet agent.

By the way, the power of using Packer with a Configuration Management tool like Puppet, Chef, Ansible, Saltstack, etc. is huge!!!

init.sh file content:

#! /bin/bash


curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
sudo apt-get install docker-ce -y
sudo usermod -aG docker ubuntu

Post-Processors:

Can be used to generate and upload artifacts. In my case, I am creating an aws-ami.tar.gz to convert it to a AWS Vagrant Box.

    "post-processors": [
        {
            "type": "vagrant"
        },
        {
            "type": "compress",
            "output": "aws-ami.tar.gz"
        }

    
    ]

Putting All Togheter

Using all the above code snippets, the variables json file and the Packer template json file should look like this:

aws-ami-variables.json:

{
        "aws_access_key": "XXXXXXXXXXXXXXXXXXXX",
        "aws_secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "aws_subnet_id":  "subnet-09b0256884253cb29",
        "aws_security_group": "sg-0369c1444201957e9",
        "aws_region": "us-east-1",
        "virtualization_type": "hvm",
        "root_device_type": "ebs",
        "owners": "XXXXXXXXXXXX"

}

aws-ami.json:

{
	    "builders": [
	        {
	            "type": "amazon-ebs",
	            "region": "{{user `aws_region`}}",
	            "access_key": "{{user `aws_access_key`}}",
	            "secret_key": "{{user `aws_secret_key`}}",
	            "subnet_id": "{{user `aws_subnet_id`}}",
	            "security_group_id": "{{user `aws_security_group`}}",
	            "source_ami_filter": {
	                "filters": {
	                    "virtualization-type": "{{user `virtualization_type`}}",
	                    "name": "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*",
	                    "root-device-type": "{{user `root_device_type`}}"
	                },
	                "owners": ["{{user `owners`}}"],
	                "most_recent": true
	            },
	            "ssh_username": "ubuntu",
	            "instance_type": "{{user `instance_type`}}",
	            "ami_name": "amazon-ami-test_{{timestamp}}",
	            "ami_description": "AMI Generated by Packer",
	            "tags": {
	                "Builder": "ldipaola",
	                "Role": "Ubuntu Test",
	                "Team": "DevOps"
	            }
	        }
	    ],
	    "provisioners": [
	        {
	            "type": "shell",
	            "inline": ["sudo apt update -y && sudo apt upgrade -y"]
	        },
	        {
	            "type": "shell",
	            "scripts": "init.sh"
	        },
	        {
	            "type": "shell",
	            "inline": [
	                "sudo apt update -y && sudo apt upgrade -y",
	                "sudo hostnamectl set-hostname packer",
	                "wget https://apt.puppetlabs.com/puppet6-release-focal.deb",
	                "sudo dpkg -i puppet6-release-focal.deb",
	                "sudo apt update -y",
	                "sudo apt install puppet-agent -y",
	                "echo 'Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin' | sudo tee -a /etc/sudoers.d/extra",
	                "bash"
	            ]
	        }
	    ],
	    "post-processors": [
	        {
	            "type": "vagrant"
	        },
	        {
	            "type": "compress",
	            "output": "aws-ami.tar.gz"
	        }
	    ]

	
 }

It is now time to validate and build the AWS AMI with Packer:

packer validate aws-ami.json
packer build -var-file=aws-ami-variables.json aws-ami.json

It may take some minutes for the packer build process to finish. Once it is done, the output should be similar to the one shown in below picture:

All the code is in a GitHub Repository:

SSH - [email protected]:ldipaolaIT/packer-aws-ami.git

HTTPS - https://github.com/ldipaolaIT/packer-aws-ami

Feel free to introduce changes, fork it, clone it or simple use it as a reference if you are just starting with Packer.

Enjoy and many thanks for reading!