Container versioning within the same host

What is versioning?

Software versioning is the process of assigning either unique version names or unique version numbers to unique states of computer software. There are many ways to name a version. It could be a word, a sentence, a number, a timestamp, or even a mix of all of these. Recently many ecosystems have opted to use semantic versioning. Semantic versioning allows to specify not only a specific version in itself, but also rules and policies that describe ranges of appropriate versions. Which, in turn, leads well to automatic updates, bug fixes, and other wonderful time-saving benefits for developers.?

Why version?

We know it is important to version our software, but why is it important?

To answer this question let’s begin at the end.

At the end we need our software to run and to do what it is built for.

In order for it to do what it is built for we need it to be in, what we call, a good state.?

In order to know what a good state is, we need to be able to tell apart the changes from one state to the next.

That’s where the versioning comes in.?

We assign a version, which is just a type of a name, after every set of changes we apply to our software.?

Versioning allows us to evolve our software forwards, as well as know and communicate exactly what changes were applied and when. What is even more important, versioning allows us to move our software backwards in time. This becomes especially critical when we find out that the recent changes to our software do not behave as expected. In this situation we can revert to the last known good state and recover, meaning - return our software to the working order. This is why it is also important that the versions, once created, remain immutable. This immutability guarantees a consistent transition to the known state. Or, in case of a recovery, to the last known good state.

Why run on the same host?

Navigating versions forwards and backwards is a solid approach to keeping a good state, however installing a version takes time. Since we need our software to run and do what it is built for, taking our software offline in order to switch versions results in downtime, which is less than optimal. That’s why some use a strategy known as blue/green deployments. Blue/green deployments minimize the downtime, and in some cases are unavoidable, but they are costly. This strategy requires doubling the resources, and thus doubling the cost. A better approach is to architect software in such a way where different versions of software run in parallel using the same resources. Running multiple versions of the same software in parallel allows not only for a nearly seamless full switch from one state to another, but also opens up a door to such features like:

1. Backward compatibility. In cases where breaking changes may be required to move forward.
2. A/B testing. Where a study may be needed in what new features would be more appropriate to customers.
3. Canary releases. Where a new feature needs to be tested on a smaller group of customers.
4. Benchmarking. Where an apples to apples comparison is needed to verify changes in behavior.
5. Sustainability. This one is a no-brainer. Running versions in parallel reduces the required hardware footprint, which in turn reduces energy requirements and cost.

How to select the desired version on the same host?

Containers in themselves may run multiple types of workloads:

• Single instance, single version. These are usually standalone apps and may have only one instance across the whole ecosystem, i.e. a scheduler. We never want to run more than one instance of a scheduler, since we’ll end up spinning up the same task at the same time more than once. This could lead to all kinds of interesting outcomes. This one is the simplest of them all, and this case is all about the naming convention. We tag the image version as current, and run with image container-name:current and the name container-name-current. Since names are unique and the name is pointing to the current tag we are guaranteed that only one instance will be executed within the container environment.
• Single instance, multiple versions. These are usually TCP/UDP daemons and may have multiple versions, but only one of the versions may be used at a time, i.e. a reverse proxy. It is impossible to run multiple processes on the same port, but we still want to be able to switch from one version of a reverse proxy to another. This one requires a bit of port mapping magic. Specifically we’ll be telling iptables on the container host to redirect traffic from the requested port to the port where the service is actually listening. Below are the bash functions that can help us with this. When we use container-activate container-name-1.0.0, this function will inspect container named container-name-1.0.0 for ports it requires, and then will assign the published ports to the actual ports used by the container. container-deactivate container-name-1.0.0 will unassign the host ports. At this point we should be able to run different versions of the same service and access all of them by their individual ports, while allowing our external clients access only to the version we want them to reach.

function iptables-assign-port() {(
    set -e

    local srcPort=$1
    local dstPort=$2
    local portType=$3

    if [ -z $dstPort ]; then
        echo "Usage: $FUNCNAME <src port> <dst port> <port type>"
        echo " e.g.: $FUNCNAME 80 50080 tcp"
        return 1
    fi

    iptables-unassign-port $srcPort $portType
    local comment="redirect:port:$portType:$srcPort"
    local redirect="-t nat -p $portType --dport $srcPort -j REDIRECT --to-port $dstPort -m comment --comment $comment"
    local accept="-A INPUT -p $portType -j ACCEPT -m comment --comment $comment --dport"

    sudo /sbin/iptables $accept $srcPort
    sudo /sbin/iptables $accept $dstPort
    sudo /sbin/iptables -A PREROUTING $redirect
    sudo /sbin/iptables -A OUTPUT -o lo $redirect
)} 

function iptables-unassign-port() {(
    set -e

    local srcPort=$1
    local portType=$2

    if [ -z $portType ]; then
        echo "Usage: $FUNCNAME <src port> <port type>"
        echo " e.g.: $FUNCNAME 80 tcp"
        return 1
    fi

    comment="redirect:port:$portType:$srcPort"

    sudo /sbin/iptables-save | while read line; do
        if [[ $line =~ ^\*(.*) ]]; then
            table=${BASH_REMATCH[1]}
            continue
        fi

        command=$(echo "$line" | grep -- "$comment" | sed 's/^-A/-D/g')

        [ -n "$command" ] || continue

        echo $command | xargs sudo /sbin/iptables -t $table
    done
)}

function iptables-list-assigned-ports() {

    comment="redirect:port:$srcPort"

    sudo /sbin/iptables-save | grep "$comment"
}

function container-activate() {(
    set -e

    local svcName=$1

    if [ -z $svcName ]; then
        echo "Usage: $FUNCNAME <svc name>"
        echo " e.g.: $FUNCNAME container-name-1.0.0"
        echo
        echo "Services:"
        docker ps --format json|jq -r .Names
        return 1
    fi

    while read dst src type; do
        iptables-assign-port $src $dst $type
    done < <(docker ps -f name=$svcName --format json|jq -r .Ports|sed 's/, /\n/g' | sed 's/^.*://g' | sed 's/->/ /g' | sed 's#/# #g')
)}

function container-deactivate() {(
    set -e

    local svcName=$1

    if [ -z $svcName ]; then
        echo "Usage: $FUNCNAME <svc name>"
        echo " e.g.: $FUNCNAME container-name-1.0.0"
        echo
        echo "Services:"
        docker ps --format json|jq -r .Names
        return 1
    fi

    while read dst src type; do
        iptables-unassign-port $src $type
    done < <(docker ps -f name=$svcName --format json|jq -r .Ports|sed 's/, /\n/g' | sed 's/^.*://g' | sed 's/->/ /g' | sed 's#/# #g')
)}        

• Multiple instances, multiple versions. These are usually network daemons or web services and may run multiple instances of multiple versions across multiple nodes, usually routed to via a reverse proxy. Reverse proxies and their configuration are all different and are not in the scope of this specific article. That’s a whole different topic in itself and for another time. ?

How to build the versioned containers?

Hello, username! You made it to the final level!?

And… Since we began at the end…?

It’s only fitting that we end at the beginning.?

Which is the very beginning.?

The container build process.

There are many ways to build a container image, I will be using docker compose for my example because the syntax lends itself nicely to what we are trying to do here. Note that we specify the version as a part of the image name and as a part of the container_name. To review, the image name guarantees that our code did not change, and the container name guarantees that we run only one instance of the same version per host and allows us to reference that specific version during our interactions with the deployed version.?

docker-compose.yml

services:
  container-name:
    container_name: container-name-${tag:-latest}
    image: repo/container-name:${tag:-latest}
    build: .
    ports:
      - "53/udp"
      - "53/tcp"        

We can use this docker compose file to build our containers manually. But...?

Since we want immutability, why not build the image as soon as the tag is created? Let’s use a bit of git hooks magic for it.?

But…?

Before we do that….?

I have to mention that by default git is looking for hooks inside the .git folder, and that folder is not tracked. I suggest putting files into the .githooks/ folder, git adding .githooks, and then reconfiguring git repo to use .githooks folder for hooks with: git config --local core.hooksPath .githooks/?

Git doesn’t have a hook that works with tags, but it does have one that works with refs.

We can tailor it to look for a tag specific script like so:

.githooks/reference-transaction

#!/bin/bash -e

state="$1"

while read old new ref; do
    if [[ "$ref" =~ ^refs/([^/]*)/(.*)$ ]]; then
        type=${BASH_REMATCH[1]}
        id=${BASH_REMATCH[2]}

        if [[ $new =~ ^0+$ ]]; then
            action=del
        else
            action=new
        fi

        script="$0-$state-$action-$type"

        if [ -x $script ]; then
            $script $id $old $new
        fi
    fi
done        

Then we create a tag specific script that will call docker-compose with the tag like so:

.githooks/reference-transaction-committed-new-tags

#!/bin/bash -ex

tag=$1
old=$2
new=$3

tag=$tag docker compose build        

And there we have it folks, we now have a setup that builds immutable versioned containers as soon as the tag is created and allows us to run versioned containers on the same host in parallel and route traffic to the version we choose.?

Hopefully this is somewhat useful.

Have a better way? Please share!