DevOps - Part 4 - Finally, deploying!

This is part 4 of a series describing a minimum setup for a modern DevOps culture, for mature organizations that have not yet embraced it.

This time, we will be finally releasing our code!

In the previous 3 parts, we were building the necessary scaffolding, now we will actually deploy our code.

Manual deployment? oh no no!

The critical, but not a sufficient part of achieving the "DevOps Culture", is elimination any manual step in building and deploying of the system. Very often, code is manually built on someone's computer, or perhaps on a build computer, the deployable artifact is shipped to some location and then semi-manually deployed to the different environments.

This we must fix. No more manual steps in this process!

For this, you will need some kind of DevOps tool

Choose your DevOps tool

Here I am referring to a tool that will build and deploy your system to your non prod environments, eventually making its way to production. Sometimes those tools are called "Continues Delivery" tools, but that's semantics.

There are many tools that perform this - with Jenkins being a leader - and any you choose work fine. I suggest Azure DevOps Services. It is a horribly named service, as many assume it is a tool for deploying to Azure. It is not! It is a generic, all encompassing DevOps tool. By choosing this, you get the full suite of services you need to implement your DevOps setup, including the build and deploy step we will be discussing now.

If you hate Microsoft, you might want to look into Alsatian/Jira suite of tools. Expect a longer start up time, more customization work and higher maintenance effort. I have worked with both and would choose Azure DevOps anytime.

Oh yea, expertise in the tool will be needed

What ever tool you need, you will need to either learn, or hire someone with expertise on how to build and deploy your code using this tool. In case of Azure DevOps, it is the Azure Pipelines feature. Take your time to learn, will save you countless hours in the long run.

What teams usually get wrong from day one

In my experience, any organization on the path towards a DevOps setup starts trying to "automate build and release". Why? because It is the most exciting, interesting and natural thing for the dev team to do. The previous 3 parts are either skipped or done poorly, and THAT is the the reason, why the effort usually brings little fruit.

Automating build and deploy is useful, but without a proper development discipline using source control branching, without properly configured and handled environments, without a good SDLC/QA process that ensure the code is working as expected, automatic build and deploy is just a neat toy.

Very often, the team's claim "we have devops" is limited to this step - build and deploy automation.

You need the foundation. Assuming you carefully handled the foundation, now you must create pipelines that build and deploy the code, usually called the "DevOps Pipeline".

This must follow the following principles :

1. Build once
2. Deploy the same artifact to all environments
3. Use the exact same method of deployment to all environments

Usually, greenhorns get all 3 wrong.

Lets build and improve the pipeline

Usually, this is how the setup looks :

Issue 1 : Wrong branching strategy

First issue, are the branches. We already discussed the proper branch setup but I'd like to reiterate again. Maintaining branch per environment is at first most intuitive, easy to setup, easy to understand and hence most common. It is a wrong setup however. I assume the numerous pitfalls (like high likely hood of deploying different code to different environments) needs not be discussed.

Here is the setup you want:

One exception is that teams often want to setup full CI/CD (automatic deployment after every build) on development, therefore you can leave DEV on the main / develop branch:

Issue 2 : Multiple builds

Even with one branch feeding all builds, multiple build still occur. A separate build takes place for each environment.

Can we be certain that each build has the same code?

TEAM: >> Yes, we will ensure we are using the same commit.

Ok, but can you ensure that the all other build decencies, that the build machine is exactly the same each time?

TEAM: >> well, we will be careful!

Can you be sure that the build script is the same for all environments?

TEAM: >> well, we will be careful!

There is no need take this burden and risk. There is a way to avoid all of this by building only once, like so:

Objection - But we build different code for different environments

It sometime happens that configurations for different environments is baked into compiled code, therefore the above can be achieved.

I will highly warn against this approach. It is much better to store configuration outside of the compiled code, either via external files or environment variables etc. For more info please see this.

Objection - Our build has to be different as it injects different config files based on environment

While similar to above, it is slightly different. Since config is not baked into the compiled code, one artifact can be created with supporting files different per environment.

This should also be avoided. This should be handled by the deployment files. Deployment (aka environment) specific config should be injected during this step, so it can be handled by Deploy Script

Issue 3: different deployment scripts used for each environment.

As we can see, we have a Deploy Script per environment, that will likely only different based on the environment configuration. However, can we be sure? Can we be sure that a change that is made in QA, propagates properly to all other scripts?

Even if we do remember, will the change be properly propagated? There is no way to just copy and paste, each change as to be carefully implemented in each script. This is basically a classic example of the worse of "code duplication" problem. Because often DevOps engineers are not developers by trade, they often do not understand that gravity of this problem.

Fortunately, there is a trivial way of handing this problem. All DevOps platforms offer the ability to compile a list of environment specific configuration, and inject this configuration to scripts based on environment.

Here is the desired setup:

Gatekeeping - when to deploy?

Now that you have a well functioning build and deployment, there are various options for triggering the build and deploy.

Again, because this guide is retrofitting DevOps process into an existing manual change management process, I suggest this simple approach, and adjust as you learn.

DEV

Commit driven full CI/CD for DEV. The moment a change is made to the main branch, it is automatically built and deployed to DEV.

QA

Require permission /sign off from both QA and Dev teams before a built is deployed to QA.

QA teams this way acknowledges that a new change is coming and they are ready to prepare for it and the Dev team acknowledges that this particular built is QA-worthy.

STG

Require QA and management permission before a build is to to go STG. It is meant to be a last step in preparation for production release. QA's sign off signifies their acceptance of the build in QA, management's sign off indicates their understanding that a release to PRD is coming.

PRD

Require sign off from the people who are able to sign off a new production release.

All DevOps tools have the ability to setup such gates and track signoffs for the purpose of documentation and audit.

OK! so we are done? Well... no... Sorry!

We are well on our way, however, as no doubt anyone who tried it knows, we have not discuss changes to the Database. Almost all systems have some kind of datastore, which is, by definition, stateful. There is no way to simply compile a new version and replace the old like we do with code. This is a big problem because effectively, you have automated 90% of your process but this 10% prevents you from being fully hands-off.

Well, there is a solution! In the next installment of this series, I shall describe a battle tested method of handing SQL based Database changes in the same way as code.

Stay tuned!