Azure Devops with Terraform - Patterns and Design

Intro

Much of my recent work has been using Azure Devops to deploy Terraform - everything as code. However, one thing I’ve noticed are in low supply are articles about leveraging the benefits of Azure Devops when deploying Terraform. Nearly all the blog posts I see about Terraform and Azure are using well known AWS centric patterns. At best, these patterns simply don’t take advantage of the benefits Azure Devops offers, and at worse it actively introduces new problems. I have developed a preferred way of working with these technologies which I’ll get into now.

Note: The LinkedIn editor isn't really very good for code snippets, I was considering leaving them in here as block quotes, but they were really awful to read, so I've decided to screenshot instead. Not ideal for copy and pasting, I appreciate, but if enough people are interested in the code I'll add it to a public github repo at some point. Although it shouldn't take long to recreate it from this anyway

The problems, and their solutions

There are a number of issues I’ve seen arise when people are using AWS centric patterns to deploy Terraform via Azure Devops Pipelines:

1.      Lack of source control of non-secure variables due to a single tokenised variables file and library variable sets being used to transform on deploy

2.      Inability to run a terraform plan locally, safely against non-prod and production environments.

3.      Testing changes against your non-prod environments

4.      Pull Request Reviews where the reviewer has to jump through hoops to see variable changes, or is entirely blind to the variable changes made if they don’t have view permissions on the library variable sets.

5.      Abstraction layers which are not required, reduced readability, and increased complexity

The first 4 problems are related to variables. The pattern I’ve commonly seen being used in Azure Devops is to have a single vars.tfvars file for each of the terraform resource files being deployed. Inside these files are tokens which are being replaced at deploy time for each environment. However, you’ll also see in articles online stating that best practices around variables is that you shouldn’t have locally declared environment specific variable files such as is shown in the image below:

Instead, these variable will sit in a library variable set in the azure Devops instance. This creates a number of problems.

The first glaring problem being that unless these are backed by a keyvault (which we will get onto later, as there is a use case for this), there is no source control, no ability to roll these back when changed, and no audit of who changed what, and when.

The second problem is that PR’s become much more difficult to conduct. Unless you specifically tell the reviewer you’ve made library variable changes, and what changes you’ve made, they won’t know to look there, and this becomes a manual process - an additional action the reviewer has to commit to, to complete the review. The reviewer may not even have permission to view the library variable set, just the code, so this complicates things even further.

The third problem is related to running terraform plan locally. When you’re doing token replacements at deploy time, you have no ability to run a terraform init and plan locally, meaning that your dev environment becomes your testing environment. Now you might be thinking, that’s fine, it’s dev, it’s not customer impacting. Wrong. Our customer isn’t the classic end customer, our customer is the development team. Our production environment is the PTL, be that Dev, QA, UAT, Stage and Prod, or Test, Stage, and prod (or any other permutation of the PTL). If we break any of those environments we’ve broken our version of production. Being able to run a terraform plan locally gives us a lot more confidence that a new terraform change isn’t going to have unforeseen consequences such as tearing down and recreating a service which is currently in use, or simply tearing down a service entirely. As an extra layer of security and governance, you should ensure all users by default are not contributors on a subscription, so an accidental terraform apply cannot be ran locally against any environment. Having the correctly scoped reader permissions will allow a user to safely run a terraform plan.

By setting these variables in a local file, we are able to fix each of the issues mentioned above. The one obvious omission here is secure variables and what do we do with those? The answer is, ironically, a library variable set, but, no token replacements are needed. The library variable set should be backed by a keyvault to contain these secrets, and all secrets should have versioning enabled on them, to allow the rolling back of those variables. As for passing these into your terraform runner in the pipeline, you can pass them in as command line arguments. At the end of this guide I will provide the full example code templates, but for now, a snippet of what this looks like:

The “commandOptions” ${{ parameters.additionalParameters }} is where you pass these in. This also allows for you to specify a top level variable in your pipeline, such as a location for deployments, which is the same across all of your PTL, so you don’t have to redeclare it in each tfvars file you create.

Note: The snippet above is using the Charles Zipp Azure Pipelines Terraform Tasks marketplace extension. Azure Pipelines Terraform Tasks - Visual Studio Marketplace as I’ve found it to be far more user friendly and better featured than the Microsoft version.

The final problem in the list is around code complexity. It’s standard in AWS circles to write modules for everything you do in Terraform, and it makes a lot of sense, they’re reusable, they give you better control and compliance over infrastructure state, and, especially at enterprise level, they offer the ability to centralise and distribute your IaC codebase.

 In Azure Devops I see something similar being done, but rarely does it take into consideration the benefits and ways of working with Azure Devops Pipelines as Code. Thanks to the fact that you can write a deployment template in azure pipelines, and call that template multiple times with its own set of environment specific parameters, you can deploy multiple environments in the exact same way, adding new environments with very little effort if needed.

What I’ve found, especially with smaller projects, is that you only have one piece of each type of infra per environment. You’ll have a single dev database, redis cache, app service, keyvault, etc. By creating the resource file for these pieces of infrastructure and then calling that as a module in another deployment file which you will then just go ahead and call via your pipeline template, you’re just adding in a layer of abstraction that offers little benefit.

 Terraform modules are a way of allowing you to use terraform to orchestrate what to deploy, but azure Devops pipelines also offers this through its own templating function, if you’re using azure pipelines and also creating modules for single pieces of infrastructure you’re just doubling up your orchestration layer. By creating your resources directly and calling that via your pipeline template, you remove the need for the module. However, if you do have a piece of infrastructure being created multiple times per environment, such as multiple VM’s with the same config, a module in combination with azure Devops is absolutely the way to do this.

Example Code

The code example I’m using here are a striped down version of some code I’ve used in personal and professional projects before, kept generic enough that you should be able to take the code, and amend it so it fits your own needs. I’ve not included all the templates however, as they’re not relevant to this post.

Here is the main pipeline.yml file. Clean, easy to read, and very easy to add a new environment if required.

Here is the deploy-to-environment template

The example above shows a set of deployments, split into sections for networking (VNets, Subnets, Public IP’s, App gateways etc), data (Azure CosmosDB, Azure Cache for Redis) and the app service. You can of course make it more or less compact depending on how you wish to structure your main.tf files/file.

This is the “meat and bones” (or tofu and plants if you’re vegan) of the whole thing. Using the charleszipp terraform tasks referenced earlier in the post, we are running a terraform init, plan, and apply. If you look at the plan task you’ll notice under commandOptions we have both a top level variable being used, which is the location variable, along with an environment specific variable, which is the containerid. If you were also using secrets here you would include an additional parameter to the “deploy-to-environment” template which would be the variable group’s name, and call this in the variable declaration section (or better yet, create a library variable set with a naming convention that include the environment name so you can use this instead, lessening the amount of variable you’re passing around).

And that about covers it. To me, this is the best pattern for the job, at least that I’ve discovered during my experimentation with these technologies. I welcome all discussion around this, maybe you think it’s totally the wrong way around it, if so, let me know how it could be improved, I’m all for throwing defunct ideas out the window if something better comes along.