DEVOPS - Automating deployment machines with Azure Automation DSC and ARM Templates

Introduction:

In a world of ever increasing automation it's good to look into processes which require manual input and see just how far you can get with automation using the latest and greatest tech being released. It turns out that with Azure, automating a Virtual Machine and getting it into a desired state (i.e. it has all the software, certificates, folders etc set up without you having to manually go a create them) is not only possible, but fairly easy. When I first started attempting this, I didn't know how, because a lot of the technology I decided to use was so new. Stack overflow and google did very little to help. What I had instead was the, sometimes less than brilliant, Microsoft documentation (which in fairness, is improving) and the time needed to trial and error this until I had implemented it and understood it.

I'll start with a common scenario that many companies face which regularly deploy code, and talk about how I went about automating the process so that one check in is all that is required to get a deployment environment built.

Scenario:

You are a software company with a large website. You have a microservice based architecture, and so you are deploying components to test and live environments all the time, this can total up to thousands of releases a year. You are using some sort of pipeline tooling for this, probably a combination of Git/VSTS, Team City, Octopus, Azure or AWS or any other source control, pipeline tool or cloud based provider. The idea of the pipeline is that a developer checks in a change, this is then picked up, built, tested and pushed through your environments until you're ready to push to perf testing or production. To do the preceding parts, you need some way of deploying all that code to your test environments. So, I would imagine that somewhere in your infrastructure you have one or more servers sat there waiting to do the builds/tests and deploys. In some cases these might be on premise servers set up for this purpose, or perhaps they are VM's sat in a cloud somewhere. Either way, those machines, unless they are managed by some sort of desired state configuration tooling are going to be what I call "snowflake" machines, because they are unique. You might have a setup where you have 3 deployment targets waiting for your deploys, all of them should be able to deploy any of the code you're building, but that's not the case. Why? Most likely because developers have access to these machines and over time have installed various SDK's, certificates, and other moving parts needed for their newest code releases onto only a subset of these machines. Now you have machine drift where all your machines are in different states and certain code can only be built on certain machines, no one quite knows why, but "it just works so we'll leave it like that". This approach is what is going to cripple your ability to scale out and push out more releases. No one truly knows what is on the machines and if one were to die or get deleted from the cloud, you'd be stuck, because now a subset of your deployments wont work, and if you've got a critical bug fix that needs to go out yesterday, you're really going to struggle. Here is where automation using Azure can help you.

Technical Details: 

The setup I created and have worked with involves firstly creating Azure VM's through Azure ARM templates, a source controllable and fully automatable way of creating infrastructure. To do this I first hand built an Azure VM to the specs that I desired for 50% of the throughput I was expecting it to receive (if this throughput value ever changed and the VM's specs needed increasing, it's a trivial change to do). After building this VM in azure and testing it was fit for purpose, I exported the details and then wrote out an ARM template so that this VM could be created with one simple powershell command. However, I knew I wanted 2 VMs to add resiliency to my overall solution. I had to parameterise all of the inputs to the template so that I could write 2 custom parameter files for it and run them both separately to create two uniquely identified, but identically spec'd machines.

Next up, I wanted a way to automate the creation of these VMs, so I setup a pipeline using Team City and Octopus Deploy. In Team City I created a project and had it attached to the VCS git root of my repo where the ARM template for the VM's lived. Each time I checked into the git repo with new changes it would be picked up by team city. Team City then had build configurations designed to package the ARM template project and push it to a Nuget repository before then calling out to octopus to create and deploy a release of the ARM template. Octopus had a project set up for this deployment, and it was here that I set my parameters for each machine I wanted to create and scoped them to a process step per VM, meaning that adding a third or fourth VM means I only need to go into Octopus, add in a new step and scope out the relevant parameters for it alongside the existing parameters and steps, a task which would take less than 10 minutes to do. 

By this point, I've got a pipeline that requires a single check in and from that check in will build 2 VM's in Azure to a specified criteria with all networking, firewalling and disks set up as desired. Which brings me on to stage two. Getting the machines into the desired state.

Desired State Configuration is a minor powershell abstraction layer for writing simple code to install software or create directories (or basically anything else you can do in powershell) to get a machine looking and working how you want it. There are a considerable amount of DSC modules out there which simplify tasks such as registering a certificate with the machines store, formatting drives, installing msi/exe files and plenty of other server admin tasks. It has been around for a good few years now, but still doesnt have the same amount of community support as programming languages do.

A very new addition to the DSC family is Azure Automation Desired Configuration. This is essentially a DSC pull server in azure that allows you to run DSC to a VM in azure and ensure it stays in the desired state. For this part of the work I spent some time going through our existing deployment targets and finding out exactly what was installed on each one to get a comprehensive list of all required software, certificates, folder paths etc. After this was confirmed I wrote a DSC script to install each piece of the required config. Some of this was fairly straightforward, such as installing a cert or mapping a drive. Some of it got more complicated, when installs on the machine depended on other installs which came beforehand. I learned, painfully, that DSC doesnt actually run linearly, so my ordering, whilst being correct was occasionally failing when running. This can be easily rememdied by a simple [DependsOn] attribute in each block of DSC which defines a precursor install to the one being ran. Also some installs, particularly exe installs didnt play nicely with the DSC runners designed for them. At first I thought this was going to be a show stopper, but after some experimentation I discovered that DSC allows you to run any powershell command you want using the Script resource, which gives you as much power as pure powershell does inside of DSC, if you can do it in powershell, you can do it in DSC.

Once the DSC file was working as expected I set up a secondary pipeline in Team City and octopus. This pipeline, on a git check in would do the following, it would run an ARM template within the git repo which sets up the Automation Account (ARM templates are idempotent, if the account is already there, it will just move on without trying to recreate it), it then uploads all the required DSC modules to the automation account. I managed this by custom writing a powershell script in octopus to cycle through the folder with the modules in and upload them. Next it uploads the DSC file to the automation account and compiles it, meaning it's available for use. Finally a step runs which registers the compiled DSC with the two Azure VM's I had created in the last step.

At this point we have two Virtual machines in the exact same state.

Now, whenever I check in a DSC change, it will pass through the pipeline, get compiled and registered to the machines, and every 30 minutes, a check runs on the machines to ensure they are in sync with the DSC file, any new changes detected and it will pick them up and implement them without you having to RDP onto that machine once to manually do anything.

There is, however, still some room for abuse of the system in that if developers have RDP access to these machines (which might make sense for your setup) they can still go and add new software to one machine and not the other, DSC will not know about this as it's only concerned with the software it installs. If a developer went and removed some DSC software, it'd be back within half an hour, but if they add new software, it will stay there causing drift. So one final step you can take in this process is to have an Azure Automation runbook (a script which can run on a schedule) tear down the machines each week, and have your pipeline scheduled to recreate them afterwards. This way you'll only have drift for up to a week, and if your devs are following process correctly they will know to update the DSC file and not just go ahead and mess with the machine (they'll learn the hard way that it's only temporary when they are trying to shortcut the system).

I hope this has given some of you some ideas on how to automate an integral part of your infrastructure.

I don't use any other social media because it gives me a headache, so any new articles will be posted on LinkedIn.