Terraform and Terragrunt: What and Why

by Anna Hrynenko

To many of us, DevOps as a whole is a mystery. We just got comfortable saying “Docker”, “Docker Compose” and “containers”, and maybe have a vague idea of Kubernetes, and then there’s Terraform, Terragrunt, Helm charts, Infrastructure as Code, and we’ve lost the thread again. What exactly are those? How do they work? Today I am going to ask these questions of our brilliant colleague and DevOps expert,?Anatolii Kaliuznyi.

Anatolii is the?Director of DevOps Engineering?at DreamProIT and a сertified GCP solution architect with nearly a decade of experience building automated seamless CI/CD pipelines for large and complex projects. Anatolii will help us take a closer look at Terraform and how DevOps engineers and clients can benefit from using the tool.

?

Anatolii, what exactly is Infrastructure as Code (IaC)?

“Nowadays Infrastructure as Code is not just an opportunity but a need for many projects. The need for infrastructure to be written as code, that can be recreated fast and in a controllable way when necessary. It makes it possible to control and restore the versions of our code as a way to control the infrastructure, as well as audit the whole process. There are several IaC tools that allow you to create the infrastructure.”

What are those tools, and what’s your personal favorite IaC tool?

“With no doubt, Terraform. Other tools are less common, and they have some serious disadvantages that Terraform does not have. For example, Terraform has the huge support of its community and engineers. The community is pretty big and has contributed a lot over the last 5 years. Terraform allows you to create any resources (i.e. VMs, networks, DNS, even Kubernetes clusters) with just 10-15 lines of code. This is vital for disaster recovery and repeatability of the infrastructure.

When I first started using Terraform, it was version 0.7, which was not that great. Back then, it was problematic to do some basic things, and we had to write scripts that helped us customize the Terraform code because on its own it wouldn’t allow you to do these basic tasks. We had to create templates from the Terraform file which is not very good because it caused conflicts between our hand-written scripts and the Terraform manifest. Let me provide some context. A ‘manifest’ is a text that provides directive language describing the purpose of the tool and the overall goals for the state of a tool or system. You do not identify the steps needed to get to the desired state in the manifest because the system itself decides on the steps. On the other hand, a script is a sequence of instructions that a system should take to achieve the desired state. This is where the conflicts in early Terraform arose. Terraform works with a manifest, and the scripts we created for customization could interfere with manifest execution. So, people didn’t really want to use Terraform at that stage because of potential conflicts with their scripts.

Thanks to the open-source community, Terraform made huge advances: it no longer needs any tricks or specialized scripts to do basic things, and a Terraform manifest does not (usually) require additional scripts to fill in the blanks. In its updated versions, Terraform is an industry standard and is extremely popular.”

So, it looks like now Terraform allows you to do hard things in an easier way.

“Indeed. Terraform, together with some wrappers, allows structuring the code and makes it possible to more easily automate infrastructure. The whole idea is to create infrastructure in a simpler way with no need to repeat yourself.”

Let’s talk about wrappers. Are they extensions you use for Terraform?

“Yes, exactly. There are few of them, but not all are equally useful. Sometimes a wrapper is meant to do one specific thing for a very extreme case. But there are several more generic ones that add some capabilities and “syntactic sugar” for Terraform.”

Do you prefer vanilla Terraform or are there any wrappers you use?

“I would say it depends on the task. Many projects are fine without any wrappers. However, if I had to choose one, I would highly recommend Terragrunt.

Terragrunt has some requirements for the structure of the code repository. If a project already has a big code base on Terraform, and it cannot be used with Terragrant, it will take a lot of time and hard work to adjust this code to fit Terragrunt. I have worked with projects like that before, and I think the best way to go about it is to freeze the already existing repository and start using Terragrunt to do all new things. This way, elements will be added to Terragrunt, and, step by step, the repository will be fully transferred to Terragrunt. For new repositories, I try to structure them so that Terragrunt can be used from the start.”

What exactly is Terragrunt? How does it work, and what problem does Terragrunt solve?

“Basically, Terragrunt is a set of scripts – specific useful things that make the life of DevOps engineers who use Terraform easier. Terragrunt allows engineers to repeat themselves less. Of course, this invokes the DRY principle here:?“Don’t repeat yourself”.

If you use vanilla Terraform you have to identify where you keep your state file and the provider that you will use. This is something that has to be defined every time. Terragrunt allows you to do it once in one place and just use the configuration over and over again. It even allows you to define the template with variables.

Also, when launching Terraform, you have to put every file with variables in the Terraform command line every time. Terragrunt allows you to define this in code once, and all the variables will be substituted automatically.

Most importantly, Terragrunt allows you to define the variable on different levels. That means that you can create a hierarchy of variables and define them in Terragrant once, and it will be redefined on each level allowing you to move from general parameters to more specific parameters: from organization level and environment all the way down to a specific resource.”

From your experience, do teams like working with Terraform and Terragrunt?

“95% of DevOps vacancies require an engineer skilled at Terraform. It’s a must nowadays. As I mentioned previously, Terraform became an industry standard, and projects without Terraform are rare to find.

Talking about Terragrant, people always have their own opinions and experiences, but I believe we can all agree that Terragrunt makes work more pleasant. It reduces the time spent on coding, implementing, and testing. As a result, it reduces the general time spent on development.”

To me Terragrunt sounds like an efficient way to automate the work of the engineers. How do clients benefit from using Terraform together with Terragrunt in their projects?

“One of the benefits Terragrunt has is the reduction of development time. Thanks to repeated patterns, engineers can develop new things easier and faster because they do not need to rewrite the same things many times. Also, it provides an opportunity to create more complex and complicated infrastructure.

If we are talking about vanilla Terraform and IaC as an approach in development, using them improves disaster recovery. It also makes security and audits better because we know what, when, and by whom was done. Among other benefits, there are code reviews and testing of the code. Literally, it allows us to add CI/CD into the infrastructure which itself is already a great con to the stability of infrastructure.”

Could you provide any use case examples where Terraform and Terragrunt helped to solve an issue or drastically improve the project?

“A great example is an infrastructure project for a major retail customer at DreamProIT. The project itself consisted of various virtual machines but their structure, unfortunately, was dependent, meaning that if one virtual machine “dies”, they all do. That was horrific for the project.

By using Terraform together with Terragrunt, we created a separate structure that allowed us to divide and share the load between the virtual machines. We implemented a couple of classical mechanisms of balancing the load, and the project was brought to the automated scaling.

Additionally, the whole infrastructure was created in a way that allows easy restoration and management. This was one of the main priorities for the project. Basically, what previously would take about a week of work for a DevOps engineer, now would take about an hour. Most importantly, after implementing this, the infrastructure can be also managed by a non-technical person who has the technical documentation.”

Using Terraform and Terragrunt, as its wrapper, does sound like a win-win for both engineers and clients! Anatolii, thank you for your time and for the comprehensive explanations.

We’ll be coming back for more details of how DevOps engineers work with these and other IaC tools in future posts!