Rescuing My Terraform Dreams: How AWS Versioning Helped Recover My Lost Remote State?File

Introduction

Terraform is undoubtedly a potent Infrastructure as Code tool, and working with it can be exhilarating. However, managing the Terraform state file can be a tad tricky. One of the most significant challenges is to ensure that the state file is backed up and recoverable. This is an essential aspect of disaster recovery, but it can be cumbersome, necessitating meticulous planning to guarantee the integrity of data.

All hope was?lost

As a Terraform user, I recently encountered a nightmare scenario?—?I lost my remote state file! I was afraid that the infrastructure state file was gone, and I anticipated the worst. After hours of panic, searching online, shedding some silent tears, and praying, I was able to recover the state file and ultimately restore the infrastructure.

S3 versioning to the?rescue

My first bit of saving grace was having enabled “versioning” on the Terraform backend s3 bucket. Whew!

See code snippet below:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "uop-terraform-state"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status     = "Enabled"
    mfa_delete = "Disabled"
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  block_public_acls       = true
  block_public_policy     = true
  restrict_public_buckets = true
  ignore_public_acls      = true
}
        

This solution will only work if you have versioning enabled on the Terraform backend s3?bucket

The Solution

First, log into your AWS account where your state file is stored. Then, go to S3 and select the bucket. Navigate to the state file location and enable “Show versions”. Select the version of the file you are interested in and download it.

The file will be downloaded as a?.JSON file named terraform.json. Open the JSON file and copy all the contents. Then, open a new Notepad file and paste the contents copied from the JSON file. Save the new Notepad file as terraform.tfstate and upload it to the original location of the remote state file.

However, this is not enough to recover your infrastructure due to the state locking and consistency checking via DynamoDB.?

When using Terraform remote backend, it’s crucial to understand the concept of DynamoDB state locking. This feature is employed to manage and coordinate access to the state file when multiple users or processes are working on the same Terraform configuration. The way it works is that when a user or process initiates a Terraform operation (e.g., terraform apply, terraform plan), it first attempts to acquire a lock in the designated DynamoDB table.

If the lock is successfully acquired, then the operation proceeds, and the user gains exclusive access to the state file. If another user or process attempts to perform a Terraform operation on the same state file simultaneously, they won’t be able to acquire the lock and must wait until the lock is released. Once the operation is complete, the lock is released, thereby allowing others to access the state file.

terraform {
  backend "s3" {
    bucket         = "uop-terraform-state"
    key            = "global/s3/terraform.tfstate"
    region         = "ap-east-1"
    dynamodb_table = "terraform-state-locking"
    encrypt        = true

  }

}

# DynamoBD CMK
resource "aws_kms_key" "dynamodb" {
  description         = "DynamoDB Table Server side encryption"
  enable_key_rotation = true
  key_usage           = "ENCRYPT_DECRYPT"

}

resource "aws_kms_alias" "dynamodb" {
  name          = format("alias/%s-dynamodb-CMK", local.name,)
  target_key_id = aws_kms_key.dynamodb.key_id
}

# Dynamo DB 

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locking"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
  server_side_encryption {
    enabled     = true
    kms_key_arn = aws_kms_key.dynamodb.arn
  }
  point_in_time_recovery {
    enabled = true
  }
}        

To overcome the issue of the locked state file, go to DynamoDB and explore items. Select the DynamoDB table for your Terraform state-locking and under “items returned” in “LockID (String)”, select your LockID. Then, go to Actions and delete items.

Finally, go to your IDE and run ‘terraform init -reconfigure -backend-config=”access_key=AKXXXXXXXXXX” -backend-config=”secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX” ‘ and then run terraform refresh to ensure that your Terraform state file is up-to-date and reflects the current state of your infrastructure resources.

terraform init -reconfigure -backend-config="access_key=AKXXXXXXXXXX" -backend-config="secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"        

terraform refresh        

Voila! You are good to go!

Summary

The Terraform state file is a crucial component of any Terraform deployment, and its significance cannot be overstated. The entire Terraform codebase relies on the state file to determine the desired state and changes that need to be made to the infrastructure. It assists Terraform in determining which actions are necessary to align the real-world infrastructure with the desired configuration. The state file stores information about which resources have been created, their attributes, and their dependencies.

When working in a team, the state file serves as a shared source of truth for all team members. It promotes collaboration by allowing multiple users to work on the same infrastructure, ensuring that everyone has access to consistent and up-to-date information.

Given its importance, it’s crucial to have a well-planned disaster recovery strategy for the state file.