EKS deployment Lifecycle management using Flux and GitOps principles with Terraform

Use Case: Deploying an EKS cluster with an ingress controller, monitoring tools like Datadog, and managing the entire lifecycle using Flux and GitOps principles with Terraform

Situation

Your organization is migrating to Kubernetes on AWS using EKS. To ensure the cluster is observable, secure, and can handle traffic efficiently, you need to deploy Datadog for monitoring, an ingress controller for managing external traffic, and other monitoring tools. You also want to adopt GitOps principles for better infrastructure management and CI/CD automation using GitLab.

Task

You need to:

1. Set up an EKS cluster.

2. Deploy Datadog, an ingress controller, and other monitoring tools.

3. Use Flux for GitOps to manage Kubernetes manifests.

4. Implement a CI/CD pipeline in GitLab that automates the deployment process using Terraform and Kustomize.

Action

1. Set Up EKS Cluster with Terraform:

- Create an EKS cluster using Terraform scripts.

- Define necessary IAM roles, VPCs, subnets, and security groups.

2. Deploy Datadog and Ingress Controller:

- Use Helm charts to deploy Datadog and an ingress controller (like NGINX) onto the EKS cluster.

- Write Kubernetes manifests for Datadog and ingress controller configuration using Kustomize.

3. Set Up Flux for GitOps with Terraform:

- Use Terraform scripts to install Flux on the EKS cluster.

- Configure Flux to sync with a Git repository containing Kubernetes manifests managed by Kustomize.

4. Implement CI/CD Pipeline in GitLab:

- Create GitLab CI/CD pipelines that trigger on code commits.

- Use Terraform scripts in the pipeline to manage EKS resources and Flux installation.

- Automate the deployment of Kubernetes manifests using Flux and Kustomize.

Result

The organization now has a scalable, observable, and highly automated Kubernetes environment on AWS. The GitOps approach ensures that all changes are versioned, reviewed, and auditable. The CI/CD pipeline in GitLab automates the deployment process, reducing manual intervention and errors.

Sample Code for Components in CI/CD Pipeline

1. Terraform Code for EKS Cluster and Flux

`main.tf`

`variables.tf`

2. Kustomize Base for Datadog and Ingress Controller

Base for Datadog:

• base/datadog/deployment.yaml

• base/datadog/kustomization.yaml

Base for Ingress Controller:

• base/nginx-ingress/deployment.yaml

• base/nginx-ingress/kustomization.yaml

Functionality of the Provided Code

Deployment Manifests:

• The deployment.yaml files define the Deployments for Datadog and NGINX Ingress Controller. These files specify how many replicas of each Pod should run, which container images to use, and how to configure the containers via environment variables and command-line arguments.

Kustomize Configuration:

• The kustomization.yaml files are used by Kustomize to manage the Kubernetes resources. They specify which resource files to include, allowing you to build and customize the Kubernetes manifests for different environments (e.g., development, staging, production) by combining different bases and overlays.

Kustomize Base:

• The base directory contains the common configuration for Datadog and the NGINX Ingress Controller. This configuration can be reused across multiple environments by using overlays to customize specific settings.

By structuring the Kubernetes manifests in this way, you can easily manage and deploy configurations across different environments, ensuring consistency and maintainability.

3. Overlay for Environment (e.g., Production)

• overlays/production/kustomization.yaml

• overlays/production/secret.yaml

Functionality of the Provided Code

Overlay Configuration:

• The kustomization.yaml file in the overlays/production directory defines the Kustomize overlay for the production environment. It combines the base configurations for Datadog and NGINX Ingress Controller with environment-specific customizations.
• By referencing the base configurations in the bases section, the overlay inherits all the resources and configurations defined in the base.
• The resources section allows you to add additional resources or modify existing ones. In this case, it adds a Kubernetes Secret specific to the production environment.

Secret for Datadog API Key:

• The secret.yaml file defines a Kubernetes Secret resource that stores the Datadog API key. This Secret is specific to the production environment and is used to provide the API key to the Datadog agent running in the cluster.

Summary

1. Base Configuration: The base configurations for Datadog and NGINX Ingress Controller define the common resources and settings that can be used across multiple environments.
2. Overlay Configuration: The overlay configuration for the production environment customizes the base configurations for production-specific settings. It includes additional resources like the Datadog API key.
3. Kustomize: By using Kustomize, you can manage and customize Kubernetes manifests in a structured and reusable way. Overlays allow you to tailor the base configurations for different environments, ensuring consistency and ease of maintenance.

4. GitLab CI/CD Pipeline Configuration

`.gitlab-ci.yml`

Explanation and Functionality

1. Stages:

- stages: This section defines the different stages of the CI/CD pipeline. In this case, there are two stages: terraform and deploy.

- terraform: The first stage, where Terraform is used to set up the infrastructure.

- deploy: The second stage, where the Kubernetes resources are deployed using Kustomize and Flux.

2. Variables:

- TF_ROOT: This variable defines the path to the directory containing the Terraform code. It is used in the terraform job to specify where the Terraform configuration files are located.

3. Before Script:

- before_script: Commands listed under before_script are executed before each job in the pipeline. In this case, it installs Terraform and kubectl using the Alpine package manager.

4. Terraform Job:

- terraform: This job runs in the terraform stage.

- stage: Specifies that this job belongs to the terraform stage.

- script: Contains the commands to be executed in this job.

- terraform init $TF_ROOT: Initializes the Terraform configuration in the specified directory.

- terraform plan $TF_ROOT: Generates an execution plan for Terraform, showing what changes will be made.

- terraform apply -auto-approve $TF_ROOT: Applies the Terraform configuration to create or update infrastructure resources. The -auto-approve flag is used to skip manual approval.

5. Deploy Job:

- deploy: This job runs in the deploy stage.

- stage: Specifies that this job belongs to the deploy stage.

- script: Contains the commands to be executed in this job.

- kubectl apply -k ./overlays/production: Uses kubectl to apply the Kustomize configuration for the production overlay. This command deploys the Kubernetes resources as defined in the overlays/production directory.

- flux reconcile kustomization flux-system --with-source: Reconciles the Flux Kustomization resource, ensuring that the desired state defined in the Git repository is applied to the cluster. The --with-source flag forces Flux to pull the latest changes from the Git repository.

Summary of Functionality

- Terraform Stage:

- Initializes, plans, and applies the Terraform configuration to set up the EKS cluster and any other infrastructure resources defined in the Terraform code. This ensures that the necessary infrastructure is in place before deploying the Kubernetes resources.

- Deploy Stage:

- Uses kubectl with Kustomize to apply the Kubernetes manifests for the production environment. This deploys the Datadog agent, NGINX Ingress Controller, and any other resources defined in the overlays/production directory.

- Reconciles the Flux Kustomization resource to ensure that the desired state from the Git repository is applied to the Kubernetes cluster. This step ensures that the cluster's state matches the desired configuration stored in Git, implementing the GitOps principles.

This CI/CD pipeline automates the deployment process, ensuring that infrastructure is provisioned and applications are deployed consistently and reliably. By using Terraform for infrastructure as code and Flux for GitOps, you can maintain a declarative and version-controlled approach to managing your EKS cluster and Kubernetes resources.

Benefits

1. Scalability: EKS provides a scalable infrastructure for deploying Kubernetes.

2. Observability: Datadog ensures comprehensive monitoring and logging.

3. Automation: GitOps with Flux automates deployments and ensures consistency.

4. CI/CD Efficiency: GitLab CI/CD pipelines streamline the deployment process.

5. Version Control: All changes are tracked in Git, making it easier to audit and roll back if necessary.

Challenges

1. Complexity: Setting up and managing the entire stack can be complex and require significant expertise.

2. Initial Setup Time: The initial setup of Terraform scripts, Helm charts, and CI/CD pipelines can be time-consuming.

3. Learning Curve: Teams may need training to adopt GitOps principles and use new tools effectively.

4. Cost Management: Continuous monitoring and scaling resources can incur higher costs if not managed properly.

By adopting this approach, your organization can achieve a highly automated, scalable, and observable Kubernetes environment that aligns with modern DevOps practices.

References:

https://www.dhirubhai.net/pulse/using-flux-gitops-tool-amazon-elastic-kubernetes-service-rasmuson-1f/

https://aws.amazon.com/blogs/containers/gitops-model-for-provisioning-and-bootstrapping-amazon-eks-clusters-using-crossplane-and-flux/