GitOps with cdk8s, Argo CD, and GitHub Actions

You could organize your git repositories in a number of ways in your GitOps workflow. One best practice out in the wild is to keep your Kubernetes manifest files in a separate dedicated repository. This is the repository that your GitOps operator will watch for any changes. I am a fan of this idea, and I prefer to keep this repository as clean as possible. The benefits of doing it this way include keeping the manifest repository clean (i.e. pure yaml), and you could lock down access to this repository if needed. In this article I will show one possible setup, where I have separated the construction from the storage of my manifest files.

I will set up two git repositories, one repository that will have a cdk8s project written in Go, and the second repository will have the resulting output from cdk8s (i.e. Kubernetes manifest files). I will use minikube to set up a local Kubernetes cluster, and I will use Argo CD as my GitOps operator. Argo CD will watch my manifest repository for changes.

This article is available on my blog in a nicer looking format https://mattias.engineer/posts/gitops-cdk8s-argocd/

Let us GitOps

Prerequisites

If you would like to follow along there are a number of tools you should install first. I will assume you are using macOS, but if not you have to adjust the instructions for your own OS.

• GitHub CLI (install by running brew install gh, authenticate by running gh auth login)
• Minikube (install by running brew install minikube)
• Docker (install by following the guide for Intel chip or Apple silicon)
• kubectl (install by following the guide for Intel chip or Apple silicon)
• Argo CD CLI (install by running brew install argocd)
• cdk8s (install by running brew install cdk8s)

Creating my git repositories

First I create my two repositories. I start with the repository that will generate the manifest files using cdk8s:

$ mkdir ~/code/gitops-cdk8s-argocd
$ cd ~/code/gitops-cdk8s-argocd
$ git init
$ touch README.md
$ git commit -am "Initial commit"
$ gh repo create \
??? --public \
??? --remote origin \
??? --description "GitOps with cdk8s, argo cd, and github actions" \
??? --source . \
??? --push        

I do the same thing for my second repository, where I will later store the generated manifest files:

$ mkdir ~/code/gitops-cdk8s-argocd-manifests
$ cd ~/code/gitops-cdk8s-argocd-manifests
$ git init
$ touch README.md
$ git commit -am "Initial commit"
$ gh repo create \
??? --public \
??? --remote origin \
??? --description "GitOps with cdk8s, argo cd, and github actions" \
??? --source . \
??? --push        

Now I am ready to start writing some code that will generate Kubernetes manifest files for me.

Using cdk8s to create my Kubernetes manifest files

I will use cdk8s to generate Kubernetes manifest files. I begin by initializing a new project. I will use Go as my language of choice.

$ cdk8s init go-app        

I do not want to store the imports directory or the generated manifest files in this repository so I add a .gitignore file with the following contents:

# do not commit manifest files
dist

# do not commit imports
imports        

The application that I will build consists of a simple Nginx web server with a custom index.html. In Kubernetes terms I will have a deployment, a service, and a config map. When I initialized the cdk8s project I was given the following overall code (with some details omitted):

package main

import ( ... )

type WebsiteChartProps struct {
?? ?cdk8s.ChartProps
}

func NewWebsiteChart(scope constructs.Construct, id string, props *WebsiteChartProps) cdk8s.Chart {
?? ?// ... details omitted ...
}

func main() {
?? ?app := cdk8s.NewApp(nil)
?? ?NewWebsiteChart(app, "gitops-cdk8s-argocd", nil)
?? ?app.Synth()
}        

Everything is in place for us to start writing our constructs in the NewWebsiteChart function. I begin by creating my chart. A chart is something that will be synthesized into a Kubernetes manifest file. I will only use a single chart in this example.

var cprops cdk8s.ChartProps
chart := cdk8s.NewChart(scope, jsii.String(id), &cprops)        

I add a variable for common labels I want to add to my service and deployment:

label := map[string]*string{"app": jsii.String("website")}        

The first Kubernetes object I add is my service:

k8s.NewKubeService(chart, jsii.String("service"), { &k8s.KubeServiceProps
? Spec: &k8s.ServiceSpec{
??? Type: jsii.String("LoadBalancer"),
??? Ports: &[]*k8s.ServicePort{{
????? Port: jsii.Number(80),
????? TargetPort: k8s.IntOrString_FromNumber(jsii.Number(80)),
??? }},
??? Selector: &label,
? },
})        

The service type is LoadBalancer and the target port is 80. Nothing fancy! Next I add a config map for the index.html file that will be provided to Nginx in a later step:

cm := k8s.NewKubeConfigMap(chart, jsii.String("index.html"), { &k8s.KubeConfigMapProps
? Data: &map[string]*string{
??? "index.html": jsii.String("<html><h1>Version 1</h1></html"),
? },
})        

Finally, I add the deployment object:

k8s.NewKubeDeployment(chart, jsii.String("deployment"), { &k8s.KubeDeploymentProps
? Spec: &k8s.DeploymentSpec{
??? Replicas: jsii.Number(3),
??? Selector: &k8s.LabelSelector{
????? MatchLabels: &label,
??? },
??? Template: &k8s.PodTemplateSpec{
????? Metadata: &k8s.ObjectMeta{
??????? Labels: &label,
????? },
????? Spec: &k8s.PodSpec{
??????? Containers: &[]*k8s.Container{{
????????? Name: jsii.String("webserver"),
????????? Image: jsii.String("nginx:1.23.2"),
????????? Ports: &[]*k8s.ContainerPort{{ContainerPort: jsii.Number(8080)}},
????????? VolumeMounts: &[]*k8s.VolumeMount{
??????????? {
????????????? Name: volName,
????????????? MountPath: jsii.String("/usr/share/nginx/html/"),
??????????? },
????????? },
??????? }},
??????? Volumes: &[]*k8s.Volume{
????????? {
??????????? Name: volName,
??????????? ConfigMap: &k8s.ConfigMapVolumeSource{
????????????? Name: cm.Name(),
??????????? },
????????? },
??????? },
????? },
??? },
? },
})        

Here comes a moment where I would like to pause and reflect on the complexity of the deployment resource. I am used to working with yaml files directly, alternatively using the Helm package manager. So at first glance cdk8s seems to be a lot more verbose than just working with yaml. The benefit of this approach is of course that we have the strength of a programming language and that everything is typed so that it is difficult to make trivial mistakes that are easy to do in yaml.

The important parts of the deployment is that I add my config map as a volume and I mount this volume in my container. I use nginx:1.23.2 as my container image. Finally, I expose port 8080. The next step is to set up a GitHub Actions workflow that will build my Kubernetes manifest files and commit them to my manifest repository.

Set up GitHub Actions to build Kubernetes manifests

I want my GitHub Actions workflow to be triggered whenever there is a push on main:

on:
? push:
??? branches:
????? - "main"        

Next I want to checkout both of my repositories. I place the repository that generates my manifests in the src directory, and I place the repository that will store my manifests in the destination directory. I persist-credentials for the second repository because this is where I will do a git push later in the workflow.

- name: Check out source repository
? uses: actions/checkout@v3
? with:
??? path: src
??? persist-credentials: false
- name: Checkout out destination repository
? uses: actions/checkout@v3
? with:
??? path: destination
??? repository: mattias-fjellstrom/gitops-cdk8s-argocd-manifests
??? token: ${{ secrets.GITOPS_REPO_PAT }}
??? fetch-depth: 0
??? persist-credentials: true        

To be able to work with the second repository (i.e. the repository that is external to this workflow) I need to add a GitHub Personal Access Token (PAT) with read/write access to the repository. I generate such a token and store it in a GitHub secret named GITOPS_REPO_PAT. Next, I install cdk8s and run it to generate my manifests:

- run: yarn global add cdk8s-cli
- working-directory: src
? run: cdk8s import
- working-directory: src
? run: cdk8s synth        

After these three steps I have an output directory in src/dist that contains my manifests. I keep saying manifests in plural, but in reality there is only a single manifest file since I only created a single chart in my source code. Remember? In the final steps I copy the content of the dist directory to my other repository and push the changes:

- run: |
??? mkdir -p destination/dev
??? cp -rv src/dist/*.yaml destination/dev?? ?
- working-directory: destination
? run: |
??? git config --global user.name "${{ github.actor }} [bot]"
??? git config --global user.email "${{ github.actor }}@bot.com"
??? git commit --allow-empty -am "bot: update from ${{ github.sha }}"
??? git push?? ?        

I need to configure git with user.name and user.email to be able to push. Now I have workflow that can do all the things I want it to do. The next step is to add Argo CD and have it watch my manifest repository for changes. Before I do that, I commit my changes to have the first set of manifests generated. See, I am still using plural!

Set up a Kubernetes cluster with minikube, install Argo CD, and configure my application

If you read my previous article on What is GitOps then the following steps will be familiar to you.

I use minikube to set up a local kubernetes cluster:

$ minikube start        

Once my cluster is up and running I create a Kubernetes namespace for Argo CD and install Argo CD in this namespace:

$ kubectl create namespace argocd
$ kubectl apply \
??? -n argocd \
??? -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml        

To communicate with the Argo CD API server it must be exposed in some way. I set an environment variable that adds required flags to all argocd commands. This will expose the Argo CD API server on-the-fly for each command:

$ export ARGOCD_OPTS='--port-forward --port-forward-namespace argocd'        

The default username for Argo CD is admin, and I get the password from a Kubernetes secret object that Argo CD created for me:

$ PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)        

I login to Argo CD with the credentials I just fetched using the CLI:

$ argocd login \
??? --name gitops \
??? --username admin \
??? --password $PASSWORD        

I create my application named website in Argo CD with the following command:

$ cd ~/code/gitops-cdk8s-argocd-manifests
$ REPO_URL=$(gh repo view --json url --jq .url)
$ argocd app create website \
??? --repo $REPO_URL \
??? --path dev \
??? --dest-namespace default \
??? --dest-server https://kubernetes.default.svc \
??? --auto-prune \
??? --self-heal \
??? --sync-policy auto        

There we have it. I can trigger the whole flow by making a change to my cdk8s project. The change will result in a new manifest being committed to my manifest repository and Argo CD will discover this change and apply it in my cluster.