GitOps - Part 2

In the previous?post, we looked at how to use fluxv2 for deploying apps through helm and kustomization. In this we will take a brief look at how to secure your app secrets using Mozilla SOPS. There are many ways for securing the secrets and one of the ways was to use the secret store CSI driver which was covered in this post and the other popular way is to use sealed secrets for which details can be found here. These approaches come with a step of installing the extra component which acts as a controller for securing the secret.

We will look at Mozilla SOPS which provides the CLI to encrypt secrets with all the major cloud provider without having to install additional component and we will see the steps for using the same in Azure.

Create a variables which will be used for the scripts.

export?GITHUB_TOKEN=xxxxxx
export?RESOURCE_GROUP_NAME=gitopssops
export?LOCATION=southeastasia
export?CLUSTER_NAME=gitopssops
export?GITHUB_USER=girishgouda
export?GITHUB_REPO=githubsops        

Creation of AKS cluster with managed identity enabled

az?group?create?-n?$RESOURCE_GROUP_NAME?-l?$LOCATION
az?aks?create?-g?$RESOURCE_GROUP_NAME?-n?$CLUSTER_NAME?--enable-managed-identity
az?aks?get-credentials?-g?$RESOURCE_GROUP_NAME?-n?$CLUSTER_NAME
kubectl?get?nodes        

Install or bootstrap flux?in the cluster.

kubectl?create?ns?flux-system
flux?bootstrap?github?\
--owner=$GITHUB_USER?\
--repository=$GITHUB_REPO?\
--branch=master?\
--path=./clusters/$CLUSTER_NAME        

Create a kustomization which will point to manifests folder to synchronize yaml files to Kubernetes. This will be used to have encrypted secret.

flux?create?kustomization?demoapp?\
??--namespace=flux-system?\
??--source=flux-system?\
??--path="./manifests"?\
??--prune=true?\
??--validation=client?\
??--interval=5m?\
??--export?>?./clusters/$CLUSTER_NAME/demoapp-kustomization.yaml        

Create a manifests directory

mkdir?manifests        

Create a namespace called demoapp where the secret will be created

cat?>?./manifests/namespace.yaml?<<EOF
---
apiVersion:?v1
kind:?Namespace
metadata:
??name:?demoapp
EOF
        

Install the AAD pod identity using helm in a namespace aad-pod-idnetity as shown below.

cat?>?./clusters/gitops/aad-pod-identity.yaml?<<EOF
---
apiVersion:?v1
kind:?Namespace
metadata:
??name:?aad-pod-identity
---
apiVersion:?source.toolkit.fluxcd.io/v1beta1
kind:?HelmRepository
metadata:
??name:?aad-pod-identity
??namespace:?aad-pod-identity
spec:
??url:?https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
??interval:?10m
---
apiVersion:?helm.toolkit.fluxcd.io/v2beta1
kind:?HelmRelease
metadata:
??name:?aad-pod-identity
??namespace:?aad-pod-identity
spec:
??interval:?5m
??chart:
????spec:
??????chart:?aad-pod-identity
??????version:?4.0.0
??????sourceRef:
????????kind:?HelmRepository
????????name:?aad-pod-identity
????????namespace:?aad-pod-identity
??????interval:?1m
??values:
????nmi:
??????allowNetworkPluginKubenet:?true
EOF        

Push the changes to git repo and verify that the aad-pod-identity and demoapp are created.

Create a azure key vault and create a key which will be used for encryption and decryption of secret

export?KEY_VAULT_NAME=gitopsazurevault
az?keyvault?create?--name?$KEY_VAULT_NAME?--resource-group?$RESOURCE_GROUP_NAME?--location?$LOCATION
az?keyvault?key?create?--name?sops-key?--vault-name?$KEY_VAULT_NAME?--protection?software?--ops?encrypt?decrypt
az?keyvault?set-policy?--name?$KEY_VAULT_NAME?--resource-group?$RESOURCE_GROUP_NAME?--object-id?$OBJECT_ID?--key-permissions?encrypt?decrypt
az?keyvault?key?show?--name?sops-key?--vault-name?$KEY_VAULT_NAME?--query?key.kid        

The last command will provide a key "https://gitopsazurevault.vault.azure.net/keys/sops-key/xxxxx" which will be used for encryption and decryption.

Next, we need to tell kustomize controller to use the managed identity of the cluster to access the key for decryption.

First, we create a azureidentity with the details of the cluster manged identity and then bind the identity to selector called sops-aks-decryptor as shown.

cat?>?./clusters/gitops/sops-identity.yaml?<<EOF
---
apiVersion:?aadpodidentity.k8s.io/v1
kind:?AzureIdentity
metadata:
??name:?sops-akv-decryptor
??namespace:?flux-system
spec:
??clientID:?xxxxxx #client id of  the cluster managed identity 
??resourceID:?/subscriptions/subId/resourcegroups/MC_gitopssops_gitopssops_southeastasia/providers/Microsoft.ManagedIdentity/userAssignedIdentities/gitopssops-agentpool
??type:?0?#?user-managed?identity
---
apiVersion:?aadpodidentity.k8s.io/v1
kind:?AzureIdentityBinding
metadata:
??name:?sops-akv-decryptor-binding
??namespace:?flux-system
spec:
??azureIdentity:?sops-akv-decryptor
??selector:?sops-akv-decryptor??#?kustomize-controller?label?will?match?this?name
EOF        

Note: we need to specify the clientId of the managed identity which was created with the cluster creation and also the resourceId of the managed identity.

Next, we patch the kustoimze controller by modifying the kustomization.yaml to include patch yaml as shown below.

apiVersion:?kustomize.config.k8s.io/v1beta1
kind:?Kustomization
resources:
-?gotk-components.yaml
-?gotk-sync.yaml
patchesStrategicMerge:
-?sops-kustomize-patch.yaml        

The patch file will tell the kustomize controller to use azure msi for authentication.

--
apiVersion:?apps/v1
kind:?Deployment
metadata:
??name:?kustomize-controller
??namespace:?flux-system
spec:
??template:
????metadata:
??????labels:
????????aadpodidbinding:?sops-akv-decryptor??#?match?the?AzureIdentityBinding?selector
????spec:
??????containers:
??????-?name:?manager
????????env:
????????-?name:?AZURE_AUTH_METHOD
??????????value:?msi-        

Note: The same selector sops-aks-decryptor is used for aadpodidbinding

Finally, we need to tell kustomize-controller to use SOPS as decryption provider as shown below.

Install SOPS and create a SOPS conf file which will be used for encrypting the secret. The configuration file contains the details of path and the sections of the files to be encrypted using the key as shown below.

cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    azure_keyvault: https://gitopsdemokeyvault.vault.azure.net/keys/sops-key/xxxxxx
EOF        

Create a temporary secret file which will not be pushed to git.

cat > ./secret.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: demoapp-credentials
  namespace: demoapp
type: Opaque
stringData:
  username: admin
  password: password
EOF        

Note : SOPS CLI will use the logged in user in AZ CLI and hence make sure the logged in user has "encrypt" and "decrypt" key permissions.

Encrypt the secret.yam file as shown below.

sops --encrypt secret.yaml > ./manifests/secret.enc.yaml        

The encrypted file will be as shown below.

apiVersion:?v1
kind:?Secret
metadata:
????name:?demoapp-credentials
????namespace:?demoapp
type:?Opaque
stringData:
????username:?ENC[AES256_GCM,data:BYkU34g=,iv:DMTnv3Ucd5xSGHGbEIBIeypKydHwxTnYl3enK0nKdjw=,tag:vagn0DVSfjOSaDGVRfLdMg==,type:str]
????password:?ENC[AES256_GCM,data:Fey4wZyIvB4Jvg==,iv:gFtSzrtuC5MfC9OVkPNIYZ4MiYGX9Xsy81fZPUsZzIM=,tag:GW4hSN9U+YcYYwytCZKHZQ==,type:str]
sops:
????kms:?[]
????gcp_kms:?[]
????azure_kv:
????????-?vault_url:?https://gitopsazurevault.vault.azure.net
??????????name:?sops-key
??????????version:?xxxxxxxxx
??????????created_at:?"2021-09-14T05:25:57Z"
??????????enc:?ODebW_-5BhT2LdEzWrC9NFxA4A6INoTEVesUmT_9ECA0hKb4-nSEj-QDcHigjJZpGh3qUm6jZSGbjQpcPdE8eHYLlpmgb8sU4nDn3P8_Gm9KBB1txSg3fDTgEl0HEs6Re3zu7VQo0gVTbnZN-qyLRZbVvqN6NnK7npWB2SXLJNDucHiqDqBDluJxkKhM0ngTh2qvPp7MzQZ1L9A9P0tLYrE8btBJYLzGUKRFc_pbfTPMPxtoei00TShWjFhtO3WNQ9tzqJz0Ajy7ITn5ntu2YDpbE9jqsAe3k6yqeKRIZ8TbHPHm329dNhlUiQFujpVVy-ypUkyg
????hc_vault:?[]
????age:?[]
????lastmodified:?"2021-09-14T05:26:00Z"
????mac:?ENC[AES256_GCM,data:P/54bG7GUxilkhJ2gbHlW2jUA9wcL7a8Wp4cyupyaOHdyB9hRtWYkFsnBblQLi7t5QMe7XJEOisZR45l1HOHZVgvZUg5ghI/4xLmc0eC/krkqv5CNeH5JyAEuq3yr9o/nLy0+EjgNIrc7tLl2Xo8Qn3VdR1KytWgLXpK0FORwcE=,iv:1FylnUvw3RBn/mR+ZhBwukIh+TbBxvD6irsoYUTRVCE=,tag:KI3qIxwyJWKAkkX9lctG4w==,type:str]
????pgp:?[]
????encrypted_regex:?^(data|stringData)$
????version:?3.7.1

        

We can now remove the secret.yaml and push the secret.enc.yaml to git

We can finally verify the sceret being created by using the folowing command

kubectl describe secret -n demoapp demoapp-credentials        

This is how we can use sops in Fluxv2 to safely push the secrets to git. In the future articles, we will look at some more interesting features.