How to use Kubernetes External Secrets (KES) with Vault

Acknowledge

This is the first of a series of articles where I am going to share all my lessons learned by working on the Open Source project called From Monolith to Kubernetes.

The repository shows a Cloud-Native application which uses the following technologies and frameworks: Camunda Cloud, Zeebe, SpringBoot, SpringCloud, Kubernetes, Knative, GCP, etc. I do recommend people trying out the examples described in the workshop, as it is a very hands on tutorial. You can learn about Microservices patterns, architecture and different tools that can save your time. If you are willing to learn these technologies and how to work in Open Source projects and help to make these projects better, you can contact me or Mauricio Salaboy if you want guidance. I started working with Mauricio Salatino's help, and I started my Open Source journey, then... I Believe that by sharing a bit of what I learned other people in the community can benefit from it.

Hashicorp Vault

Hashicorp Vault is an open-source Secret Management tool created by Hashicorp, you can use it via UI, Command Line Interface, or using an HTTP API. Vault comes with some pluggable components called Authentication Methods and Secrets Engines. In our sample, we are gonna use the Authentication method Kubernetes and the Secret Engine KV.

I find two very interesting things about Vault:

• Centralized data for secret management - It is very interesting, imagine an application with twenty or more different services, imagine how many secrets do you have to handle. Vault centralizes all secrets and it can be clustered for a HA mode.
• Dynamic Secrets - It helps you to rotate your credentials. In my course on Caelum, I learnt about the Space Codes case, which teach me that is very important to rotate your secrets constantly.

While I found Vault very useful, as usual, by adding a new component to our application the complexity increases.

For more details about Vault, see the official documentation.

Secrets on Kubernetes

" Kubernetes provides a special object type which purpose is to save secret data: The Secret Resource Type.."

In the previous image, we can see an example of how to create a Secret object in Kubernetes. We can use this Secret in one of our Pods by consuming it through env variables:

Kubernetes encrypts our Secrets using base64 encoding and this is not enough, as anyone with access can decode our secrets regarding access, you can use Kubernetes RBAC mechanisms to make sure that only authorized people can access your secrets. Another very interesting tool is DEX, I'll leave here an article speaking about this tool.

The Kubernetes have encryption at rest, which means that if any people to have direct access to Kubernetes's etcd database, will not possible see the secret data, just the Kubernetes API can do it.

Quick tip

A secret is sensible data, confidential data that our applications or infrastructure use. Can be our connection string to access our database, our credit card data, our user and password or a Token of an API, etc.

Then, don't hardcode your secrets, by example:

Or in your configuration file like an application-prod.properties if you know SpringBoot:

Let's go to practical

Now, we'll see in practice, how to install and configure one simple environment using Kubernetes, Vault, and a SpringBoot application. We will divide this into four steps:

1. Creating our Kubernetes's cluster using Kubernetes KIND
2. Installing and configuring Vault
3. Creating secrets on Vault
4. Finding secrets and injecting it in our Springboot application

Tools required

• Docker
• Kind (If you use Windows 10, you need to configure Docker WSL2)
• Helm
• Kubernetes
• Git

1. Creating the Kubernetes cluster using KIND

Using a terminal, copy and paste this command:

$ kind create cluster --name vault

Expected result:

Creating cluster "vault" ...
 ? Ensuring node image (kindest/node:v1.19.1) ??  ...
 ? Ensuring node image (kindest/node:v1.19.1) ??
 ? Preparing nodes ??   ...
 ? Preparing nodes ??
 ? Writing configuration ??  ...
 ? Writing configuration ??
 ? Starting control-plane ???  ...
 ? Starting control-plane ???
 ? Installing CNI ??  ...
 ? Installing CNI ??
 ? Installing StorageClass ??  ...
 ? Installing StorageClass ??
Set kubectl context to "kind-vault"
You can now use your cluster with:
kubectl cluster-info --context kind-vault
Not sure what to do next? ??  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

What did it just happened? Basically, it created a Kubernetes cluster docker containers. It gives us some advantages, one of them is that it makes it easy to test and create a local Kubernetes environment (in your laptop). You can now avoid using tools like docker-compose which are very different from a real Kubernetes Cluster running on a Cloud Provider. You start with Kubernetes and ends with Kubernetes.

Let's now use our cluster Kubernetes:

$ kubectl cluster-info --context kind-vault


Kubernetes master is running at https://127.0.0.1:54108
KubeDNS is running at https://127.0.0.1:54108/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

2. Installing and configuring Vault

In this step, we'll use Helm to install the Vault in our cluster, it is very easy:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

$ helm install vault hashicorp/vault --set "injector.enabled=false"

Helm helps us a lot, it has a role to facilitate the creation, distribution, reuse, and maintainability of complex applications running on Kubernetes.

If you noted, we opted to disable the Vault injector. We'll use a Kubernetes-External-Secrets to get the Secrets.

Let's verify if the Vault is running:

$ kubectl get pods

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          109s

Hm... We can see that it is running, but is not ready yet (READY 0/1)

When we starting the vault, it starts with a sealed state.

Before any operations are made on Vault, it needs to be in an unsealed state.

To do that, let's use the terminal:

$ kubectl exec -it vault-0 sh

After, we'll start it using this command:

$ vault operator init


Unseal Key 1: C7YID5r2Ku6iAcx1BgM2UUbfH9aJjCd31P4w4daPiMNi
Unseal Key 2: Mn0A4OXN2nus4KszCBCr/c11C8M0msoiN6iDrtrnrMTE
Unseal Key 3: JfViVhphP12fbinMjttJVSaCU8svKjfLL/UD5KlN/vVd
Unseal Key 4: K+To9Jz0zAhZ8bheZz5S4JaGJndmpq4qHSy1z6+hBVBJ
Unseal Key 5: g4h7BxMP88oTd9fsmtmgwaErQz2WTBt930P57ZxcrvCW

Initial Root Token: s.WRCOyrpRLmeSgcAqFDe2A0rx

...

We can see that Vault generated five keys and one principal token. Let's see what documentation say...:

"The default Vault config uses a Shamir seal. Instead of distributing the unseal key as a single key to an operator, Vault uses an algorithm known as Shamir's Secret Sharing to split the key into shards. A certain threshold of shards is required to reconstruct the unseal key, which is then used to decrypt the master key."

2.1 Modifying the Vault to unseal state

After knowing the technique used by Vault, let's use that in practice by changing the Vault state to unseal. To do that, we'll access out Pod's terminal and to use the Vault's CLI.

$ vault operator unseal 
Unseal Key (will be hidden):

<Paste one of secrets here. Do not repeat>

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       eb57ac94-1d41-2db7-3c11-2e400cfd85e9
Version            1.5.4
HA Enabled         false

We can see the key Unseal Progress the value 1/3. It means that Vault recognized one part of the master key, now we need to use more two keys.

$ vault operator unseal
Unseal Key (will be hidden):
<Paste one of secrets here. Do not repeat>               

## hidden

vault operator unseal
Unseal Key (will be hidden):
<Paste one of secrets here. Do not repeat>
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.5.4
Cluster Name    vault-cluster-170fbc4d
Cluster ID      14c4d559-067d-0d85-0a03-cd1e99cbce2f
HA Enabled      false

After using three keys, we can see that the field Sealed is false, and if we running the command kubectl get pods, we'll see the Pod are running and ready.

2.2 Configuring Secrets Engine and Auth Methods

As we saw in the "Hashicorp Vault" section, Vault works with Secrets Engines and Auth Methods:

"Secrets engines are components which store, generate or encrypt data... Auth methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user."

The Secret Engine that we'll use is Key/Value and the Auth Method is Kubernetes.

Let's access the Vault's terminal:

$ kubectl exec -it vault-0 sh

We need to use the Root Token to log in:

$ vault login s.WRCOyrpRLmeSgcAqFDe2A0rx

Is very simple to activate the Kubernetes Auth Method:

$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

"The Kubernetes Auth Method can be used to authenticate with the Vault using Kubernetes Service Account Token, the Vault accepts this Token by any client within the Kubernetes cluster. There is authentication by the Vault to see if the token is valid to access the configured Kubernetes path."

Let's configure the Kubernetes path, paste and run this command:

$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert are created to the container by Kubernetes when is started. The env variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal Kubernetes network address.

2.3 Creating Policies

Policies in Vault basically defines which a user can access which resource.

$ vault policy write fmtok8s-policy - <<EOF
path "secret*" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: fmtok8s-policy

More about policies here.

2.4 Creating Roles

Creating a role is easy, see:

$ vault write auth/kubernetes/role/fmdtok8s-role \
    bound_service_account_names=fmtok8s-auth \
    bound_service_account_namespaces=default \
    policies=fmtok8s-policy \
    ttl=1h

Success! Data written to: auth/kubernetes/role/fmtok8s-role

Here, we are defining a role in auth/kubernetes/role/fmtok8s-role authorizing a Kubernetes Service Account object with name fmtok8s-auth on bound_service_account_names=fmtok8s-auth on namespace default on bound_service_account_namespaces=vault with ttl 1h.

2.5 Enabling a Secret Engine KV

Let's enable the Secret Engine KV, with command:

$ vault secrets enable -path=secret/ kv

3. Creating our secrets on Vault

If you saw, Vault is path based, then the secret creation will not be different:

$ vault kv put secret/data/fmtok8s username=mysecretsecret

Success! Data written to: secret/data/fmtok8s

4. Finding our secrets and injecting it in our Springboot application

We haven’t installed Kubernetes-External-Secrets in our cluster yet.

Let's install it using Helm:

$ helm install kubernetes-external-secrets external-secrets/kubernetes-external-secrets --skip-crds --set env.VAULT_ADDR=https://vault:8200 --set serviceAccount.name=fmtok8s-auth

If you are not using Helm V3 the parameter --skip-crds is not necessary (See documentation).

We need to set Vault host (--set env.VAULT_ADDR) for Kubernetes-External-Secrets to know where Vault is. You are instructing Helm to create a ServiceAccount with name fmtok8s-auth on --set serviceAccount.name=fmtok8s-auth the same one we mapped when we created the role.

4.1 Installing our SpringBoot application

$ git clone https://github.com/mcruzdev/fmtok8sjvault
$ cd fmtok8s
$ kubectl apply -f ./k8s

You can access all files here. Let's verify if all Pods are running:

$ kubectl get pods 
NAME                                           READY   STATUS    RESTARTS   AGE
fmtok8sjvault-deployment-78896df6df-7ptjv      1/1     Running   0          76s
kubernetes-external-secrets-5c49879bbf-s66gm   1/1     Running   0          13m
vault-0                                        1/1     Running   0          100m

Let's map our SpringBoot port:

$ kubectl port-forward fmtok8sjvault-deployment-78896df6df-7ptjv 8080:8080

Now, accessing the SpringBoot Application, you can see the secret username wrote on Vault.

If you want to see the same steps in action and contribute, you can look at the services and technologies inside the application provided in that Github repo.

I hope that you enjoyed the reading. Thank you??

References: “Cloud Native DevOps with Kubernetes by John Arundel and Justin Domingus (O’Reilly). Copyright 2019 John Arundel and Justin Domingus, 978–1–492–04076–7.”, https://www.vaultproject.io/docs