Enhancing Secret Management: Multi-Node HashiCorp Vault Setup with High Availability on Kubernetes

What is HashiCorp Vault?

HashiCorp Vault is an open-source tool designed for securely storing and managing sensitive information such as secrets (API keys, passwords, certificates, etc.), tokens, and encryption keys. Vault provides a unified interface to manage these secrets securely across different environments, including cloud infrastructure, applications, and services. It can handle dynamic secrets, encryption as a service, and supports various authentication methods to control access.

Why Use HashiCorp Vault?

1. Secure Secret Management: Vault securely stores secrets like passwords, API keys, and certificates.
2. Dynamic Secrets: Automatically generates short-lived credentials, reducing the risk of exposure.
3. Access Control: Granular policies define who can access which secrets.
4. Encryption as a Service: Offers encryption for sensitive data without exposing keys.
5. Secret Leasing & Revocation: Allows automatic expiration and revocation of secrets.
6. Audit Logging: Provides detailed logs for compliance and monitoring.
7. Multi-Environment Support: Works with cloud, on-prem, and hybrid environments, with diverse authentication methods like LDAP, Kubernetes, AWS, etc.

Prerequisites for Setting Up HashiCorp Vault with Helm:

1. Kubernetes Cluster:

• A running Kubernetes cluster
• Ensure you have access to the cluster with kubectl.

2. Helm Installed:

• Helm 3.x installed to manage and install charts on your Kubernetes cluster.
• Install Helm via the official instructions https://helm.sh/docs/intro/install/

3. Add the official HashiCorp Vault Helm repository to Helm:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update        

Step-by-Step Setup Guide

Step 1: Pull and Customize the Helm Chart

Pull the Vault Helm chart and modify the custom-values.yaml file to enable high availability (HA) and configure the Raft backend:

#custom-values.yaml
server:
  affinity: ""
  ha:
    enabled: true
    raft:
      enabled: true        

HashiCorp Vault uses Raft for its storage backend to ensure high availability, data replication, and strong consistency across a cluster of Vault nodes. Raft allows Vault to operate without external dependencies, making it easy to manage and scale while ensuring fault tolerance and reliable recovery during failures.

Step 2: Install the Helm Chart

Install Vault using the customized values file:

helm install vault hashicorp/vault -f custom-values.yaml --namespace vault --create-namespace        

Step 3: Verify the Vault Pods

Check the status of the Vault pods:

kubectl get all -n vault
NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-0                                 0/1     Running   0          4m13s
pod/vault-1                                 0/1     Running   0          4m13s
pod/vault-2                                 0/1     Running   0          4m13s
pod/vault-agent-injector-7bcc447788-f8m66   1/1     Running   0          4m13s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   10.43.30.75     <none>        8200/TCP,8201/TCP   4m15s
service/vault-active               ClusterIP   10.43.39.184    <none>        8200/TCP,8201/TCP   4m15s
service/vault-agent-injector-svc   ClusterIP   10.43.196.201   <none>        443/TCP             4m15s
service/vault-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP   4m15s
service/vault-standby              ClusterIP   10.43.204.153   <none>        8200/TCP,8201/TCP   4m15s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           4m15s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-7bcc447788   1         1         1       4m15s

NAME                     READY   AGE
statefulset.apps/vault   0/3     4m15s        

You should see pods with the status 0/1. This indicates that the Vault pods are running but not initialized or unsealed yet.

kubectl exec -n vault vault-0 -- vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.17.2
Build Date         2024-07-05T15:19:12Z
Storage Type       raft
HA Enabled         true        

Step 4: Initialize Vault

To initialize Vault (starting with vault-0):

kubectl exec -n vault vault-0 -- vault operator init        

This command will generate 5 unseal keys and an initial root token:

Unseal Key 1: <Unseal Key1>
Unseal Key 2: <Unseal Key2>
Unseal Key 3: <Unseal Key3>
Unseal Key 4: <Unseal Key4>
Unseal Key 5: <Unseal Key5>

Initial Root Token: <Root Token>

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.        

Important: Store these keys securely, as you will need at least 3 keys to unseal the Vault in case of a restart or reseal.

Step 5: Unseal the Vault

Unseal the Vault by using three of the unseal keys on each Vault pod:

kubectl exec -n vault vault-0 -- vault operator unseal <Unseal Key1>
kubectl exec -n vault vault-0 -- vault operator unseal <Unseal Key2>
kubectl exec -n vault vault-0 -- vault operator unseal <Unseal Key3>        

Repeat the unseal process for the other Vault pods:

kubectl exec -n vault vault-1 -- vault operator unseal <Unseal Key1>
kubectl exec -n vault vault-1 -- vault operator unseal <Unseal Key2>
kubectl exec -n vault vault-1 -- vault operator unseal <Unseal Key3>        

kubectl exec -n vault vault-2 -- vault operator unseal <Unseal Key1>
kubectl exec -n vault vault-2 -- vault operator unseal <Unseal Key2>
kubectl exec -n vault vault-2 -- vault operator unseal <Unseal Key3>        

Step 6: Check Vault Status

To confirm the unseal status, check Vault’s status:

kubectl exec -it -n vault vault-0 -- sh

/ $ vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.17.2
Build Date              2024-07-05T15:19:12Z
Storage Type            raft
Cluster Name            vault-cluster-20cbbb3a
Cluster ID              2d624502-6b6f-8d02-04f5-d2efc228e2e0
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2024-10-03T11:02:52.601536479Z
Raft Committed Index    54
Raft Applied Index      54
/ $ vault login
Token (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.        

Step 7: Join the Vault Nodes for High Availability

Join the other Vault pods (vault-1 and vault-2) to form the Raft cluster:

kubectl exec -it -n vault vault-1 -- sh
vault operator raft join https://vault-0.vault-internal:8200

kubectl exec -it -n vault vault-2 -- sh
vault operator raft join https://vault-0.vault-internal:8200        

Step 8: Check the Raft Cluster Status

Verify that the Vault pods have successfully joined the Raft cluster:

k exec -it -n vault vault-0 -- sh

/ $ vault operator raft list-peers
Node                                    Address                        State       Voter
----                                    -------                        -----       -----
2cc4d3af-9ddf-4a26-449e-09835fef74c4    vault-0.vault-internal:8201    leader      true
ee47b694-f5c8-702e-6e4d-705480f1f41b    vault-1.vault-internal:8201    follower    true
627c75a9-ebd0-0e41-a73c-c368c6bc8a5d    vault-2.vault-internal:8201    follower    false        

You should see output showing the leader and followers in the cluster.

Step 9: Verify All Resources

To ensure everything is set up correctly, check all resources in the vault namespace:

kubectl get all -n vault        

To expose the HashiCorp Vault UI via Ingress and secure it with TLS, follow these steps. Be sure to have Cert-Manager installed to handle certificate provisioning via Let’s Encrypt.

Step 10: Expose the Vault UI Using Ingress

1. Set up Cert-Manager to issue certificates using Let’s Encrypt for TLS. You can follow this guide: Let’s Encrypt Certificate Using Cert-Manager on a Kubernetes Cluster.

https://medium.com/@bhojeshwar.sahu/lets-encrypt-certificate-using-cert-manager-on-a-kubernetes-cluster-a851c3a67950

2. Create the Ingress resource to expose Vault’s UI. Here’s an example Ingress manifest that routes traffic to the Vault service and uses TLS

# vault-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault-ingress
  namespace: vault
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: example.com  # Replace with your domain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vault
                port:
                  number: 8200
  tls:
    - hosts:
        - example.com  # Replace with your domain
      secretName: ingress-cert  # Replace with the TLS secret name created by Cert-Managerkubectl apply -f vault-ingress.yaml        

Apply the Ingress resource to your Kubernetes cluster:

kubectl apply -f vault-ingress.yaml        

The Vault UI should now be accessible at https://example.com in your browser, with a valid TLS certificate provided by Let's Encrypt.

After completing these steps, you will have a secure, highly available, and TLS-enabled HashiCorp Vault UI running on Kubernetes.

Conclusion

By following these steps, you have successfully set up a multi-node HashiCorp Vault cluster with high availability, using Raft for data replication and consistency. This setup ensures that your secrets management system is resilient, scalable, and secure on Kubernetes. With the Vault UI exposed securely via Ingress, managing and monitoring secrets becomes even more streamlined.

For more insights and detailed explanations, feel free to reach out or explore further in the HashiCorp Vault documentation!

Deploy, secure, and automate!

Happy deploying! ??

Happy Kubernetings! ?

Written by Bhojeshwar Sahu