Kubernetes Secrets & Security – Protecting Sensitive Data

Welcome to Day 14 of my 100-day challenge! Today, we dive into one of the most critical aspects of Kubernetes security: secrets management.

In a cloud-native environment, ensuring the protection of sensitive data like API keys, credentials, and tokens is essential to avoid potential breaches and unauthorized access.

Kubernetes, by default, provides mechanisms for storing and managing secrets, but understanding the best practices, tools, and strategies to protect them at rest and in transit is key to building a secure and resilient system.

In this post, we'll explore the best tools, techniques, and methods for securing secrets in Kubernetes, from leveraging the Secrets CSI Driver for safe storage to using advanced policy enforcement with OPA and Kyverno.

Plus, we’ll discuss the importance of external secret management solutions, such as HashiCorp Vault and Cloud-native managers like AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault.

Whether you're new to Kubernetes or looking to bolster your cluster's security, this guide will provide you with valuable insights to keep your secrets safe and avoid critical vulnerabilities.

Let's get started and take a deep dive into Kubernetes secrets security!

Understanding Kubernetes Secrets

1. Introduction to Kubernetes Secrets

Kubernetes Secrets are a way to store and manage sensitive data such as passwords, OAuth tokens, SSH keys, API keys, and other credentials securely.

Instead of storing sensitive information in plain text within ConfigMaps or environment variables, Kubernetes provides Secrets as a secure and encrypted mechanism to manage confidential data.

Why Use Kubernetes Secrets?

• Prevents exposing sensitive information in configuration files.
• Enhances security by keeping sensitive data separate from application configurations.
• Controls access to secrets with Kubernetes Role-Based Access Control (RBAC).
• Enables automated secret rotation and updates without redeploying applications.

? Encoding vs. Encryption → Kubernetes stores secrets base64-encoded but not encrypted by default.

2. Types of Kubernetes Secrets

Kubernetes supports different types of Secrets, including:

Opaque (Default Type)

• Used for arbitrary key-value pairs.
• Example: API keys, passwords, database credentials.

Service Account Token Secrets

• Contains a token for accessing the Kubernetes API.
• Automatically created for each service account.

Docker Config Secrets

• Stores credentials for accessing private container registries.
• Used in imagePullSecrets.

Basic Authentication Secrets

• Stores usernames and passwords for authentication.

SSH Authentication Secrets

• Stores SSH private keys for authentication to remote services.

TLS Secrets

• Stores TLS certificates and private keys for secure communication.

3. Creating Kubernetes Secrets

(A) Creating a Secret from a File

kubectl create secret generic my-secret --from-file=./username.txt --from-file=./password.txt
        

(B) Creating a Secret from Literal Values

kubectl create secret generic my-secret --from-literal=username=admin --from-literal=password=securepass
        

(C) Creating a Secret with YAML Definition

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=   # Base64 encoded "admin"
  password: c2VjdXJlcGFzcw==   # Base64 encoded "securepass"
        

To apply this secret:

kubectl apply -f secret.yaml
        

4. Accessing Kubernetes Secrets in Pods

There are two primary ways to use secrets in pods:

(A) Using Environment Variables

Modify the Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-container
          image: nginx
          env:
            - name: USERNAME
              valueFrom:
                secretKeyRef:
                  name: my-secret
                  key: username
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: my-secret
                  key: password
        

(B) Mounting as a Volume

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secret"
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: my-secret
        

This mounts the secret files under /etc/secret.

5. Decoding Kubernetes Secrets

Kubernetes stores secret values in Base64 encoding. To decode:

echo "YWRtaW4=" | base64 --decode
        

This will output admin.

6. Managing and Updating Secrets

Updating a Secret

If you update a secret, running pods won't automatically receive the new values. You need to manually restart the pod:

kubectl delete pod <pod-name>
        

To update a secret:

kubectl create secret generic my-secret --from-literal=password=newpassword --dry-run=client -o yaml | kubectl apply -f -
        

7. Deleting a Secret

To delete a secret:

kubectl delete secret my-secret
        

Storing & Managing Secrets Securely in Kubernetes

Managing secrets securely is critical in Kubernetes to prevent unauthorised access to sensitive data like API keys, database credentials, and private tokens.

Below are different approaches to storing secrets, their security implications, and step-by-step implementation.

1. Using Environment Variables vs. Mounted Files → Which is Safer?

Environment Variables

• Secrets can be injected into containers using environment variables.
• Example:

apiVersion: v1
kind: Pod
metadata:
  name: env-secret-pod
spec:
  containers:
    - name: my-container
      image: nginx
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password        

• Issues with Environment Variables: Stored in the process memory, making it accessible to any process inside the container.
• Security risk: If an attacker gains access to the container, they can extract secrets via env or /proc/$PID/environ.
• Cannot dynamically update secrets without restarting the pod.

Mounted Files (Preferred)

• Secrets can be mounted as files inside the container.
• Example:

apiVersion: v1
kind: Pod
metadata:
  name: file-secret-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secret"
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: db-secret        

• Advantages of Mounted Files:
• More secure: Not stored in process memory, reducing exposure.
• Auto-updates: If the secret is modified, the new value is available without restarting the pod.
• Supports file-based credentials like TLS certificates.

?? Best Practice: Use mounted files instead of environment variables to reduce exposure and enhance security.

2. RBAC (Role-Based Access Control) for Secrets

Why Use RBAC for Secrets?

By default, secrets in a Kubernetes namespace can be accessed by all service accounts.

To restrict access, use RBAC policies.

Step 1: Define a Role with Secret Access

Create a Role to allow a specific service account to read secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]        

Step 2: Bind the Role to a Specific User or Service Account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: my-app-sa
    namespace: default
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io        

Step 3: Use the Service Account in a Pod

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-app-sa
  containers:
    - name: my-container
      image: nginx        

?? Now, only my-app-sa has access to secrets, limiting exposure to unauthorized users.

3. Restricting Secret Visibility

Problem: By default, all pods in a namespace can access secrets.

Solution: Use RBAC and separate namespaces.

Step 1: Create a Namespace for Secrets

kubectl create namespace secret-ns
        

Step 2: Store Secrets in the Restricted Namespace

kubectl create secret generic db-secret --from-literal=password="mypassword" -n secret-ns
        

Step 3: Allow Specific Pods to Access the Secret

Use Role and RoleBinding (as shown earlier) only for required service accounts.

?? Now, secrets are protected within a dedicated namespace and aren’t exposed to all pods.

4. Using Secrets with Pods

Kubernetes Secrets can be injected into pods in two ways:

? As Environment Variables (easier, but less secure)

? As Mounted Files (recommended)

Method 1: Inject Secrets via Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: env-secret-pod
spec:
  containers:
    - name: my-container
      image: nginx
      env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: username
        

?? Pros: Simple to use.

?? Cons: Less secure, exposes secrets in process memory.

Method 2: Mount Secrets as Files (Preferred)

apiVersion: v1
kind: Pod
metadata:
  name: file-secret-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secret"
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: db-secret
        

?? Pros: More secure, auto-updates secrets without restarting.

?? Cons: Slightly more complex setup.

Best Practices for Secure Secret Management

? 1. Use Mounted Files Instead of Environment Variables

• Reduces exposure to process memory leaks.

? 2. Implement RBAC Policies

• Restrict secret access to only the required service accounts.

? 3. Use Namespaces for Secrets Isolation

• Store secrets in dedicated namespaces to minimize access.

? 4. Encrypt Secrets at Rest

• Enable Kubernetes Encryption at Rest: apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: ["secrets"] providers: - aescbc: keys: - name: key1 secret: <base64-encoded-key> - identity: {} Apply with: kubectl apply -f encryption-config.yaml

? 5. Use External Secret Management Tools

• Integrate Kubernetes with Vault, AWS Secrets Manager, or Azure Key Vault for enhanced security.

Encrypting Secrets at Rest & In Transit in Kubernetes

By default, Kubernetes stores Secrets unencrypted in etcd, making them accessible to anyone with etcd access.

To enhance security, we must encrypt secrets at rest (stored data) and in transit (network communication). This guide covers:

? Enabling etcd encryption for secrets

? Securing API communication with TLS

? Restricting etcd access to prevent unauthorized access

1. Enabling etcd Encryption for Secrets

Why?

• By default, Kubernetes stores Secrets in plain text in etcd.
• If an attacker gains access to etcd, they can extract all secrets.
• Enabling encryption ensures secrets are stored securely at rest.

Step 1: Create an Encryption Configuration File

The encryption is configured using the EncryptionConfiguration file.

1?? Connect to the Kubernetes Control Plane node (where kube-apiserver runs):

ssh user@master-node
        

2?? Create the encryption configuration file:

sudo nano /etc/kubernetes/encryption-config.yaml
        

3?? Add the following configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: a2V5LWZvci1lbmNyeXB0aW9uLXNlY3JldHM=  # Base64-encoded key
      - identity: {}
        

?? Explanation:

• aescbc: Uses AES-CBC encryption (strong encryption).
• secret: The Base64-encoded encryption key (use a secure key!).
• identity: Falls back to unencrypted storage if AES encryption fails.

Step 2: Apply the Encryption Configuration

Modify the kube-apiserver configuration to use this file.

1?? Edit the API server manifest (/etc/kubernetes/manifests/kube-apiserver.yaml):

sudo nano /etc/kubernetes/manifests/kube-apiserver.yaml
        

2?? Find the command section and add the encryption config file:

    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
        

3?? Save and exit, then restart the API server:

sudo systemctl restart kubelet
        

Step 3: Encrypt Existing Secrets

Once encryption is enabled, existing secrets remain unencrypted until they are re-written.

1?? Manually trigger re-encryption by re-creating secrets:

kubectl get secrets -A -o yaml | kubectl apply -f -
        

2?? Verify Secrets are now encrypted in etcd Check the etcd database:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret --prefix --keys-only
        

?? If you see Base64-encoded values instead of plain text, encryption is working!

2. Securing API Communication (Encrypting Secrets in Transit)

Why?

• API communication between kubectl, the API server, and etcd must be encrypted.
• If not secured, attackers can intercept secrets in transit.
• Kubernetes supports TLS encryption by default, but it must be correctly configured.

Step 1: Ensure API Server Uses TLS

Check if the API server is running with TLS enabled.

1?? View the kube-apiserver manifest:

sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
        

2?? Ensure the following flags are present:

    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
        

? If missing, add them and restart the API server:

sudo systemctl restart kubelet
        

Step 2: Verify TLS is Enforced

1?? Run the following command to check if the API server only allows HTTPS:

kubectl cluster-info
        

? If the API server uses https://, it's secure!

? If it allows https://, update kube-apiserver.yaml to disable insecure port 8080:

    - --insecure-port=0
        

2?? Test API security with curl:

curl -k https://<KUBE-API-SERVER>:6443/version
        

? If the response shows Kubernetes version info without errors, TLS is properly configured.

3. Restricting etcd Access

Why?

• etcd stores all Kubernetes cluster data including secrets.
• By default, anyone with etcd access can read secrets.
• We must restrict access to only the Kubernetes API server.

Step 1: Secure etcd with TLS

Check if etcd is using TLS encryption:

1?? View the etcd configuration file (/etc/kubernetes/manifests/etcd.yaml):

sudo cat /etc/kubernetes/manifests/etcd.yaml
        

2?? Ensure the following flags exist:

    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
        

? If missing, generate TLS certificates and update the configuration.

Step 2: Restrict etcd Access

1?? Disable direct access to etcd from outside the cluster

Modify etcd.yaml to listen only on localhost (127.0.0.1):

    - --listen-client-urls=https://127.0.0.1:2379
        

? This ensures only the Kubernetes API server can connect to etcd.

2?? Verify etcd is now protected:

ETCDCTL_API=3 etcdctl member list
        

? If unauthorised users get permission denied, access is restricted!

External Secret Management Solutions in Kubernetes

Kubernetes Secrets provide a way to store sensitive data, but they are stored in etcd, which requires manual encryption and strict RBAC controls.

External secret management solutions provide a more secure, automated, and scalable way to store and inject secrets into Kubernetes.

Below are the following solutions and how to implement them:

? HashiCorp Vault → Securely store and inject secrets into Kubernetes.

? AWS Secrets Manager, Azure Key Vault, GCP Secret Manager → Cloud-native solutions.

? Sealed Secrets (Bitnami) → Encrypt secrets before applying them to Kubernetes.

? External Secrets Operator → Sync secrets from external secret managers.

1. Using HashiCorp Vault for Kubernetes Secrets

Why HashiCorp Vault?

• Centralized secure storage for secrets.
• Dynamic secrets (temporary credentials for databases, etc.).
• Access control & audit logs for security compliance.

Step 1: Install HashiCorp Vault in Kubernetes

1?? Add the Helm repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
        

2?? Install Vault:

helm install vault hashicorp/vault --set "server.dev.enabled=true"
        

? This deploys a development mode Vault (not for production).

Step 2: Configure Vault for Kubernetes Authentication

1?? Enable Kubernetes authentication:

vault auth enable kubernetes
        

2?? Configure Vault to communicate with Kubernetes API:

vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        

? This allows Kubernetes pods to authenticate with Vault.

Step 3: Store a Secret in Vault

1?? Create a Vault policy to allow reading secrets:

vault policy write my-policy - <<EOF
path "secret/data/my-secret" {
  capabilities = ["read"]
}
EOF
        

2?? Store a secret in Vault:

vault kv put secret/my-secret password="SuperSecretPassword"
        

Step 4: Inject Secrets into Kubernetes Pods

Use Vault Agent Injector to pull secrets into pods.

1?? Annotate your deployment to use Vault secrets:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/secret-path: "secret/data/my-secret"
    vault.hashicorp.com/agent-inject-secret-password: "password"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-app-sa
      containers:
        - name: my-container
          image: nginx
        

? The secret is injected into the container at runtime.

2. Cloud-Native Secret Managers (AWS, Azure, GCP)

Why Cloud-Native Solutions?

• Fully managed, no need to maintain infrastructure.
• Automatic rotation of secrets.
• Integrated IAM controls for better security.

A. AWS Secrets Manager with Kubernetes

1?? Store a secret in AWS Secrets Manager:

aws secretsmanager create-secret --name MySecret --secret-string '{"username":"admin","password":"mypassword"}'
        

2?? Deploy AWS Secrets CSI driver:

kubectl apply -k "github.com/aws/secrets-store-csi-driver-provider-aws/deploy"
        

3?? Mount AWS Secret into a pod:

apiVersion: v1
kind: Pod
metadata:
  name: aws-secret-pod
spec:
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets"
          readOnly: true
        

? The secret is automatically fetched from AWS Secrets Manager.

B. Azure Key Vault with Kubernetes

1?? Store a secret in Azure Key Vault:

az keyvault secret set --vault-name MyKeyVault --name MySecret --value "SuperSecretPassword"
        

2?? Deploy Azure Key Vault CSI driver:

kubectl apply -f https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/main/deployment/secretproviderclass.yaml
        

3?? Use Azure Key Vault secret in a pod (similar to AWS example). ? The pod automatically mounts the secret.

C. GCP Secret Manager with Kubernetes

1?? Store a secret in Google Cloud Secret Manager:

echo -n "SuperSecretPassword" | gcloud secrets create my-secret --data-file=-
        

2?? Deploy GCP Secrets CSI driver:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp/main/deployment/secrets-store-csi-driver.yaml
        

3?? Mount secret into a pod (similar to AWS/Azure examples). ? Secrets are securely pulled from GCP Secret Manager.

3. Sealed Secrets (Bitnami)

Why Sealed Secrets?

• Encrypts Kubernetes secrets before storing them in Git.
• Decryption only happens in the cluster, ensuring security.

Step 1: Install Sealed Secrets Controller

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.20.0/controller.yaml
        

Step 2: Encrypt a Kubernetes Secret

1?? Create a Kubernetes Secret:

kubectl create secret generic my-secret --from-literal=password="SuperSecretPassword" --dry-run=client -o yaml > my-secret.yaml
        

2?? Encrypt the secret using kubeseal:

kubeseal < my-secret.yaml > my-sealed-secret.yaml --format yaml
        

3?? Apply the sealed secret to the cluster:

kubectl apply -f my-sealed-secret.yaml
        

? Even if this file is stored in Git, it is encrypted and safe.

4. External Secrets Operator

Why External Secrets Operator?

• Synchronizes secrets from external secret managers automatically.
• Supports AWS, Azure, GCP, HashiCorp Vault.

Step 1: Install External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets
        

Step 2: Configure an External Secret

1?? Create an ExternalSecret resource:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  secretStoreRef:
    name: aws-secret-store
    kind: ClusterSecretStore
  target:
    name: my-k8s-secret
  data:
    - secretKey: password
      remoteRef:
        key: my-secret
        

2?? Apply the ExternalSecret:

kubectl apply -f my-external-secret.yaml
        

? This syncs the secret from AWS, Azure, or GCP automatically!

Kubernetes Service Accounts & Identity Management

In Kubernetes, Service Accounts are used to provide an identity to pods, allowing them to authenticate and interact securely with other services.

However, using native Kubernetes Service Accounts alone is not enough when interacting with cloud services like AWS, Azure, and GCP.

To securely grant cloud permissions to pods, we use Workload Identity and IAM roles for Kubernetes Service Accounts (IRSA, Workload Identity Federation).

1. Understanding Kubernetes Service Accounts

What is a Kubernetes Service Account?

• A special type of account used by pods to authenticate within a cluster.
• Allows fine-grained access control using Role-Based Access Control (RBAC).
• Each pod automatically gets a default service account, but you can create custom ones.

Step 1: Creating a Kubernetes Service Account

1?? Create a new service account:

kubectl create serviceaccount my-service-account
        

? This creates a service account named my-service-account.

2?? Check the created service account:

kubectl get serviceaccounts
        

3?? Use the service account in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  containers:
    - name: my-container
      image: nginx
        

? This pod runs with my-service-account, allowing it to authenticate securely.

2. Workload Identity for Cloud Permissions

When Kubernetes workloads need access to cloud resources (AWS, Azure, GCP), we need a secure way to assign IAM permissions to the pods without hardcoding credentials.

A. AWS: IAM Roles for Kubernetes Service Accounts (IRSA)

AWS supports IAM Roles for Service Accounts (IRSA), allowing pods to use IAM roles without static credentials.

Step 1: Enable OIDC Provider for EKS

1?? Get the OpenID Connect (OIDC) Provider URL:

aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text
        

2?? Associate the OIDC provider with AWS IAM:

eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve
        

? This allows Kubernetes pods to authenticate with AWS using OIDC.

Step 2: Create an IAM Role for Kubernetes Service Account

1?? Create an IAM policy granting access to AWS resources:

aws iam create-policy --policy-name S3ReadAccess --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/*"]
    }
  ]
}'
        

2?? Create the IAM role and link it to the Kubernetes service account:

eksctl create iamserviceaccount \
  --name my-service-account \
  --namespace default \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::123456789012:policy/S3ReadAccess \
  --approve
        

? This links the IAM role to a Kubernetes service account.

Step 3: Use the Service Account in a Pod

Modify the pod spec to use the IAM-enabled service account:

apiVersion: v1
kind: Pod
metadata:
  name: aws-pod
spec:
  serviceAccountName: my-service-account
  containers:
    - name: app
      image: amazon/aws-cli
      command: ["sh", "-c", "aws s3 ls s3://my-bucket"]
        

? The pod can now access AWS S3 without storing static credentials.

B. Azure: Workload Identity Federation

Azure Kubernetes Service (AKS) supports Workload Identity, allowing pods to access Azure resources securely.

Step 1: Enable Azure AD Workload Identity

1?? Register the Workload Identity feature:

az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
        

2?? Create a federated identity credential for AKS:

az identity federated-credential create --name my-k8s-identity \
  --resource-group my-rg \
  --identity-name my-aks-identity \
  --issuer "https://oidc.eks.amazonaws.com/id/XXXX" \
  --subject "system:serviceaccount:default:my-service-account"
        

? This enables AKS to use Azure identities securely.

Step 2: Assign an Azure Role to the Identity

1?? Grant access to Azure Storage:

az role assignment create --role "Storage Blob Data Reader" --assignee "<CLIENT_ID>"
        

? Now, the Kubernetes service account has access to Azure Storage.

Step 3: Use the Service Account in a Pod

Modify the pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: azure-pod
spec:
  serviceAccountName: my-service-account
  containers:
    - name: app
      image: mcr.microsoft.com/azure-cli
      command: ["sh", "-c", "az storage blob list --account-name mystorageaccount"]
        

? The pod can now access Azure Storage securely.

C. GCP: Workload Identity Federation

Google Kubernetes Engine (GKE) supports Workload Identity to assign IAM roles to Kubernetes service accounts.

Step 1: Enable Workload Identity on GKE

1?? Enable Workload Identity for your cluster:

gcloud container clusters update my-cluster \
    --workload-pool=my-project.svc.id.goog
        

2?? Create a GCP IAM service account:

gcloud iam service-accounts create gke-service-account \
    --display-name "GKE Service Account"
        

3?? Grant IAM permissions to the service account:

gcloud projects add-iam-policy-binding my-project \
    --member "serviceAccount:[email protected]" \
    --role "roles/storage.objectViewer"
        

? This service account can now read objects from GCS buckets.

Step 2: Link the IAM Service Account to a Kubernetes Service Account

1?? Allow the Kubernetes service account to use the GCP IAM account:

gcloud iam service-accounts add-iam-policy-binding [email protected] \
    --member "serviceAccount:my-project.svc.id.goog[default/my-service-account]" \
    --role "roles/iam.workloadIdentityUser"
        

? This links the Kubernetes service account to GCP IAM.

Step 3: Use the Service Account in a Pod

Modify the pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: gcp-pod
spec:
  serviceAccountName: my-service-account
  containers:
    - name: app
      image: google/cloud-sdk
      command: ["sh", "-c", "gsutil ls gs://my-bucket"]
        

? The pod can now access GCP Storage securely.

?? Best Practices for Kubernetes Secret Management

? Never hardcode secrets in YAML files, ConfigMaps, or Git repositories.

? Rotate secrets regularly and automate updates.

? Audit secret access with Kubernetes Audit Logs & monitoring tools.

? Use short-lived secrets instead of long-lived credentials.

? Enable strict RBAC policies to control who can read secrets.

? Use network policies to limit secret exposure within the cluster.

Tools & Techniques for Secret Security in Kubernetes

Managing secrets securely is a critical aspect of Kubernetes security.

Below are the best tools and techniques to protect secrets, detect leaks, and enforce security policies.

? Kubernetes Secrets CSI Driver → Securely mount secrets as ephemeral storage.

? OPA (Open Policy Agent) & Kyverno → Enforce policies on secret usage.

? Secret Scanner Tools → Detect exposed secrets using TruffleHog, Git-Secrets, and Kube-hunter.

1. Kubernetes Secrets CSI Driver

The Kubernetes Secrets CSI (Container Storage Interface) Driver allows you to mount secrets as ephemeral storage instead of environment variables, which is more secure.

Why Use the Secrets CSI Driver?

? Secrets are not exposed as environment variables → Prevents accidental leaks.

? Secrets are dynamically loaded from providers → No need to store them in Kubernetes.

? Works with external secret stores → Supports AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, etc.

Step 1: Install the Secrets CSI Driver

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/main/deploy/rbac-secretproviderclass.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/main/deploy/secrets-store-csi-driver.yaml
        

? This deploys the Secrets CSI Driver.

Step 2: Mount Secrets Using CSI Driver

Create a SecretProviderClass to define the secret source (e.g., AWS, Azure, Vault, etc.).

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-secret-provider
spec:
  provider: vault
  parameters:
    roleName: "my-vault-role"
    vaultAddress: "https://vault.example.com"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/my-db"
        secretKey: "password"
        

? This pulls a secret securely from HashiCorp Vault.

Step 3: Use the Secret in a Pod

Modify the pod spec to use CSI-mounted secrets:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  volumes:
    - name: secret-vol
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "my-secret-provider"
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-vol
          mountPath: "/mnt/secrets"
          readOnly: true
        

? The secret is securely mounted as a file at /mnt/secrets.

2. Enforcing Secret Security with OPA & Kyverno

Why enforce policies on secrets?

• Prevent developers from creating plaintext secrets.
• Ensure secrets are only mounted using CSI and not exposed as environment variables.
• Restrict secret access to authorized workloads only.

A. Enforcing Secret Policies with OPA (Open Policy Agent)

OPA is a policy engine that lets you define and enforce security rules.

Step 1: Install OPA in Kubernetes

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml
        

? This installs OPA Gatekeeper, which enforces policies.

Step 2: Define a Secret Policy

Create a constraint template to block secrets in environment variables:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: deny-secret-env-vars
spec:
  crd:
    spec:
      names:
        kind: DenySecretEnvVars
  targets:
    - target: admission.k8s.gatekeeper
      rego: |
        package kubernetes.admission
        deny[msg] {
          input.review.kind.kind == "Pod"
          input.review.spec.containers[_].env[_].valueFrom.secretKeyRef
          msg := "Secrets should not be used as environment variables."
        }
        

? This prevents secrets from being stored in environment variables.

Step 3: Apply the Policy

kubectl apply -f deny-secret-env-vars.yaml
        

? Now, any pod attempting to use secrets in env vars will be rejected.

B. Enforcing Secret Policies with Kyverno

Kyverno is an alternative to OPA that makes policy management easier.

Step 1: Install Kyverno

kubectl apply -f https://github.com/kyverno/kyverno/releases/latest/download/install.yaml
        

? This deploys the Kyverno policy engine.

Step 2: Define a Secret Policy

Create a Kyverno policy to block plaintext secrets:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: prevent-plaintext-secrets
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-plaintext-secrets
      match:
        resources:
          kinds:
            - Secret
      validate:
        message: "Secrets should not contain unencrypted data."
        pattern:
          stringData:
            password: "!/^.{8,}$/"
        

? This ensures secrets are encrypted before being stored in Kubernetes.

Step 3: Apply the Kyverno Policy

kubectl apply -f prevent-plaintext-secrets.yaml
        

? Now, any plaintext secret will be rejected.

3. Secret Scanner Tools

Even with security policies in place, secrets can still leak in Git repositories, logs, and containers.

Secret scanners help detect and prevent such leaks.

A. TruffleHog (Finds leaked secrets in Git & logs)

TruffleHog scans repositories and files for hardcoded secrets.

Install TruffleHog

pip install truffleHog
        

? Now, scan a Git repo for leaked secrets:

truffleHog --regex --entropy=True https://github.com/my-org/my-repo.git
        

? TruffleHog detects secrets like API keys and credentials.

B. Git-Secrets (Prevents committing secrets to Git)

Git-Secrets blocks commits that contain secrets.

Install Git-Secrets

brew install git-secrets  # macOS
sudo apt install git-secrets  # Ubuntu
        

? Configure Git-Secrets for your repo:

git secrets --install
git secrets --register-aws
git secrets --scan
        

? If a secret is detected, the commit will be blocked.

C. Kube-Hunter (Finds vulnerabilities in Kubernetes)

Kube-Hunter scans Kubernetes clusters for misconfigured secrets.

Install Kube-Hunter

pip install kube-hunter
        

? Scan the Kubernetes cluster for vulnerabilities:

kube-hunter --remote my-cluster-ip
        

? Kube-Hunter detects exposed secrets, open ports, and misconfigurations.

?? Kubernetes Secrets CSI Driver → Mount secrets securely instead of using environment variables.

?? OPA & Kyverno → Prevent secrets misuse with automated policies.

?? Secret Scanner Tools → Detect exposed secrets before they become a security risk.

Follow Shruthi Chikkela for More!

If you found this guide helpful, make sure to:

? Follow me for more Kubernetes security tips!

? Like & Share to help others secure their clusters!

? Drop a comment if you have any questions!

?? Stay Secure & Keep Your Secrets Safe! ??

#Kubernetes #DevSecOps #CloudSecurity #K8s #CyberSecurity #SecretsManagement #KubernetesSecurity #CloudNative #DevOps #K8sSecrets #HashiCorpVault #AWS #Azure #GCP #OPA #Kyverno #KubeHunter #SecurityBestPractices #TruffleHog #GitSecrets #SecretsCSI #RBAC #WorkloadIdentity #IAM #Learnwithshruthi