EP 12 - 5 Secrets to Secure Kubernetes Secrets—And Why Vault Should Be Your Go-To
Malchiel Urias
DevOps Engineer || Site Reliability Engineer || Cloud Engineer || 4x AWS Certified || CEH Master
How Does Kubernetes Handle Sensitive Data
In today's world of technology, where cyber attacks are more prevalent than ever, it is important that sensitive information is kept as secure as possible in the application environment. Kubernetes, as a container orchestration system, understands this and provides an API object that can help us handle this called Secrets.
Are Your Secrets Truly Secret?
The issue, however, is that although Kubernetes tries to obscure and secure the data contained in secrets, it only does that by encoding the data in base64. This level of security still does not provide enough security for sensitive data.
Encoding, in effect, simply means that anybody who gets access to the encoded data can simply decode it and would easily have the secret.
What you would need in your Kubernetes cluster to truly protect your sensitive data (DB connection strings, passwords, SSH private keys, etc.) is a tool that can truly encrypt the secret data at rest and in transit so that even if the secret is exposed, it cannot be compromised without a root key. This tool should also be able to act as a central storage location for all secrets in the cluster to avoid secret sprawl. Finally, access to this tool needs to be managed, authenticated, authorised and audited.
Harshicorp Vault is one tool that meets these requirements efficiently.
5 Secrets for Keeping Your Secrets Truly Secret
Secret #1: Encryption of Secrets at Rest
Kubernetes:
By default, secrets in Kubernetes are simply encoded when at rest and stored in etcd. Anyone with access to the etcd datastore or the Kubernetes secret object can easily get the secret, hence making it vulnerable. Without proper encryption at rest, secrets are vulnerable to exploitation by bad actors if exposed
Vault
Harshicorp Vault provides centralised storage where it encrypts and stores secrets. This makes Vault a much better alternative for managing secrets. It ensures that even when secrets are exposed, they cannot be decrypted without the secret key, which is also kept secure outside of Vault.
Secret #2: Secret Rotation and Expiry
Kubernetes:
Secrets in Kubernetes are static. This means that they remain the same until they are manually updated. When a secret is the same for a long time, it can be a significant vulnerability because if the secret is compromised, the bad actor will retain access to the privileged system for as long as that secret is valid.
Vault:
Vault, on the other hand, provides the feature of secret rotation. The idea behind this is to make the secret a moving target such that the secret changes over time instead of staying the same, and even if it somehow gets compromised, it would be invalid once it is rotated. Vault also can generate secrets which can be automatically rotated, thereby reducing the risk of long-lived secrets.
Secret #3: Fine-Grained Access Control
Kubernetes
Role-Based Access Control (RBAC) in Kubernetes provides a means to control who has access to read and use secrets. This provides a level of security however, it tends to be over-permissive, giving users and services access that they should not have.
Vault
Vault uses Access Control Lists, which go the extra mile to provide granular details on who can access the secrets and under what conditions. It can also provide specific details on how secrets are accessed by users and services.
Secret #4: Auditing and Logging
Kubernetes
Kubernetes does not provide built-in mechanisms for auditing who has accessed a secret, when they accessed it and how. This information is important for maintaining security and limiting the access of the secret. We need to know how the secret is being used.
Vault
Vault offers robust auditing and logging capabilities. Every interaction within Vault is logged, allowing administrators to see who accessed the secret, what secret was accessed, how it was accessed and when. These audit logs can be integrated into external logging services for monitoring and alerting. This level of transparency provides a clear audit trail, making it easier to detect and respond to unauthorised access or suspicious activity.
Secret #5: Injecting Secrets into Pods
Kubernetes
Applications can use secrets because Kubernetes injects the secrets into pods as environment variables or volume-mounted files. This practice can expose secrets to anyone with access to the pod's environment, leaving the secrets open to accidental logging or exposure in debugging sessions, posing a serious security risk.
Vault
Vault alleviates this risk by securely injecting secrets into Kubernetes Pods at runtime using the Vault Agent Injector. With this method, secrets are not placed in the pod as environmental variables, instead, they are dynamically retrieved when the pod needs them.
Vault can also ensure that secrets are delivered only when the application is running and removed when no longer needed.
Conclusion
With the growing number of attack vectors and increasing complexity of cloud-native architectures, Kubernetes native approach to secret management is not secure enough, engineers need to adopt a more sophisticated, secure, and automated solution.
With HashiCorp Vault, you are not just only able to strengthen the security posture of your Kubernetes clusters, but you can also implement best practices like automatic secret rotation, granular access control, and comprehensive auditing. If you’re serious about securing your cluster’s most valuable assets—its secrets—then Vault should be your go-to solution.
Are you confident your Kubernetes secrets are truly secure? If you're using Vault or have insights on secret management best practices, we’d love to hear your thoughts!
Share this post with your network, or drop a comment below with your experiences. Let’s build a stronger, more secure Kubernetes community together!
Further Reading: