Secret 0: Keys to the Kingdom

Today I'm going to introduce you to something called "Secret 0," which I will also refer to as a "0 secret." I'll explain what one is, why it's so important, why it's hard to manage, and some modern solutions that make managing secret 0 easier than in the past.

Then in the second half of this article I will give you some brief description of various tools and techniques to manage secrets, so you can evaluate whether they are a fit for you. Some of them require a 0 secret, and some manage to get rid of them.

Most computer systems need to authenticate users, and most also need to authenticate to other elements of their environment, whether that's a database, another computer system via an API, or somewhere else. If the system needs to prove its identity, it needs to authenticate to another system. If it needs to prove who as user is, then it needs to authenticate that user. In both cases, the system is using authentication to validate an identity, either a person or another system.

There are a number of ways that systems can authenticate users and other systems. Many have traditionally used a simple as a login and a password, and many continue to do so. More advanced solutions might use something much more complex, like a X.509 cryptographic certificate. Collectively, all of these ways of authenticating are referred to as "credentials" or sometimes "authenticators." Since they need to be kept secure so someone doesn't steal them, they fall into the general category of "secrets that need to be managed."

Most computer systems use secrets to authenticate, in order to prove identity.?It may be a user's password, or the key associated with an X.509 certificate.?Most systems need anywhere from 2 to 10 secrets, and some systems may have dozens or hundreds. Typical uses for these secrets would be validating a user when they want to use the system, to let the system access a database, or to communicate with another system. Whether it's a password or a cryptographic key, all of these secrets need to available to for the system to use, but stored safely so not just anyone can get at them. This proves to be surprisingly hard.

A na?ve solution to the problem is to just put the secret directly into the application code, or in application configuration files.

Don't do this.

Embedding the secrets in the source code, or in the configuration files that go with it exposes it to developers, who don't need to know the secrets in production. If the source or the configuration is checked into a source code repository, anyone with read access to the source code repository can also see the secrets. This clearly violates need to know: most people don't need to know the secret in order to do their job.?

Another solution is to store the secret on the filesystem somewhere, rather than in the code. The code then retrieves the secret from filesystem when needed. This is better than having the secret in source code, but in the event the platform is compromised, it can expose the secret. If an attacker gains access to the platform, they can read the secret, and then "pivot" from your system into any other that you have a secret for. So better, still not good.

Storing secrets in a database has more or less the same problem as storing it on the filesystem, but introduces another secret in the process, the one needed to access the database.

Storing a secret in a database, or encrypting a secret, whether in source code or configuration files introduces a chicken and egg problem: how do you manage the key that's used to decrypt the secrets?

These cases introduce us to a "primordial" secret, the one that is used to unlock all the other secrets. This is "Secret 0." If you have that Secret 0, you can get the rest of them.

Managing Secret 0 is surprisingly difficult. We need it to be accessible, usable (which usually means unencrypted and readable) and secure, all at the same time. Historically we've not had a good way to do all of those at the same time. The good news is that there are newer tools available to developers to help them manage 0 secrets, which I review briefly below.

Docker

The Docker container system has a secret management solution called Docker Secrets. Using Docker Secrets makes a secret available to a swarm, where it shows up as a filesystem inside the container. In this way, the solution strongly resembles storing secrets on the filesystem in traditional deployment models, except the store is encrypted by Docker Secrets. The application simply reads the secret as though it were any other file. The secrets have to be provisioned somehow into Docker Secrets, which may expose the secrets via an infrastructure-as-code pipeline, or to the platform administrators who provision the secrets, but there is no reason they need to be visible to the developers. This solution doesn't require a 0 secret for the application to manage.

Ansible Vault?

Ansible Vault is a tool that provides file-based encryption, and supports multiple passwords in the vault. It however requires the application to have a password (a 0 secret) to decrypt the vault, and doesn't provide a way (without using Ansible Tower or AWX) to manage that 0 secret. Secrets also need to be added into the vault file via some means, but once that's done, the vault file can be checked in safely to source code control, because it's encrypted.?

Kubernetes?

Kubernetes is a cluster management system that runs within Docker. Kubernetes Secrets is a service you can add to handle secrets, managed by the cluster. These secrets require provisioning into the service like Ansible Secrets. The secrets can appear in the cluster as a filesystem mount, via Kubelet, or as a container environment variable. They do not require a 0 secret, but they do require a human to provision the secret into Kubernetes Secrets.?

Microsoft Azure

Azure Key Vault is a service that's available in the Microsoft Azure environment. Key Vault provides a common vault for storing secrets, and can cooperate with Managed Service Identities to eliminate Secret 0. If you don't use MSI, it requires a secret to open the vault.

When you use MSI, a secret is automatically provisioned for your application's Azure runtime instance, and that secret grants access to the key vault. Since it's provisioned on the fly by the environment, this means there is no secret in the application source code or the configuration files. It eliminates secret 0, but it's still not a perfect solution, since secrets in the Azure Key Vault still need provisioning. That can expose them to whoever has to provision the secret. If the environments are provisioned using Infrastructure as Code (IaC), it may also expose them via any code pipelines or source repositories used by the IaC. It's better than secrets in the application source though.?

If you don't use MSI with Azure Key Vault, you will need a key, (a 0 secret) and you'll have to find a way to manage that. At that point, Key Vault is not any better than the best solution we've covered above.?

Amazon Web Services?

If you're in the Amazon Web Services environment, a combination of AWS Roles and either Parameter Store or Secrets Manager is a strong option, allowing you to avoid having a 0 secret. AWS Role assignments can be used to grant the access to the secret store instead.?

AWS Secrets Manager stores secrets in an encrypted format, and makes them available either by language-specific Software Development Kits (SDKs), or via an HTTP request. It relies on AWS Key Management. As with so many of the solutions we look at, it still requires that the secret be provisioned into the environment somehow, which is a possible means of exposure. It is a licensed product, charging per-secret, per-month.?

Parameter Store, a component of AWS Systems Manager, can also be used to store secrets. It supports the ability to store parameters encrypted, which is what you'll want to do if you choose this route.?

The main functional difference between Parameter Store and a Secrets Manager is the support of automatic rotation in Secrets Manager. If both sides of your relationship support Secrets Manager and automatic rotation (such as if your secret is for an AWS RDS database), Secrets Manager is probably the preferred route, since it can automatically manage rotation for you. Parameter Store does not support automatic rotation, making it more friendly for environments where secrets are relatively static.?

HashiCorp Vault?

Another tool that can be used in both cloud and on-premise environments is HashiCorp Vault. This tool provides runtime access to secrets via a user interface, a command-line tool, or an HTTP request.

HashiCorp Vault is a little different than the other solutions, it requires the vault to be "unlocked" when it is started. This unlock can be set up to require more than one person to provide their password/key, and this can be based on roles. For example, this would allow you to set up the vault so that two techs and a manager need to participate to open the vault, if you so wished.?

HashiCorp Vault provides a number of authentication mechanisms for systems when they want to retrieve secrets. Perhaps the most compelling of these are the AWS-specific mechanisms, and these can replace the need for a Secret-0 to authenticate access to the vault. See the Auth Methods section of the HashiCorp Vault documentation for more details.

CyberArk

CyberArk is popular with many larger enterprises as a way to manage sensitive logins and passwords, such as those used by administrators of systems. It provides ways to "check out" a password for a computer you need to manage, keep track of who checked out the credentials, and depending on the environment can record the user's session, and automatically rotate credentials after every use or periodically.

CyberArk itself is primarily people-facing, but it has a complementary product called Conjur which can be used by systems to retrieve secrets from the vault. It requires adding a broker in the application, which handles access to the CyberArk Vault on behalf of the app. It works with container environments such as Docker, where it authenticates the container, and then provides access to the vaulted secrets to the application running within the container. This could remove the need for a 0 secret.?I encounter CyberArk all the time, but have not seen Conjur in use.

Summing up

Many of the solutions we looked at here have a common problem, in that they still require someone to know the secret, and to add it in to the vaulting system. If your organization is also pursuing infrastructure-as-code, and you are provisioning secrets this way too, then you still have secrets checked in to source code control, which is pretty dangerous. Despite that problem, some do a good job of addressing the need for secret 0, so that's helpful.?

Systems that can use cloud-based techniques to avoid having a 0 secret, such as with AWS Roles or Azure Managed Service Identities and Azure Key Vault.?

We're still waiting for the perfect solution to managing secrets, a problem made somewhat worse with the Infrastructure as Code approach to provisioning systems. We've found a few ways to eliminate Secret 0, but we're still waiting for a better solution to provisioning those secrets as well.