Secrets Management in Cloud – HashiCorp Vault, AWS Secrets Manager and Parameter Store

Secrets Management on Cloud

This paper examines a few of the industry standard secrets management solutions for use in AWS. This is not an exhaustive list and one should perform an analysis based on their use cases before selecting a secrets management tool.

Why Vaults and What are the Typical Use Cases?

When we say that Vaults limit the secrets fall throughout your infrastructure, it means that you might have username and password embedded in the source codes, say for e.g.

• you have a web application that needs to connect to a database or
• there is a connection string in there and it has a username and password, often times, that’s hard coded in them or
• you have a username and password sitting in a configuration file
• maybe your web application is connecting to an external API and hence you have API keys in there
• especially when you are talking about AWS, you are talking about highly privilege API keys and you don’t want them to be sitting around, you don’t want them accidentally uploaded into GitHub or something like that

So when you are looking at the complete landscape, we are competing with people storing their credentials in text files, maybe all the way up-to some other password managers out there, so Vaults help us to centrally store all these secrets so that you can get a correct, audited and logged workflow around secrets management.

HashiCorp Vault Third Party Integration with AWS

I want to lay some ground-work before we start talking about HashiCorp Integration with AWS, so let us start talking about Vault Unsealing.

What is Vault Unsealing?

The Vault server is in a sealed state. The Vault knows where and how to access the storage but does not know how to decrypt any of the data from that storage. So, Unsealing is the process of obtaining the plaintext master key necessary to read the decrypt keys to decrypt the data and allow accessing the Vault.

The Master Key stored alongside all other Vault data but is encrypted by the Unseal Keys. When you first configure your Vault services, it is in a sealed state, so Vault knows where and how to access the storage but does not know how to decrypt the data in that storage, so for that reason, Vault needs to be unsealed.

So, when we have unsealed keys which is used to encrypt the master key, the master key is used to encrypt the decryption key which is used to access the Vault data. So, let us look what are the manual unsealing process looks like.

What is Manual Unsealing Process in Vaults?

By default, Vault is configured with Shamir algorithm. Each shard can be stored on a distinct machine for better security. This process works well but can be very challenging.

In this case, instead of distributing the unsealed keys as a single key to an operator, Vault uses the Shamir secrets algorithm to split this key into shards and a certain threshold of shards are required to be able to reconstruct the unsealed key to decrypt the master key (something like the RAID 5 in server mirroring based on parity)

So, we understand that the process can be very challenging when you have many Vaults clusters because in that case there will be multiple different keyholders with multiple different keys so it requires a lot of coordination and this is not the best setup for an automated secrets management.

So, for this reason, HashiCorp has few integrations with AWS that can automate the unseal process.

HashiCorp Vault Integration with AWS for Automatic Unsealing of Keys

Auto Seal with AWS HSM (Available with HashiCorp Vault Enterprise Edition)

Cloud HSM is a cloud-based hardware security model that enables you to easily generate and use your own encryption keys on the AWS cloud. It protect encryption key with master Key.

AWS Cloud HSM has zero config, high availability that includes automatic backup stored in Amazon S3.

In this automation approach, Vault protects its master keys for encryption rather than splitting into key shards. In this integration, the Vault put its encrypted master key from the storage and transmits this key that it pulled from the storage through HSM for decryption.

Once the master key is decrypted, the Vault uses the master key to decrypt the encryption to resume with the Vault operation.

The customer must ensure to control access to HSM by configuring the IAM permission by making sure that the right users and groups have access to it.

They can also restrict actions that are allowed on that key, customer also can make sure that they use MFA (Multi Factor Authentication) with each of these accounts or if they need to use FIPS end point, i.e., if they want to use the FIPS validated cryptographic modules.

Auto Seal with AWS KMS (Key Management Service)

You are no longer giving the master key to humans. It protects encryption key with master key.

You are using AWS KMS to store the master key. Provide Vault with credentials to get the keys.

When you are configuring Vault on AWS, it gives an option for automatic sealing using AWS KMS. In this case, you are no longer giving the keys to humans but using KMS to store the master key. The KMS key protects the master key which protects the encryption keys.

So, if you have a need for a Vault setup and you want to be able to restart the Vault setup at 3 AM in the morning, then without involving any manual interventions, auto unseal is the way to go.

AWS ensures that the key materials in KMS is encrypted and it is stored in durable persistent storage, AWS will also ensure that it meets the regulatory and compliance requirements of programs such as SOC, PCI, FedRAMP, HIPAA etc.

AWS also provides certain tools and mechanism like Cloud Trail Logs and CloudWatch Logs to monitor the keys and customer is responsible to control access to KMS by configuring the IAM permission by making sure that the right users and groups have access to it.

What is HCP (HashiCorp Cloud Platform)?

It was GA in April 2021 is a fully managed service for AWS environments and give you the power and security of Vault without the complexity and operational overhead of managing it yourself. HCP Vault also enables secure secrets management across EC2, EKS, RDS, Lambda and many other AWS services.

Can we use Database Credentials Rotation for RDS?

There are two different workflows by which you can use database credentials for RDS via Vaults. The first is the simple Static Workflow where the Vault has the capability to store static secrets that is stored in source codes, connection strings to database and so on and putting them into Vault.

Next one is more expanded Dynamic Workflow and typical use cases for generating and storing credentials like :

• you want to access an AWS service like RDS and Vault can integrate with RDS and generate short-lived credentials based on some policy or
• maybe you need to access to spin up an RDS instance or create a database or access an Amazon S3 bucket for database snapshot backup job, you can ask Vault to generate you a short-lived credential that has this associated policy and Vault will actually revoke it after the time period
• So, you can have both static and dynamic workflows and expand it on Amazon RDS and via SQL login to the database generates that credential for you, so you do static credential
• Also, you can rotate credential, say for e.g. for compliance reason or something like that you need to rotate this particular database credential once a month and Vault can setup to do that
• For dynamic credentials where for e.g. you have a web application which you have spun up and you can ask Vault to generate you with a credential for this database and the Vault will login to the database and provide the credential to the web application.

If KMS is down and the keys are in KMS, is there a way to we can unseal the Vault in this case?

AWS KMS is designed to be highly available, hence the encrypted data that is stored in KMS is replicated across multiple AWS Availability Zones (AZs) , so changes of multiple AZs in a Region going down at the same time is really negligible and also it is designed for 99.99% durability, so KMS is a highly secure service to work with.

Why would you use AWS KMS?

If you don’t have the Auto Unseal functionality enabled in AWS and Vault for some reason encounters an operational issue and it is 2 AM in the morning, you could be notified saying that the Vault is down and it is in the critical path of all the applications

As sort of core functionality in there that it allows Vault to auto unseal or heal itself, so we always recommend people and applications should use Auto Unseal when possible either like On-Prem, if they have a HSM or you know obviously in AWS with HSM/KMS.

AWS Secrets Manager

AWS Secrets Manager is a robust way to store secrets natively in AWS. AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, API keys, and other secrets throughout their lifecycles.

It uses AWS Identity and Access Management (IAM) to produce policies to govern both access and management of secrets, and it utilizes AWS Key Management Service (KMS) to encrypt secrets at rest.

By using IAM resource-based policies, trust relationships can be established to allow cross-account access of secrets from a centralized account. A distinctive feature of secrets manager is the ability to rotate secrets. In order to achieve this, a separate AWS Lambda function must be created and appropriate roles must be granted to the function to execute rotation of the secret(s).

In addition, Secrets Manager has a built-in password generator that is used during the rotation of secrets and can be invoked on the CLI using the?get-random-password?command.

AWS Systems Manager Parameter Store

AWS Systems Manager Parameter store is a simple AWS native solution that allows for the storage of two types of secrets, called parameters: standard and advanced.

Standard parameters is the default tier that holds secrets up to 4 KB in size and have no additional charge associated with them. Advanced parameters increased the maximum size to 8 KB and allows for parameter access and management policies expressed in IAM. Standard parameters can be changed to advanced, but not vice-versa.

Like AWS Secrets Manager, all secrets are encrypted using AWS KMS. Unlike AWS Secrets manager, there are no auto-rotation or password generation capabilities.

HashiCorp Vault on AWS

HashiCorp Vault is described as the “Swiss army knife” for cloud security with capabilities well beyond secrets management. Vault is cloud agnostic and has been deployed at scale in multiple public clouds and hybrid clouds with success.

Vault secrets are encrypted at rest and supports password auto-rotation and dynamic credential capabilities with wide compatibility using a variety of secret backends. In addition to secrets management, Vault can be used for other tasks such as brokering SSH access, encryption on demand, and password generation.

Use Case(s) - Based on Capabilities

• Parameter Store provides a bit more versatility. It has the option to store data unencrypted or to encrypt the data with a KMS key.
• With Secrets Manager, the secrets are stored encrypted and there’s no option to store unencrypted data. So that’s one use-case for Parameter Store.
• Parameter Store comes with no additional charges. There’s a limit on the number of parameters you can store, and that limit is currently 10,000.
• Secrets Manger has the ability to automatically rotate secrets. So, Secrets Manager can rotate keys and actually apply the new key/password in RDS for you automatically.
• Secrets stored in Secrets Manager can be shared across AWS accounts that is not feasible for secrets stored in Systems Manager Parameter Store.
• Secrets Manager has the ability to generate random/dynamic short-lived secrets and SDK can be used to do this in your application code. The Systems Managers Parameter Store cannot create random/dynamic secrets.
• HashiCorp supports multi-loud and hybrid cloud uses cases which is also the main difference vs Amazon Secrets Manager.

AWS Secrets Manager is a perfect choice if you're launching your start-up or have small number of secrets to manage but tight regulations - PCI DSS, ISO 27001 and others.

HashiCorp can be wiser choice if you need multi-cloud or hybrid cloud options or will need to manage thousands of secrets.

*A pricing estimate for all the three Secrets Manager is examined below

AWS Secrets Manager

AWS Secrets Manager is $0.40 per secret per month, for secrets that are stored in less than a month the price is prorated. There is an additional charge of $0.05 per 10,000 API calls.

Pricing Example for AWS Secrets Manager

If 1000 secrets are stored using AWS Secrets Manager, with 400,000 API calls there is:

• A monthly charge of $400 per month
• API calls will cost an additional $2

AWS Systems Manager Parameter Store

AWS Systems Manager Parameter Store consists of standard and advanced parameters. Standard parameters are available at no additional charge.

Advanced parameters are charged $0.05 per secret per month. Charges for parameters stored in less than a month are prorated. There is an additional charge of $0.05 per 10,000 API calls.

Pricing Example for AWS Parameter Store - Standard Parameters. Using standard parameter secrets are allowed at no additional charge

Pricing Example for AWS Parameter Store - Advanced Parameters

For example, if 1000 advanced parameter secrets are stored with 400,000 API calls using AWS Systems Manager Parameter Store, there is:

• A monthly charge of $500 per month
• API calls will cost an additional $2

HashiCorp Vault

HashiCorp Vault comes in both open-source and two enterprise versions: Enterprise Platform and Enterprise Modules. Both enterprise solutions come at an additional cost, which can be given by contacting HashiCorp.

HashiCorp Vault is also a self-hosted solution for both open source and enterprise, therefore both provisioned infrastructure and operational upkeep must be considered when calculating cost. There is no additional cost for number of stored secrets or secrets retrieval.

Pricing Example for HashiCorp Vault

Using Vault’s reference architecture above: three m5.large instances with 25GB gp2 drives attached to achieve n-2 redundancy (where the loss of 2 objects within the failure domain can be tolerated), we can provide an estimated cost for HashiCorp Vault. The calculation below was performed using the AWS pricing calculator:

• 3 instances x $0.096 x 730 hours in month = $210.24 (monthly, on-demand instances)

Conclusion

When to use HashiCorp v/s Amazon Secrets Manager v/s Parameter Store?

You have a Vault, you have centralized all your secrets and you have an audited, logged workflow, so you have a bunch of different teams/applications that have a single workflow to access the Vault.

Now a team comes in and say that they want to access these particular secrets and you might have some internal procedures that states how you should access the Vault and what are the secrets that you have access etc.

This works great if you are in On-Prem and then you want to go to cloud and lots of teams want to use the same workflow, hence is one of the main reason why you will use HashiCorp Vault in multiple regions (hybrid cloud) / multi cloud.

Vault is like a Swiss army knife and there is lot of different functionality in terms of the enterprise features side like say Tokenization, like you are processing credit card numbers, membership numbers, if you look at a simple online store applications, they have lots of different secrets in there, they are going to have lots of different API keys and they will be processing PII data, so Vault is like a Swiss army knife that has bunch of different features and integration known as Secret Engines which are basically plug-ins that can integrate with Amazon RDS, so it is more around workflow and use cases that will drive the decision to use HashiCorp Vault.

But if you are tightly in AWS and you are already using secrets there, you may not need to have the Vault and extended features, then you can totally use AWS Secrets Manager just because it is much simpler, you don’t have to maintain a third party product like Vault.

For low quantities of secrets, AWS Systems Manager Parameter store can be a cost-effective solution. It is appealing, if you have less than 10,000 secrets and no secrets greater than 4 KB since there is no additional cost but is not recommended due to the lack of security features - namely lack of parameter policies. It may still be useful for Non-Production systems and applications.