Let's talk secrets

Australia saw?one of the most significant cyber attacks?unfold recently. While corporates and customers still grapple with the fallout, cyber security and its importance have again gotten the spotlight. This article discusses an essential aspect of cyber security, "Secrets." I will be exploring why managing secrets are so important and why it is so challenging. I will also explore the options available to implement a secret management solution.

What are secrets?

What kind of secrets you use might differ based on your use case, solution, technology, and tools. However, the following are broadly the secrets you might end up using.

• Username Passwords: -?Typical credentials that can be used to login into any system, server, service, etc.
• Credentials: -?These could be backend or middle tier credentials like Databases, MQs, etc., which users or applications use to connect to these services.
• Authorization Tokens: -?Typically used to intercommunicate between applications and services.
• SSH Keys:?- Keys to log in to servers, services etc.
• Private Certificates (SSL, TLS):?- Used by applications and services to perform encryption at rest and in transit.
• Encrypted Strings: -?Any other encrypted data you do not want to expose to the outer world.

Why keep secrets?

Secrets address your configuration, authentication, authorization, and encryption needs and helps protect sensitive information you do not want to expose to the outside world.

Can you keep a secret?

Apparently not. Let's admit it, we all have knowingly unknowingly handled secrets in the past where we knew it could be vulnerable.

• Kept credentials and passwords in plain text files on unsecured workstations or laptops
• Accidentally exposed credentials in video calls
• Checked in credentials to source repos
• Exposed secrets in log files or config files

In the interest of velocity, we all have taken shortcuts. How often have we heard, "Let's get it working without TLS first, and then we can add TLS support later." While some of us make these changes in time, in some cases, many of these security tasks keep languishing until a security incident happens.

Taking the pin out of a grenade and hoping it does not blow up in our faces is not uncommon in software engineering.

While most of the time, we have lived charmed lives with these vulnerabilities, some of us have been at the rough end and victims of a security breach. And if you are one of those unfortunate souls, you would know it is not a pleasant experience.

What do we stand to lose?

A few nights' sleep if we are lucky. Loss of reputation, revenue, and more if we are not so fortunate.

Security breaches cost companies and governments millions, sometimes billions of dollars, from ransom payments to stolen funds to settlements to stolen IP and data.

Are we (engineers) to blame?

I would have liked to say No, but let's settle for a "No and Yes." Let's understand that security comes at a cost. Things like

• Procuring tools to manage secrets well
• Scoping for security implementations
• Scoping for security review and remediation
• Developer training on security
• Incorporating security checks right from the drawing boards

All come at a cost. Not many shops identify and incorporate these in cost estimates until a security breach occurs. The onus often lies on the engineers to deliver the most secure implementation at the least possible time and budget, which is not fair. While companies spend a lot to secure their corporate resources and processes, engineering still does not get enterprise-level focus.

Companies need to proactively invest towards embedding security in the engineering tools, processes, and culture, instead of them being an afterthought.?

However, I feel a mindset change needs to happen with us engineers. We must understand that security is not an absolute concept. It is fluid, and your security posture may vary from implementation to implementation. At the end of the spectrum, you have government, health care, or financial deployments, where you must comply with stringent security standards. Implementation is complex, technical, and time taking in such projects. Unfortunately, engineers often think of security as these implementations and shy away from the deep waters. We must understand that

Not every house that we build must be a fortress. Neither should every house be “Hello Word.”

We need to understand the neighborhood we are building for, assess our resources, and add security measures accordingly. And we need to ask the relevant questions when in doubt. Also, we must understand that security is a skill that does not come automatically, and just like any other tools and technology, you need to invest in training for security.?

This is turning out to be more philosophical discourse than a technical chat. So let's get back to the meat of the topic.

Why is it challenging to manage secrets?

In any decent-sized setup, if secrets have been an afterthought, you could spend months clearing accumulated tech debt related to secrets.

If you have not designed your secrets consumption and recycling effectively and do not have a way to audit your secrets reliably, a recycling event or a security breach can quickly become a nightmare.?

If you have managed secret or certificate lifecycles and implementation and debugged TLS or SSL issues, you will know how sometimes hours can quickly dissolve into days and weeks.

Fixing security issues at times is like finding a needle from the haystack of encryption algorithms, protocols, sketchy documentation, non-existent community examples, cryptic logs, operating system idiosyncrasies, divergent terminologies, and varied implementations of security standards.

No wonder engineers tend to avoid security issues in the backlog, like the plague.

How is it usually done?

There are several ways secrets are managed currently, and we often use multiple approaches.

• Using plain text and config files.?This is fast becoming a thing of the past as security takes center stage.
• As part of pipeline secrets.?Almost all pipeline orchestration tools provide a way to store and use secrets.
• As part of IaC secrets.?Like CI CD pipelines, the IaC pipelines also keep and use secrets.
• Using cloud-native services.?All cloud service providers have services for secrets management.
• Use homegrown implementation.?Companies build and support their own implementations, which get leveraged by teams.?
• Using third-party tools.?Mature implementations offload a lot of the secret management workloads to third-party tools.
• Manually.?Look around your implementations and see how many manually created credentials, keys, certificates, or secrets you have. Surprising right?!

What features do we need to do robust secrets management?

• Just in time, ephemeral credentials: -?Issuing credentials only when needed and for the duration required.
• Credential lifecycle management: -?Ability to facilitate periodic recycling of secrets with the least possible downtime.
• Encryption at rest and in transit: -?Ability to encrypt data at rest and in transit.
• Encryption as a service: -?APIs or services to help facilitate encryption and decryption of data.?
• Key lifecycle management: -?Applications or hardware security modules (HSMs) to generate, establish, distribute, store, account for, suspend, revoke, or destroy cryptographic keys and metadata.
• Certificate lifecycle management: -?Ability to consolidate, control, manage, monitor, and audit the entire life cycle of SSL/TLS certificates. Ability to automate the issuing, distribution, validation, revocation, and recycling certificates.
• Identity/Attribute-based access: -?Ability to define granular access control of various components for individuals and processes.
• Tools and application integration: -?Ability to integrate with the development and delivery toolchain and third-party and in-house applications.
• Highly scalable and available: -?Ability to provide all the above in a highly available and scalable way.

What are our options?

Cloud-native services

Cloud vendors have recognized the need to build secure solutions, and all vendors have a slew of services catering to various aspects of secrets management. The features, ease of use, availability, and scalability built into these services provide a compelling option to build your secret management infrastructure.

?AWS

There are several services in AWS that cater to secrets management.

AWS Secrets Manager?provides encryption and rotation of encrypted secrets used with?other AWS-supported services.

AWS Key Management Service (KMS)?provides tools for generating?master?and other?data keys. AWS KMS also interacts with many other AWS services to encrypt their service-specific data.

AWS CloudHSM?provides?hardware security modules (HSMs)?that can securely store cryptographic keys, including master and?data keys.

AWS Certificate Manager (ACM)?generates, issues, and manages?public and private SSL/TLS certificates?for use with your AWS-based websites and applications.

AWS Private Certificate Authority (AWS Private CA)?is a managed private certificate authority (CA) service with which you can manage your CA infrastructure and?private certificates.

AWS Encryption SDK?provides a client-side encryption library for implementing encryption and decryption operations on all data types.

Amazon DynamoDB Encryption Client?provides a client-side encryption library for encrypting data tables before sending them to a database service, such as?Amazon DynamoDB.

AWS Identity and Access Management (IAM) provides a way to authenticate and authorize access to these secret management services and all other AWS services.?????

AWS Security Token Service is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users you authenticate (federated users).

Third-party Vendors

Several vendors in the market develop and sell specialized tools to cater to various secret management needs. Some of the notable ones are the following.

Hashicorp Vault: - Probably the most well-known secret management tool in the market, Vault is a one-stop shop for all your secret management needs. Vault provides encryption and secret management services gated by authentication and authorization methods.

Doppler: - Positioning itself as a SecretOps Platform, Doppler enables developers and security teams to keep their secrets and app configuration in sync and secure across devices, environments, and team members.

AKeyless Vault: - Another popular solution in the market, AKeyless helps protect and automate access to secrets like credentials, keys, tokens, and API-Keys across the DevOps tools and cloud platforms using a secured vault.

Open-source tools

If you are keen to implement and manage your own secret management solution, then quite a few robust options are available that can give you a good headstart.

https://github.com/pinterest/knox: - Purpose built by Pinterest to help developers access/use confidential secrets, keys, and credentials and provide a way to easily rotate these secrets and keep an audit log of secrets usage.

https://github.com/lyft/confidant: - From the developers at Lyft, Confidant, provides features like KMS authentication, at-rest encryption of versioned secrets, and a user-friendly web interface for managing secrets.?

https://github.com/square/keywhiz: - From the developers at Square, Keywhiz is yet another system for managing and distributing secrets.

https://github.com/fugue/credstash: - CredStash is simple, easy-to-use credential management and distribution system that uses AWS Key Management Service (KMS) for key wrapping and master-key storage and DynamoDB for credential storage and sharing.

Homegrown solutions

If you want to take the road less traveled and have the resources to do so, the option to create and maintain a bespoke secret management system is always there. However, with so many existing mature solutions at hand, it is a less likely path.

Conclusion

Secrets management can be an unwieldy beast, however, if you plan well and give it the required focus, you can turn it placid and purring.

Shift left: - Even at the risk of moving out of frame. If you are not talking about secrets and how to manage them at the onset of a project, if you do not have security tasks in your backlog from the very beginning, you are setting yourself up for some troubled times ahead.

Invest: - In tools, training, resources, people, and implementations, related to security.

Know your neighborhood: - As I said before, not every house you build must be a fortress. Neither does it need to be “Hello World!" Assess your security requirements, take stock of the resources you can muster, and build accordingly.

Plan for the worse: - Hope is not a plan. If something can go wrong, it will go wrong. Plan for a worst-case scenario. Run simulations. Action on learning outcomes.

No fun in fear: - Remember

If you are not prepared, you are scared (or blissfully ignorant).