Challenges with the Supply Chain Security Ecosystem - An OpenSSL Story

This is an article about the challenges with the software supply chain ecosystem. It uses the recently announced OpenSSL vulnerability as an example walkthrough. It's about what companies would need to address the issue. It is also as a result both about the gaps and the role of supply chain efforts aiming at making life easier to deal with these security issues. It is a cautiously optimistic article about what is currently a very difficult problem.

I would like to thank Dan Lorenc, Vinod Anandan & Abhijeet Udas and all those who contributed feedback to the article.

Let’s talk about the new OpenSSL vulnerability to start. OpenSSL is a very ubiquitous and important piece of software. It's used to estable secure communication (the 'S' in HTTPS for example). In 2014 a critical vulnerability was discovered (commonly known as heartbleed), which led to a mass mobilisation effort to fix it. On Tuesday November 1st a patch to another critical vulnerability will be released. There are several things organisations would do as part of their response.

Understand the vulnerability - When the security advisory is out for the latest OpenSSL release, it’s key to understand which software versions are affected and under what conditions can it be exploited by an attacker. In this case the impacted versions are about a year old so it’s important to remember that it might not be as widely an issue like the critical Log4J issue last year. Context is also important. If you look back at the OpenSSL heartbleed issue it was both a server-side and client-side vulnerability (e.g. send a link to Reddit to a domain you can control and the attack works) so fixing the servers while neglecting the client agents would have exposed a company to that security issue.

• Supply chain challenge: Currently security tooling can tell you if you are using a vulnerable component. They cannot tell you if you are using them in an insecure context, meaning the process of assessing each instance is still manual and prone to error. Developers would either have more work (patch everything) or ignore issues due to the number of false positives and lose faith in the security notifications they are being provided. False negatives are equally devastating (missing the issue entirely), and there are questions on whether existing tools would pick up all the instances of OpenSSL even if they can.
• Promising development: There is a new process and specification called VEX that allows for communication of a security flaw’s context in order for that to be provided to users of that component. It will require more work on maintainers of software as part of fixing an issue (or security vendors), but it would reduce the noise.

Understand your organisation's exposure - Essentially a lot of services use the TLS protocol to communicate securely. OpenSSL is the most popular implementation of the TLS protocol (formerly SSL hence the name). As a result, a lot of HTTPS rollouts for websites as well as other services such as VPNs, SMTPS (mail) may be impacted. This means it’s not just the software you create it’s also the third parties, enterprise software and saas platforms you use that impact your risk.

• Supply chain challenges: Currently the process of understanding the impact of your third parties is manual. You would need to pull a list of them from inventory (which may or not be complete), call them up or comb advisories, and then patch where applicable. You have control over the software you build so you can generate software bills of materials (SBOM) to identify them yourself. There is no such mandate for third parties (yet) nore any obligations for them to release their bill of materials.
• Promising developments: There is a specification (and tooling) to generate software bill of materials (SBOMS). OWASP CycloneDX has a SaaS SBOM specification. It would still need adoption, and work out how companies would share them in a standardised approach, but it is a start and we should encourage that conversation.

Remediation - This is a combination of updating the software you have developed, patching server / enterprise software & coordinating with your service providers (cloud/saas) on the roll out of any of their patching (might push a change at your end or service disruption on their end). Remember you might need to patch multiple times if gaps in the initial patch or new issues arise during the process (it has happened before). Please keep in mind your partners & clients might reach out to you asking about your response to this issue as well.

• Supply chain challenges: As mentioned earlier, you can build an SBOM for the software you own and manage, it’s a different story for the software you use. The tools you leverage such as Snyk/Sonatype/DependencyTrack/Others will have a repo of the SBOMs you have ingested but not necessarily for that NGINX server you use or a proprietary one like Microsoft IIS (especially if it's not aware of it). To take a step back you need to understand what all those software services are. To help, you can vulnerability management tools (such as Tenable/Qualys/Trivy/Others) to generate lists of software per asset using specs like CPEs (not great) or (hopefully soon) SWID Tags. and (ideally) PURLs which are URLs to the binary packages. There are SBOM specs for different types of software, but having the solutions to identify them, and report on them is another issue.
• Promising developments: We have all the ingredients for this but it all needs to be put together. My personal "nice to have" is pulling all an inventory of all running services e.g. Tomcat 7, NGINX, etc.. (and their versions) into a central system which then pulls their SBOMs (if available), and companies that with the SBOMs generate from software by our dev teams. The combination of the two would give you a clearer picture. It will need coordination between VM and SCA vendors (or a solution in the middle). The hope would be to have a hierarchy such as:

1. Workload (container/os image)

2. Running software (PURLS/CPEs/SWIDs)

3. SBOM for each piece of software

4. Vulnerabilities for each component (ideally with VEX info)

Monitoring and Preventive controls: Security Operations teams need to monitor the environments for any indicators of compromise and or add preventative controls depending on the vulnerability and avenue of exploitation. Threat Intel is also key to understanding how it is exploited in the wild or if there are any changes to the vulnerability (e.g. the patch did not completely solve the problem etc..)

• Supply chain challenges: Security Operations team would need information from both the AppSec and Vulnerability management teams, right now each organisation has their own approach, and the information is not the easiest to come by. In short it's a manual and difficult process.?
• Promising developments: I am personally optimistic of the Open Cybersecurity Schema Framework to help solve the issue of standardising cybersecurity information between domains and vendors. Another issue I am very hopeful and supportive of is code signing. If you sign your code and the workloads they run in (containers, virtual machines etc..), you have a much better indication of what is authorised to run in your environments, and this triggers better alerts or preventive controls to stop this. Project Sigstore is used for code signing, Kyverno is a policy engine that can leverage the image signatures, and Chainguard is a promising new company tackeling supply chain security issues. It's worth noting OpenBSD has been signing their code for a while now using signigy.

It's often easy to feel pessimistic about the state of cybersecurity and not see some of the progress made. Today, many companies are preparing to address a critical vulnerability armed with lessons learned from previous issues and some new capabilities that will help along the way (SBOMs, improved tooling, specs and processes). This also has the attention of major government agencies as well as the wider technology community under foundations such as OpenSSF (sigstore, Alpha & Omega) and OWASP? Foundation (OWASP CycloneDX SBOM Standard & Dependency Track). Millions for dollars are being poured in to support open source communities, secure the most critical projects, and different parts of the ecosystem are working together to make life easier for engineering teams to address their security problems. It will take us all working together, and it will take time, but it's encouraging.

Finally in closing, for me there are a few things I would personally like to see:

• SaaS/Enterprise Software SBOMS: Large SaaS companies that are members of OpenSSF/OWASP to start looking at how they would be able to publish (securely) their SBOMs to their customers in a standardised way. (I am looking at you GitHub!)
• Security Vendors & OSS Foundations Adopting VEX: As a defender I would (love) to see more security context when a vulnerability in a software library/dependency is released. It doesn't have to be specifically VEX, but a standardised way of getting that context is needed. Also understanding if that context is added by the vendor or the software maintainer is also important.
• SWID/CPE to SBOM mapping: It's not enough for me to have the SBOM of a piece of software. I also need to know where it is in my environment! A combination of vulnerability management & SCA tooling can help there, but they need to learn how to communicate with each other (or perform each other's function).
• Signed releases: One thing the human immune system excels at is self/non-self identification, we need that in the digital realm. The ability to sign everything from the OS, the services running on top of it and applications. This would allow runtime security solutions to make a lot better decisions regards to monitoring and preventative decisions.