Recognizing the Critical Role of Software Supply Chains: Lessons from a Narrow Escape

Let's start with SBOM.

SBOM stands for Software Bill of Materials. It is a detailed list of all the components, libraries, modules, and other dependencies that make up a software application. The concept is analogous to a list of ingredients on a food package, providing a comprehensive inventory of everything included in the software. Here are the key aspects of SBOM:

1. Components: It includes open-source and third-party libraries, proprietary code, and other software elements that are part of the application.

2. Versions: Each component listed in the SBOM is specified along with its version, which is critical for identifying vulnerabilities and ensuring compatibility.

3. Licenses: Information about the licenses under which each component is distributed is also included, helping organizations comply with licensing requirements.

4. Dependencies: The SBOM details the relationships and dependencies between different components, which can help in understanding the software architecture and potential impact of changes or vulnerabilities.

5. Security: By maintaining an SBOM, organizations can better track and manage security vulnerabilities, as it allows them to quickly identify which components might be affected by a newly discovered vulnerability.

6. Compliance and Transparency: SBOMs support regulatory compliance and promote transparency in software supply chains, making it easier to audit and ensure that all components are up to date and secure.

SBOMs are increasingly important in the context of software supply chain security, as they help organizations manage risk, improve software quality, and respond promptly to vulnerabilities.

Now, let's see the XZ Utils that was almost done!

In April 2024, The world recently dodged a significant cybersecurity threat—a nation-state attack that had the potential to cause widespread damage across the global Internet. This incident, though it didn't happen, highlights a critical issue: the security of the Internet relies heavily on numerous small pieces of software, often maintained by unpaid, part-time volunteers.

These volunteers, while dedicated, are often overworked and distracted, leaving the software they maintain vulnerable to exploitation by malicious actors. Despite the importance of their work, there's not enough support to ensure these software components are secure.

This near-miss is a reminder that we need to pay more attention to these hidden vulnerabilities and provide better support for those who maintain our digital infrastructure. The security of the global Internet depends on it.

Another factors :

• Efficiency in Programming: Programmers prefer using pre-written code to avoid extra work.
• Code Repositories: These collections of reusable code, known as libraries, are available on platforms like GitHub.
• Variety of Libraries: There are libraries for virtually every functionality, including 3D object display, spell-checking, complex mathematics, e-commerce management, and file transfer.
• Importance in Software Development: Libraries are crucial for modern programming, acting as the building blocks of complex software.
• Modularity: The use of libraries allows for modular software development, making projects more manageable.
• Pervasiveness: Almost all software relies on multiple libraries, which can be either commercial or open source.
• Functional and Security Role: Libraries are essential not only for the functionality but also for the security of the finished software.

But, I have given some idea why things 'just happened', but what happened?

XZ Utils

Indeed, XZ Utils is not a household name but it plays a significant role behind the scenes in various software systems and applications. Developed as a part of the LZMA SDK (Software Development Kit), XZ Utils focuses on providing high compression ratios and is designed for portability and flexibility across different platforms. It supports multiple filters for compression and decompression, with LZMA2 being the default and most efficient algorithm.

This library is used extensively in Linux distributions for compressing package archives, system backups, and log files. For instance, Debian, Ubuntu, Fedora, and many other Linux-based systems utilize XZ compression for their package formats. It's also used in source code version control systems like Git for compressing repository data, and in file archivers like XZ itself or tar.xz files.

XZ Utils, like many open-source projects, embodies the spirit of collaboration and transparency in software development. Its wide adoption underscores the importance of such utilities in maintaining efficiency and managing the ever-growing data volumes in computing environments.

Open-source libraries like XZ Utils, though they may not be familiar to the average computer user, form the backbone of modern technology infrastructure. They enable developers to build upon existing, well-tested solutions rather than reinventing the wheel, which speeds up development cycles and ensures better software quality overall. These libraries often become integral parts of operating systems, web servers, databases, programming languages, and various applications we interact with daily, even if we don't directly notice their presence.

Many open-source libraries, like XZ Utils, are maintained by volunteers. XZ Utils, for example, has been managed by Lasse Collin since he created it in 2009. Collin, who has been dealing with long-term mental health issues as of 2022, is not to blame here; the problem lies within the system itself.

Starting in at least 2021, Collin was targeted by unknown individuals using aliases such as Jia Tan, Jigar Kumar, and Dennis Ens. These individuals pressured him to relinquish control of XZ Utils. In early 2023, they succeeded. Tan then spent the year methodically incorporating a backdoor into XZ Utils, disabling detection systems, and finally adding the complete backdoor. On March 25, someone using the alias Hans Jansen attempted to push various Unix systems to upgrade to the compromised version of XZ Utils.

This situation highlights several vulnerabilities in open-source software development and maintenance:        

1. Reliance on a Single Maintainer: Many open-source projects rely heavily, or even entirely, on a single individual. This creates a single point of failure, both for technical expertise and for project security.         

2. Burnout and Mental Health: Open-source maintainers often face burnout due to the workload and pressure of maintaining critical software, especially when they are volunteers. Mental health issues can further exacerbate this challenge.        

3. Targeted Attacks: Open-source maintainers can become targets of social engineering or other attacks aimed at gaining control over the project or introducing malicious code.        

4. Trust and Verification: Open-source projects rely on community contributions, but verifying the trustworthiness of every contributor is difficult, especially in large projects.        

Mitigations:        

 Shared Responsibility: Projects should strive for multiple maintainers with diverse skill sets, reducing the burden on any single individual and increasing resilience against attacks.        

 Financial Support: Funding mechanisms can help compensate maintainers for their work, reducing burnout and improving security by allowing them to dedicate more time to code review and security practices.        

 Stronger Authentication and Verification: Implementing stricter authentication and code review processes can help prevent unauthorized changes and identify potentially malicious contributions.        

 Community Awareness: Raising awareness about the challenges faced by open-source maintainers and the importance of contributing to projects can foster a stronger community and encourage more individuals to participate in maintenance.        

It is crucial to address these vulnerabilities to ensure the long-term health and security of open-source software, which plays a vital role in our digital infrastructure.        

Everyone was set to install the update. It was a routine process. Within weeks, the new version of XZ Utils would have been integrated into both Debian and Red Hat Linux, which power the majority of servers on the Internet. However, on March 29, Andres Freund, an unpaid volunteer who works for Microsoft in his spare time, noticed something unusual about the processing demands of the new XZ Utils version. It was a subtle anomaly, easy to miss and even easier to ignore. Yet, Freund decided to investigate further and uncovered the backdoor.

The backdoor was a sophisticated piece of engineering. It exploited the SSH remote login protocol by adding hidden functionality that required a specific key to activate. With that key, an attacker could use the compromised SSH to upload and execute arbitrary code on the target machine. Since SSH operates with root privileges, this code could do anything imaginable.

This wasn’t a quick hack but the result of a meticulous, years-long engineering effort. The code’s ability to evade detection in its source form, lie dormant until activated, and then exhibit immense power and flexibility, strongly suggests that a major nation-state was behind this effort.

The averted crisis in the XZ Utils case demonstrates the precarious nature of open-source software security and underscores several crucial takeaways:        

1. The Power of Vigilance: The discovery of the backdoor was only possible due to the diligence of unpaid volunteers who meticulously reviewed the code. This emphasizes the need for robust code review processes and the importance of individuals who are willing to dedicate their time and expertise to this critical task.        

2. Sophistication of Threats: The complexity and stealth of the XZ Utils backdoor indicate the involvement of highly skilled adversaries, possibly nation-state actors. This highlights the escalating sophistication of cyber threats and the need for constant vigilance and advanced security measures.        

3. Systemic Vulnerabilities: The incident exposes systemic vulnerabilities in open-source software development and maintenance, including the reliance on volunteers, the potential for burnout, and the risk of targeted attacks. These vulnerabilities require comprehensive solutions that address funding, community engagement, and security practices.        

4. Hidden Risks: The XZ Utils backdoor could have had catastrophic consequences if it had gone undetected, potentially compromising the security of countless servers worldwide. This underscores the hidden risks in software supply chains and the importance of proactive security measures to mitigate these risks.        

The XZ Utils incident serves as a stark reminder of the importance of investing in open-source software security, fostering a strong community of contributors, and implementing robust security practices to protect critical infrastructure from sophisticated threats.        

Had the backdoor gone unnoticed, it could have eventually spread to nearly every computer and server on the Internet. While its impact on Windows and macOS is uncertain, it definitely would have compromised Linux systems. Think back to 2020, when Russia's backdoor in SolarWinds affected 14,000 networks. That was significant, but this incident could have been exponentially more damaging. The disaster was avoided only because a volunteer happened to discover it. The vulnerability existed in the first place because an unpaid volunteer—who turned out to be a critical single point of failure for national security—was personally targeted and exploited by a foreign actor.

This situation highlights a serious flaw in how we manage critical national infrastructure. It was an attack on our software supply chain, subverting software dependencies. While the SolarWinds attack compromised the update process, other attacks focus on system design, development, and deployment. These types of attacks are becoming more common and effective, increasingly favored by nation-states.

The averted XZ Utils disaster underscores the urgent need for a paradigm shift in how we approach the security of critical software infrastructure:        

1. Beyond Volunteerism: While the dedication of volunteers is admirable and essential, relying solely on them for the security of critical software is unsustainable. We need to establish sustainable funding mechanisms to compensate and support maintainers, enabling them to dedicate more time and resources to security practices.        

2. Systemic Resilience:  The software supply chain is vast and complex, with numerous dependencies and potential vulnerabilities. We need to build systemic resilience by adopting a zero-trust approach, implementing robust code review processes, and diversifying the development and maintenance of critical software.        

3. Proactive Defense:  Instead of solely reacting to attacks, we need to adopt a proactive defense strategy. This includes threat modeling, vulnerability scanning, and penetration testing to identify and mitigate potential risks before they can be exploited.        

4. Collaboration and Information Sharing:  The fight against software supply chain attacks requires collaboration between governments, industry, and the open-source community. We need to establish platforms for sharing threat intelligence, best practices, and resources to enhance our collective defense capabilities.        

5. Raising Awareness:  It's crucial to raise awareness about the risks and consequences of software supply chain attacks, not only among technical experts but also among policymakers and the general public. This will help garner support for necessary investments in security and foster a culture of security consciousness.        

The XZ Utils incident serves as a wake-up call, highlighting the urgent need for a comprehensive and coordinated approach to secure our software supply chain and protect critical infrastructure from increasingly sophisticated and devastating attacks. It's time to move beyond reliance on volunteers and adopt a more sustainable and proactive model for software security.        

It's impossible to quantify the number of single points of failure within our computer systems. We also can't determine how many unpaid and unappreciated maintainers of critical software libraries are susceptible to pressure. (Again, the fault lies not with these volunteers but with an industry that exploits their unpaid labor.) Additionally, we don't know how many have inadvertently created exploitable vulnerabilities. How many ongoing coercion attempts are there? A dozen? A hundred? It's unlikely that the XZ Utils incident was an isolated case.

Finding solutions is challenging. Banning open source isn't viable; it was precisely because XZ Utils is open source that an engineer discovered the problem in time. Banning software libraries won't work either; modern software relies heavily on them. For years, security engineers have advocated for a "software bill of materials" (SBOM)—an ingredients list of sorts—so that when a package is compromised, network owners can quickly determine their vulnerability. Despite industry resistance, this idea is gaining traction and might soon become a standard practice.

The XZ Utils incident underscores a critical reality: open-source software is both a boon and a bane for security. It fosters innovation, transparency, and collaboration, but it also introduces numerous vulnerabilities due to its decentralized nature and reliance on volunteers.        

Addressing these vulnerabilities requires a multi-faceted approach:        

1. Beyond the Software Bill of Materials: While a software bill of materials is a valuable tool for identifying vulnerable components, it's merely a starting point. We need to go further by establishing comprehensive vulnerability databases, implementing automated vulnerability scanning, and developing tools for patching and updating dependencies quickly and efficiently.        

2. Incentivizing Security: We need to create incentives for developers and maintainers to prioritize security. This could involve financial rewards for discovering and fixing vulnerabilities, recognition programs for security champions, and industry standards that prioritize security in software development practices.        

3. Empowering Maintainers: Open-source maintainers need more support, both financially and technically. This could include funding for security audits, tools for code review and vulnerability scanning, and training programs on secure coding practices.        

4. Community Collaboration: The open-source community is a powerful resource for identifying and addressing vulnerabilities. We need to foster collaboration between maintainers, security researchers, and industry stakeholders to share knowledge, best practices, and resources.        

5. Government Intervention: Governments have a role to play in promoting software security. This could involve mandating software bills of materials for critical infrastructure, providing funding for open-source security initiatives, and establishing regulatory frameworks that incentivize secure software development practices.        

The XZ Utils incident is a wake-up call. It's time to move beyond reactive measures and adopt a proactive approach to software security. By investing in the open-source ecosystem, empowering maintainers, and fostering collaboration, we can build a more secure and resilient software supply chain.        

Open-source software is here to stay, and its benefits are undeniable. But we need to address its vulnerabilities head-on to ensure that it continues to drive innovation and progress without compromising security.        

This incident was a stroke of luck, but it offers a valuable lesson. Just like the power grid, communications network, and transportation systems, the software supply chain is critical infrastructure. It plays a crucial role in national security and is vulnerable to foreign attacks. It's time for the US government to acknowledge this reality and treat it as a national security issue.

Resources :

• https://www.lawfaremedia.org/article/backdoor-in-xz-utils-that-almost-happened
• https://www.schneier.com/blog/archives/2024/05/supply-chain-attack-against-courtroom-software.html