Bad Voltron Attacks — when the sum is worse than the parts

On Friday, March 29th, the information technology industry had a near miss. A lone developer, Andres Freund, to whom we owe a huge debt of gratitude, discovered an intentional backdoor in a xz, widely used compression library in the Linux ecosystem. This backdoor seems intended to compromise SSH server communication integrity, which, if it had made it into production Linux distributions, would have embedded a very sophisticated and nuanced backdoor into SSH, one of the primary communication protocols stitching the internet together. Fortunately, Mr. Freund acted quickly, and major Linux distribution maintainers followed suit to ensure that the backdoored version of xz had not made it into production releases (details).

Many aspects of this incident are concerning, and more details are surely to emerge from the global post-mortem that began this weekend, but there is one characteristic that sticks out to me, one that has long concerned me in a theoretical sense, and now has exhibited itself in the wild: malware broken up into obfuscated parts that only become virulent/active when brought together by common system architectures and practices.

This seems to be what happened with xz: compression/decompression actions were modified in a way that when involved with SSH (and perhaps other software) allowed malicious code to be executed as root. When considered with the rest of the sophisticated methodologies used to obfuscate its true nature, this malware revealed a devastating tactic that I’ve come to call Bad Voltron.

For those not familiar with Voltron, it is an animated franchise featuring a heroic super robot which assembles from five lion robots to become more powerful than any of them individually.

Similarly, Bad Voltron attacks are pernicious since their true power is only realized when their component parts are brought together by no fault of system designers and maintainers. The xz package has been part of SSH and Linux distributions for quite some time. Fortunately, at this time, it appears that SSH and other Linux system components that this malware affected did not have malicious code in them, though a future version of this tactic that embeds fragments of malware within multiple packages seems like a natural evolution for bad actors.

Per CVE-2024–3094, this multi-stage injection reveals how multiple innocuous and obfuscated components could come together at build-time to form the malware. All of the steps and components below are part of normal software build/packaging processes, and yet provide assembly of the malware into its final form:

• The malicious injection present in the xz versions 5.6.0 and 5.6.1 libraries is only included in the tarball download package.
• The Git distribution lacks the M4 macro that triggers the build of the malicious code.
• The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present.
• Without the merge into the build, the 2nd-stage file is innocuous. In the finder’s demonstration, it was found that it interfered with the OpenSSH daemon.
• While OpenSSH is not directly linked to the liblzma library, it does communicate with systemd in such a way that exposes it to the malware due to systemd linking to liblzma.

There is likely to be a lot of discussion about supply chain security, and rightful speculation about what other Bad Voltron attacks exist in the wild right now. There will also probably be controversy about the characteristics of open source software development that contribute to this sort of attack. I personally take the position that open source saved the enterprise IT world in this case, especially if one considers what the outcome would have been if the xz backdoor had been implemented in closed-source software.

Any nation state actor with halfway decent tradecraft must have embedded agents within every major software vendor in the industry; it would be tradecraft malpractice if they didn’t. Consider what the outcome would be if those insider threats within two closed-source vendors whose software is commonly deployed together jointly conspired to build a Bad Voltron across their code. It would be very difficult to detect that split malware until it was too late because there would likely not be any common sight picture with sufficient depth across all software involved.

The fact that Linux, xz, and all the other software components involved in this incident were open source means that there is a world of developers and security researchers who can see and evaluate the code and its interactions and dependencies as it progressed through communities before it entered production distributions. In a single weekend, this incident spread from a lone discovery by Mr. Freund into a wildfire blaze across the entire industry. There were no NDAs to be signed, no barriers to investigations or actions; all evidence and code was freely sharable and actioned at the speed it needed to be.

There is a place for both open source and closed source software in the world, but it is arguable that the outcome of this incident is a triumph of the global open source community. I hope this incident motivates the enterprise IT industry, both open and closed source, to build and maintain cross-vendor lines of communication and information sharing mechanisms to find and respond effectively to this tactic.

Bad actors are getting better at what they do. They are specializing (eg. initial access brokers) and collaborating to produce higher quality malware and attacks. The good guys like Mr. Freund, and the majority of the software community made up of responsible and talented developers, need to stay united against the forces of evil and keep assembling together every day into a better and powerful force just like Good Voltron.