Issue #20 | April, 2024
Backdoors to Somewhere Else
The discovery of malicious code inside the XZ Utils project sent shockwaves inside the entire IT community, raising questions about the development process of open-source software and how distributions are integrating vast amounts of code made by hobbyists. One question, however, is likely going to remain unanswered: where was this backdoor supposed to lead?
Supply chain attacks use vendors or projects like XZ as stepping stones to reach their ultimate targets. This could have been an attack on government institutions, the water supply, defense contractors, or major software developers. XZ itself was just a necessary step to reach corporate Linux distributions – and we know this because of a few lines of code that specifically triggered only on certain flavors of the operating system.
Realizing that your vendors and software have value to attackers because you or someone else uses them is key to understanding third-party risk, and it’s also why we always talk about measuring risk based on relationships. For the affected software, XZ Utils is a system component, so this is a system backdoor. Everything else about this story is just about something breaking at the weakest link, as usual.
We’ll go into more detail on everything that happened just below. We’re also following up on the United Health incident we reported last month and going over new third-party incidents that were disclosed in March. As usual, there are new regulations we think you will probably want to check out, as well as new guidance on third-party risk management that was published by the press and the NSA. Enjoy!
XZ backdoor compromises Linux distributions
Software engineer Andres Freund was just trying to figure out why his OpenSSH server was running a little slow, but he ended up uncovering a multi-year campaign that successfully backdoored several Linux distributions through the XZ Utils project. While many stable flavors of Linux had yet to integrate the malicious XZ version and remained unaffected, the incident sparked several discussions regarding the security of open-source projects and their impact on the software supply chain.
Many events involving this story are remarkable – not just because the backdoor was found by sheer luck, but also due to the amount of effort and persistence that went into gaining the trust of the project’s maintainer and engineering an obfuscated backdoor.
Freund’s message to the OSS-Security mailing list from March 29 explains?how the backdoor was found?while also impressively managing to piece together most of the events that took place, but it doesn’t cover the more recent findings. We recommend this?gist?which has a great summary, and this?good post?by Evan Boehs that details the “contributions” made by the malicious personas. The excerpt below is from?the Risky Business article:
What happened:?A backdoor mechanism was discovered in XZ Utils, a library that supports lossless compression. The library is extremely popular and is used with most major Linux distros and with a ton of Linux and macOS apps.
What does the backdoor do:?In simple terms, it intercepts SSH RSA key decryption operations, which are redirected to the backdoor code. This allows the attacker to pass special arguments during an SSH auth operation and execute code on remote systems if they use a backdoored XZ Utils version.
You can also hear this story told directly by Andres, as Risky Business also?interviewed him on their excellent podcast.
As far as official communications go, there is?a CISA advisory, a?security alert from Red Hat, and?an announcement from XZ’s maintainer, Lasse Collin. The issue is now known as CVE-2024-3094 – it’s a Remote Code Execution vulnerability with a CVSS score of 10, but the exploitation is?gated, requiring an attacker-controlled key – even though we know of the backdoor now, there’s still no way to use it or detect vulnerable systems over the network.
So far, there has been no mention of the backdoor being used in the wild. Given the previous incidents where the software supply chain was compromised (SolarWinds, CCleaner, 3CX), the targets are usually large organizations in key sectors (telecommunications, software, government, energy), which rarely run bleeding-edge Linux distributions. If that’s the case, then the attackers ultimately failed.
As significant as this story is, it’s unfortunately complex, rendering it difficult to quickly summarize for the layperson. Despite the “XZ backdoor” moniker, XZ itself remained “safe.” The malicious code becomes relevant inside OpenSSH, and even then, it only targets distributions that use .deb and .rpm packaging – which are Debian, Red Hat, and their derivatives (such as Ubuntu and Fedora). The posts in?the Gentoo bug tracker, for instance, note that the malicious code was not triggered.
Open-source projects often split their code into two components, one being a library for other applications. The library part of XZ is?liblzma, aimed at compressing and decompressing data with the well-regarded LZMA algorithm from 7-Zip. When OpenSSH is patched to use systemd – the init system required by Red Hat and Debian –?liblzma?is loaded by OpenSSH.
This isn’t evidence of some specific problem in XZ or systemd that happened to hit those distributions, however. The attackers likely worked backward – they wanted to compromise OpenSSH in Red Hat and Ubuntu, so they looked at the libraries that could be used, found?liblzma, and devised an attack.
The malicious code was added to the official release of XZ by “Jia Tan,” an online persona that appeared on GitHub in 2021. Through public and private efforts, Jia Tan gained Lasse Collin’s trust, while additional personas?pressured Collin?into delegating tasks, accusing the maintainer of “choking” the project. Another social engineering effort was made to pressure distribution maintainers into including the newest (and backdoored) releases of XZ.
“Jia Tan” obfuscated the backdoor code by splitting it into three parts:
There is a fundamental problem with?overwhelmed maintainers and social engineering, and Linux distributions should be looking for ways of preventing this from happening again. In this incident, microbenchmarks succeeded where everything else failed, and the community that runs bleeding-edge packages saved the day. But it’s not acceptable to rely on fortunate events like this, and running backdoored code isn’t appealing to anyone – even to beta testers.
Law enforcement agencies disrupt LockBit ransomware operations
LockBit, a notorious ransomware operation, was disrupted by U.S. and U.K. authorities:
U.S. and U.K. authorities have seized the darknet websites run by LockBit, a prolific and destructive ransomware group that has claimed more than 2,000 victims worldwide and extorted over $120 million in payments. Instead of listing data stolen from ransomware victims who didn’t pay, LockBit’s victim shaming website now offers free recovery tools, as well as news about arrests and criminal charges involving LockBit affiliates.
Dubbed “Operation Cronos,” the law enforcement action involved the seizure of nearly three-dozen servers; the arrest of two alleged LockBit members; the unsealing of two indictments; the release of a free LockBit decryption tool; and the freezing of more than 200 cryptocurrency accounts thought to be tied to the gang’s activities.
It was later stated that an individual responsible for LockBitSupp, an online persona closely affiliated with the group, has “engaged with law enforcement,” a claim that was met with some skepticism.
There is a little bit of context to unpack here. Ransomware operations are complex in many ways, and it is commonly understood that they enjoy unofficial state protection – they become unreachable to outside enforcement as long as they follow certain rules. Analysts from Cisco Talos have called them “privateer” groups for this reason.
Law enforcement could be making a clever attempt to break the conspiracy that keeps these groups going, even if they have no means of directly pressuring parties outside of their jurisdiction. By sowing enough doubt in the operation, key people may want out of the deal, and the aftermath could work in favor of the authorities trying to dismantle the operation for good. Conti is a good example of this: the group was thrown into disarray after a single (but major) chat leak.
Furthermore, few active groups compare to LockBit. A recent report from NCC Group found that LockBit was the most active ransomware operator in an already concerning landscape. By their numbers, 2023 had the highest number of victims ever recorded, with an 84% increase in ransomware incidents. Despite the challenges, even temporary relief is enough reason to celebrate – and we intend on following any major developments in this story.
We’re moving on to other topics related to the government and security. The NIST released version 2.0 of their popular Cybersecurity Framework (CSF). As we already covered back when we mentioned the drafts of the new standard, the CSF added a “Govern” function focused on “the establishment of cybersecurity strategy and cybersecurity supply chain risk management.”
The U.S. government is also trying to push new requirements onto contractors. One rule, demanding “full access,” was deemed particularly problematic. The “full access” requirement appears to be mostly related to incident response, however. The government may be looking to accelerate the formal step of getting a court order, as any cyber incident in government institutions is usually under the close watch of the FBI or a similar entity. The industry must carefully consider its arguments if the public catches wind of this discussion – saying you don’t like the government to investigate you because you’re trying to protect people’s privacy just after getting hacked will be very difficult.
President Biden ordered an investigation into the risks posed by Chinese-made “smart” cars. This is yet another chapter on the geopolitical and supply chain tensions that we’ve been seeing, but cars will draw more attention than the previous disputes over semiconductors, 5G equipment, cameras, and drones.
Nevertheless, cars are still part of the transport and logistics infrastructure of the country. To go along with that, the Biden administration is looking into Chinese cranes being used in U.S. ports (White House announcement here, Department of Transportation advisory here). Given how much trust is being eroded – even more so in light of the Volt Typhoon actor mentioned earlier – it’s difficult to imagine these investigations leading to an outcome favorable to Chinese suppliers.
The government is also looking to improve the security of small office/home office routers, and CISA has renewed the Information and Communications Technology Supply Chain Risk Management Task Force for another two years. This task force is aimed at producing guidance or frameworks to improve the security of the supply chain as it relates to IT.
The FTC has disclosed some details of the terms it is imposing on Blackbaud to settle charges over an incident that took place in 2020. The noteworthy part is that the company will be required to delete personal data “it doesn’t need,” which shows the FTC’s willingness to make clear operational demands of companies – a practice that could have been considered an overreach in the past.
And finally, a small update to the MGM Resorts incident from last year: the company has disclosed that regulators are investigating the incident.
Cyberattacks hit Change Healthcare, Viamedis, Romanian hospitals, and Covid e-passports
There were quite a few healthcare-related cybersecurity breaches in the last month. One major incident involved Change Healthcare, causing?disruptions at several pharmacies and hospitals?that rely on their systems to process prescriptions:
The cyberattack at Change Healthcare began on February 21 early on the U.S. East Coast, causing widespread outages at pharmacies and healthcare facilities. Change Healthcare said it took much of its systems offline to expel the hackers from its systems.
Change Healthcare’s incident tracker page shows nearly all of its customer-facing systems remain offline.
Hospitals, healthcare providers and pharmacies have reported that they are unable to fulfill or process prescriptions through patients’ insurance.
The outage resulted from an attack by?the BlackCat ransomware?group, which claims it has a total?of 6 TB of data. Since the incident affects subsidiaries that Change Healthcare acquired, two separate pages are tracking the incident: one from?Optum, and another at?UnitedHealth Group. There’s also?an SEC filling, as is now required. It appears the situation has improved considerably, and many systems are back online.
领英推荐
Researchers who follow the crypto wallets belonging to ransomware groups noted that BlackCat received a US$ 22 million payment. BlackCat isn’t coming out of this unscathed, however, as?reports claim the gang is imploding. One “representative” of the group posted to a crime forum that the ransomware source code will be sold and that the operation is shutting down.
Many “ransomware-as-a-service” operations (“RaaS”) rely on affiliates that focus on finding their way inside corporate networks. Normally, these affiliates receive a share of the profits after a ransom payment is negotiated and received by the group.
Things aren’t always that smooth between criminals, and that was the case this time. The affiliate responsible for the breach at Change Healthcare – who goes by “Notchy” – stated they didn’t get their share. This accusation likely didn’t go over well with other affiliates, but the shutdown notice followed soon after.
BlackCat was already on shaky ground after being infiltrated by the FBI and having their Tor website seized. If their leaders were looking for a way out, a major payout like this – which is probably going to make them even more of a focus for law enforcement – may have sealed the deal.
While it’s too early to say that the group is over, Change Healthcare and its customers still have a problem. Notchy is said to be in possession of the stolen data, which could be released even if the company did pay the ransomware group.
Ransomware is considered?a top hazard for the health tech sector. Coincidentally, on the same day the outage at Change Healthcare started, the Department of Health and Human Services (HHS) announced its?second settlement over a ransomware attack. The incident took place in 2019 at Green Ridge Behavioral Health.
In France, criminals managed to?phish the credentials for two separate health systems?that house information on 33 million individuals – half the population of the country. The systems are operated by Viamedis and Almerys, two service providers that work with over 150 mutual insurance partners. Here’s?additional coverage from Bleeping Computer, and?analysis at ISACA.
This was not the only data leak. In India, 3.5 million individuals were exposed after?Covid-19 e-passport data was compromised. In the US, Rotech Healthcare is reviewing an incident at Respironics, a Philips unit that they work with, to come up with a list of affected patients.
And while a children’s hospital in Chicago was forced to?go back to pen and paper, more than?100 Romanian healthcare facilities suffered disruptions?after a ransomware attack hit at least 25 hospitals, and systems were taken offline to prevent the malware from spreading.
Bringing this section to a close, we have the news that the Healthcare and Public Health Sector Coordinating Council published a Cybersecurity Strategic Plan (HIC-SP,?PDF here) that addresses several issues, including third-party risk. The healthcare sector is a key infrastructure when we think of systemic risk and, as these attacks have shown, incidents have far-reaching effects. They’re also not being spared by criminals. Additional coverage of the plan can be found here.
Cloudflare, AnyDesk, Bank of America, and others disclose cyber incidents
Content delivery and web security company Cloudflare disclosed that hackers managed to breach some of its corporate systems?using credentials stolen from the Okta breach:
To access its systems, the attackers used one access token and three service account credentials stolen during a previous compromise linked to Okta’s breach from October 2023 that Cloudflare failed to rotate (out of thousands were leaked during the Okta compromise).
Cloudflare detected the malicious activity on November 23, severed the hacker’s access in the morning of November 24, and its cybersecurity forensics specialists began investigating the incident three days later, on November 26.
The Okta breach from October 2023 is the one in which hackers managed to access Okta’s support system and obtain information sent by customers inside tickets. Some of the data sent for troubleshooting contained valid sessions. As per?Cloudflare’s blog post on the incident, the access was limited to its Atlassian environment.
Bank of America is?warning customers?of a data breach that took place at Infosys McCamish Systems, a third-party service provider. Meanwhile, a researcher?found 380 million exposed data records, including customer data, associated with Zenlayer, a global network services provider.
Yet another leak?exposed 155,000 drivers?working with student rideshare startup HopSkipDrive. The company explained this data was stored in “third-party applications,” but the applications weren’t named and it’s not clear if they were at fault. Of note here is the fact that this is only being disclosed nearly a year after it happened.
In Germany, control systems provider PSI Software is?struggling with a ransomware attack. PSI provides systems for European energy suppliers, but it isn’t clear what kind of setbacks customers are facing while PSI systems are down. As of early March, the response to the incident was still ongoing.
Remote access solutions developer AnyDesk also?disclosed a breach. They assured customers that their software is still safe to use, but they did release a new version with a refreshed code signing certificate (their previous one was stolen) and revoked passwords for their online portal. As they have numerous corporate clients, this breach could have been very concerning. AnyDesk’s statement can be found?here.
Our last story here has a bit of both hardware and software supply chain woes. Camera and smart home appliances manufacturer Wyze initially revealed that a glitch?allowed “at least a dozen users” to see other people’s camera feeds. Then, after determining that the problem was caused by a third-party caching library, they revised that number to?13,000 customers.
In smart appliances, customers usually have no viable software options other than the one provided by the hardware maker – any mishaps in the official service or software mean the hardware is functionally useless. Any “third-party library” (or business partner) the hardware maker decides to integrate into their ecosystem is part of the service. This level of “connectedness” between parties is unlike anything we used to have before the digital world and the internet, so we must take that into account when building our businesses.
Gartner: top trends to shape cybersecurity by 2024
The risks posed by third parties made it into?another list of trends and recommendations from Gartner:
Overall, Gartner advises IT security leaders to improve organisational resilience by implementing continuous, pragmatic, business-aligned risk management efforts across their organisations’ digital and third-party ecosystems. This includes expanding the role that identity and access management plays in reducing cyber security risk.
This paragraph is packed with guidance – perhaps more than it appears at first glance. Striving for “continuous, pragmatic, and business-aligned risk management” has a lot of implications for how a business deals with risk.
It echoes what we’ve said in the past about the role of continuous monitoring and how important it is when it comes to making sure that what you see about a partner’s security is factual and up to date. But being pragmatic and business-aligned also establishes a foundation for looking at the risk posed by partners based on their relationship with the business, which is another suggestion we often make.
In this perspective, a vendor is not high risk simply because it doesn’t tick all the boxes the company is looking for. Instead, it is high risk when it is performing a critical task for the business or gaining access to its critical data. Not all partners are equal in this regard. The more critical a vendor is, the more worthwhile it is to build a closer relationship for managing risk and to have the appropriate procedures for a joint incident response.
Having proper controls for the vendor security is essential for regulatory compliance, too. Law firm Squire Patton Boggs published an?overview of third-party incident reporting?under the new SEC rules that explains the whole context behind the new SEC rules and how the Commission decided to not make certain concessions when it came to incidents at third parties because their understanding is that investors don't care about where an incident happened. What matters is the impact to the business.
In order to quickly determine the materiality of incidents, companies need to implement proper controls and monitoring throughout their supply chain.
5 software supply chain attacks you can learn from
There’s an interesting recap of software supply chain attacks at ReversingLabs, covering incidents at?3CX, NuGet, MOVEit, and others.
Software supply chain threats have spiked dramatically over the past three years (up 1,300%), and attacks continued to rise in 2023. The analyst firm Gartner found that almost two-thirds (61%) of all U.S. businesses were directly impacted by software supply chain attacks between April 2022 and April 2023.
We have a couple more links with new research on software supply chain security. At Phylum, analysts are?uncovering more malicious packages?in npm, the Python package repository. Their research indicates some of these packages were published by a state-sponsored group.
Lastly, a survey of 368 IT professionals in the United States and Canada carried out by Data Theorem on behalf of the Enterprise Strategy Group found that 91% of organizations have experienced a software supply chain incident in the last 12 months, often due to vulnerabilities and cloud misconfigurations.
We leave you with a single bonus link this time, also covering software supply chain attacks, but specifically about AI models. See you again next month!
Cybersecurity researchers have found that it’s possible to compromise the Hugging Face Safetensors conversion service to ultimately hijack the models submitted by users and result in supply chain attacks.
“It’s possible to send malicious pull requests with attacker-controlled data from the Hugging Face service to any repository on the platform, as well as hijack any models that are submitted through the conversion service,” HiddenLayer said in a report.