Issue #24 | August, 2024
Once upon a time, there was a Channel File
By this point, we’re sure you’re aware of the CrowdStrike outage and how an update – an unfortunate “channel file” – set off a series of cascading failures that delayed flights, disrupted healthcare, stopped freight, and caused billions of dollars in damages. It is yet another example of the interconnected nature of our computer systems, showing how one failure can lead to widespread chaos.
Still, systemic risk doesn’t disappear because you disconnect a computer system. Airlines can be affected by outages that disrupt other airlines in the same airport, for example. As we share physical infrastructure, we’re connected by things much closer than the cloud.
So, is it truly fair to say a single channel file caused it? Wasn’t there something more we could have done to prevent an incident like this? As philosophical as this question may seem, we must remember that this is not a drill. For us in cybersecurity, it’s time to think about how to secure this great interconnected machine we have built.
As usual, Alice in Supply Chains is here to bring you some news stories and thoughts to help us find some answers. There’s some more commentary on the CrowdStrike breach right below (more tangible than this, we promise!), our usual round-up of third-party security incidents, guidance on risk management, and security research.
We hope you enjoy it!
CrowdStrike update causes mass IT outage
The CrowdStrike outage wasn’t simply a third-party risk management story – it was perhaps the tech story in July. The New York Times highlighted the impacts to hospitals and airlines, DC Velocity noted that the outage stopped freight flows and the BBC described the situation using the word “chaos.” The cost is estimated at $5.4 billion just for Fortune 500 companies.
It’s not often in IT that mistakes have such clear, direct, and tangible consequences. Nevertheless, this is still about data – it’s about data not being available, or not being trustworthy. It’s also about a software package that couldn’t process its own data without crashing both itself and the operating system.
If you’re still looking for a good explanation of what happened, Patrick McCormack posted a detailed write-up on LinkedIn. Some inaccurate technical details have been floating around – such as the idea that Channel Files contain code to be executed by the device driver – so you might want to take a quick look at McCormack’s explanation just in case.
To avoid repeating what you’ve probably already read elsewhere, we will look at this outage from the perspective of systemic risk. Ahmed Sallam commented on the challenges of securing Windows systems (there’s also a follow-up post).?
The debate over whether EDR/XDR should run in Windows kernel mode is valid. However, with the current Windows architecture, running EDR/XDR entirely in user mode is infeasible due to simple techniques that can bypass all user-mode protections.
Over the years, significant efforts by Microsoft Research, myself, and a few others have been invested in rearchitecting Windows (from within or from underneath) to create such an isolated environment. This would allow security measures to run safely while effectively monitoring and controlling Windows operations without impacting system performance. Despite these efforts, Microsoft has yet to accept / adopt these architectural changes in a manner accessible to others.
Some of the first articles regarding this outage treated it as a “Microsoft outage.” It’s easy to see why – the systems displayed Blue Screens of Death, the trademark of a Windows crash. Even though everyone knows that this was a CrowdStrike issue now, plenty of people want to pass the blame around.
While CrowdStrike posted guidance to help clients, the company also attempted to buy some goodwill with $10 gift cards, which didn’t go very well – at least in media headlines.
For its part, Microsoft blamed regulations that allegedly prevented it from making changes that would restrict security software from working so deeply into the Windows kernel. That's disingenuous, though. Regulators essentially just ruled that competing endpoint security solutions must be given the same level of access given to Microsoft's — a very reasonable attempt at ensuring a level playing field. It is by the company's own choice that Microsoft's endpoint security products also use kernel drivers — they could have long ago developed userland APIs, migrated their own products to use them, and then gradually locked competing products out of the kernel as well...
Microsoft also has its own issues often – they just don’t take down airlines and hospitals to that extent. The status page for Windows 11 23H2, for example, mentions an issue that causes the system to boot into BitLocker recovery (which is another way of saying they don’t boot without intervention from IT, much like the CrowdStrike incident).
One way or another, the finger-pointing may not help Microsoft here, as Delta?is suing Microsoft and CrowdStrike while CEO Ed Bastian calls Windows “probably the most fragile platform”, an opinion he likely didn’t form over a single issue. As Dan Geer points out, many knew for decades that this was bound to happen, and perhaps this is an opportunity to make sure it doesn’t happen again. It’s kind of miraculous that incidents like this one don’t happen more often, as Tenchi CTO Alexandre Sieira said.
As much as it’s tempting to blame CrowdStrike and rest easy by thinking we have all the answers, perhaps we should be asking what kind of computing environment we need to create so this cannot happen – or at least how our systems (and our economy) should behave so that they are actively trying to prevent this from happening.
Snowflake incident updates and MFA news: issues at Google and Twilio?
The aftermath of the Snowflake incidents we reported on last month could drag on for quite a while, now that a class-action lawsuit was filed against AT&T due to the data breach:
The case (3:24-cv-1797) is a class-action lawsuit against the Dallas-based telecommunications giant, filed in the U.S. District Court for the Northern District of Texas late Friday by 15-year AT&T customer and named plaintiff Dina Winger. The suit alleges AT&T wasn’t transparent about the severity of the breach, didn’t safeguard important data from malicious parties and earned “unjust enrichment” from customers after failing to protect their information.
CNBC reported that Snowflake shares slipped when it came to light that AT&T was one of the victims (this was confirmed by AT&T).
The Ticketmaster data leak, also linked to Snowflake, is also causing logistical problems, according to a story by 404 Media. The company also finally notified users and gave recommendations to affected customers, which are believed to be 560 million people worldwide.
The Neiman Marcus and Advance Auto Parts breaches have also been confirmed to involve 31 million and 2 million customers, respectively.
For its part, Snowflake announced that its customers are now able to enforce multi-factor authentication (MFA) – making the entire market wonder why such a basic security configuration option was not previously available for an enterprise product.
While we are on the topic of MFA, we could say it had a rough month in July. Twilio has confirmed that an API endpoint allowed hackers to confirm 33 million phone numbers used for MFA and we at Tenchi found that Google had an issue with 2FA enforcement after a service update that was meant to enforce mandatory 2FA for admin users.
Hackers leak documents from Pentagon IT provider: breaches round-up
Hackers have leaked internal documents stolen from a service provider of a service provider of the United States government. From Bloomberg:
Hackers have leaked internal documents stolen from Leidos Holdings Inc., one of the largest IT services providers to the US government, according to a person familiar with the matter.
Leidos recently learned of the issue and believes the documents were stolen in a previously disclosed breach of a Diligent Corp. system it used, said the person, who asked not to be identified because the information isn’t public. Leidos is investigating the issue, the person added.
A data breach notification suggests that Diligent Corp notified Leidos of data leaks in November 2022, so the news is that it has been confirmed that the stolen data is in circulation. The story is also available from CPO Magazine, The Register, and Reuters, among others.
Meanwhile, Brian Krebs reports that criminals bypassed Google’s email verification to create and associate Google Workspace accounts with domains they didn’t own. This allowed them to impersonate domain holders, bypassing authentication to services that used “Sign-in with Google.” In one case mentioned by Krebs, the hackers managed to access the user’s Dropbox account.
A similar issue happened at Squarespace, where some domains were hijacked. According to another report by Krebs, the campaign mostly targeted cryptocurrency businesses and redirected them to phishing sites. The attack is said to be related to domains migrated to Squarespace when the company acquired Google’s domain name business in 2023, but Squarespace blamed the incident on an OAuth weakness.
There were a couple more cryptocurrency-related incidents involving third parties. “Web3 Identity Solutions Provider” Fractal ID disclosed a breach after an attacker managed to gain operator access to their systems. The intrusion was traced to a compromised credential – this privileged operator reused a password that was compromised in 2022 (the operator was let go after the incident for not following security policy). As Fractal provides identity verification for cryptocurrency platforms, users of these platforms could be among the 6,300 individuals affected.
Cryptocurrency exchange platform Gemini also notified users of a data breach after a banking partner suffered a security incident. Gemini says the stolen data includes bank account numbers and names, but no other personal information, such as addresses or social security numbers.
A ransomware attack against South Africa’s National Health Laboratory Service (NHLS) is causing delays for “millions of blood tests,” according to Health-E News. The same outlet explains that “most of the country’s public health facilities don’t have their own laboratories to run blood tests,” so they all rely on the NHLS for that service. The attack was attributed to the BlackSuit group.
Another ransomware attack by BlackSuit motivated a disaster declaration in Clay County, Indiana, after system disruptions prevented the county from functioning normally.
India’s Piramal Group blamed a third party for a data leak that contains information on its employees, but it’s unclear if this is a vendor or an outside third party that happened to collect this data. Shopify also denied being breached after a hacker posted a data leak on a hacking forum and explained that the information was exposed by an unnamed third-party app.
In Singapore, another breach exposed the personal data of 128,000 individuals – all being clients of 12 moneylenders that used the same IT services provider, according to the country’s Ministry of Law.
The last third-party security incident we have here is also a software supply chain issue. Yet another malicious package was distributed in the Python Package Index repository, one that is intended to steal Google Cloud passwords. There’s something curious about this attack, however: the package had a limited number of targets and checked the system’s Universally Unique Identifier (UUID) against a hard-coded list containing 64 hashes. Although the package was only downloaded 59 times, that would be enough to hit over 90% of its intended victims.
领英推荐
Judge dismisses most of the SEC charges against SolarWinds, but the lawsuit will proceed
There is some third-party risk management news from each branch of the US government this month. Starting with the courts, a judge dismissed most of the lawsuits filed by the Securities and Exchange Commission (SEC) against SolarWinds. However, the core issue surrounding their “security statement” posted on their trust center remains:
Judge Paul Engelmayer of the U.S. District Court Southern District of New York sustained the SEC’s claims of securities fraud based on SolarWinds’ security statement. However, the court dismissed other claims, including all claims involving post-Sunburst disclosures.?
The court also dismissed claims related to the company’s internal accounting and disclosure controls and procedures, as ill-pled.
This means that the lawsuit is just starting and can still break new ground for the SEC as long as regulators can argue that this “security statement” was relevant to investors. Many legal teams probably want to follow this case – especially in light of the rules for cybersecurity risk management and incident disclosure that the Commission adopted in 2023.
As for the Executive, the Department of the Treasury, through its Cloud Executive Steering Group (CESG), started publishing resources for secure cloud adoption in the financial sector. As part of this initiative, the Financial Services Sector Coordinating Council published a report titled Cloud Outsourcing Issues and Considerations.
Meanwhile, the Board of Governors of the Federal Reserve System, the Federal Deposit Insurance Corporation (FDIC), and the Office of the Comptroller of the Currency (OCC) issued a joint statement (PDF) on “arrangements with third parties to deliver bank deposit products and service.” It has a series of recommendations and warnings.
However, legislators don’t always agree with these movements. The House Financial Services Subcommittee on Financial Institutions and Monetary Policy held a field hearing entitled “Financial Institution-Fintech Partnerships: Leveraging Third-Party Relationships to Increase Access to Financial Services.” Remarks from Chairman Andy Barr, a Kentucky Republican, are available from the House’s website, and he criticized the FDIC and the OCC for some of their actions (he’s a member of the opposition).
In Europe, the United Kingdom is introducing legislation that is expected to impose mandatory reporting for ransomware attacks. According to The Record, the Cyber Security and Resilience Bill was supposed to cover all businesses in the country, and extortion payments would require a license. The new version will only apply to “regulated entities,” but we’ll have to wait for the final version to see how far into the supply chain the requirements will go. Additional coverage is available from The Independent.
A similar proposal for disclosing ransomware payments, the Cyber Security Act, is currently under discussion in Australia. All businesses “with an annual turnover of more than $3 million,” according to ABC News, would be covered. The reporting mentions that the proposal has drawn criticism, so we’ll also have to wait and see what comes out of this effort.
Lastly, Germany has decided to bar the use of components from Chinese companies ZTE and Huawei in its 5G infrastructure. As The Associated Press notes, the United States has been trying to pull its allies away from Chinese equipment, arguing that the government in Beijing could use its control over the infrastructure for cybersnooping or sabotage. Despite China’s denials, Japan, Australia, New Zealand, Sweden, Portugal, and Canada have all taken steps to restrict the use of Chinese equipment.
PKfail: security research shows device manufacturers kept insecure key, undermining Secure Boot
Researchers from Binarly found that hundreds of devices include a compromised security key in their BIOS/UEFI, allowing unauthorized code to run during boot. This happened even though the keys were flagged as untrusted:
The test keys are shared with commercial partners and vendors with the expectation they be treated as completely untrusted.?
By scanning an internal dataset of firmware images, we confirm that devices from unrelated vendors contain the same Platform Key, meaning that these keys must have been generated at the root of the firmware supply-chain.?
These test keys have strong indications of being untrusted (for example, the certificate issuer contains the “DO NOT TRUST” or “DO NOT SHIP” strings).
Secure Boot requires all boot code to be signed by one of the keys installed in the UEFI firmware. It helps prevent certain attacks that are difficult to remediate, such as “bootkits” (boot sector rootkits). An attack that leverages this untrusted key could essentially undo many of the OS-level security enhancements from the last 20 years, as malware code could run before the OS itself. Additional reporting is available from Dan Goodin at Ars Technica.
In more research news, Google has released its newest Threat Horizons report (PDF). Google’s Anton Chuvakin published a post with “informal, uncertified, unreviewed and otherwise completely unofficial” commentary here. The report says that 58.8% of intrusions were motivated by cryptocurrency mining and that weak credentials and misconfigurations are the most common vectors for initial access (47.2% and 30.3%, respectively).
A study from last year found a link between ransomware attacks and mortality rates in hospitals. It appears there was no coverage of this when it came out, but Risky Business picked up the story last month.
Gartner published results from a new survey on the top priorities for legal, compliance, and privacy leaders. Third-party risk management was among the top 5 for 40% of respondents and was the number one priority for 6%.
Bringing this section to a close, we have a follow-up to a vulnerability found by Assetnote in ServiceNow that was mentioned in a previous edition due to their decision to withhold technical details and give time for users to patch. The full write-up on the vulnerability is now available.?
Building Resilience in the Chip Supply Chain and addressing third-party risks
“Building Resilience in the Chip Supply Chain” notes that we semiconductor manufacturers need to improve their cybersecurity, especially given their role in driving innovation:
Most of the attention — in part due to the CHIPS and Science Act signed into law by President Biden in August 2022 — has been focused on expanding manufacturing capacity domestically and diversifying the supply chain to not be overly dependent on singular suppliers and/or fabricators. While those actions are important, one critical yet overlooked facet of resilience is ensuring the supply chain is digitally secure.
The semiconductor supply chain is highly interconnected, with sensitive IP constantly being shared across fabs, designers, suppliers, installers, etc. This means that breaches like the ransomware attack of MKS Instruments materially impact not just MKS, but the vast web of third parties it works with.
It seems like the industry may have already noticed that this is a problem. TSMC (which makes chips for Apple, AMD, and Nvidia, among others) announced that it hosted its first supply chain security workshop. The fact that it’s the first one for a company that earns billions of dollars in revenue and is almost 40 years old is perhaps a sign of the growing relevance of TPCRM initiatives. The workshop idea is also one to take note of – we always say that cooperation is key, and each industry should find what works for them.
The Basel Committee on Banking Supervision issued a consultative document proposing “Principles for the sound management of third-party risk in the banking sector” (PDF, 21 pages). It’s essentially a long checklist for each step of working with a third party – from hiring to termination. As usual, guidelines like these can be helpful for other industries, so you may want to use them as a reference for your third-party risk management processes.
Freight Waves has an article on addressing third-party risks targeted at logistics companies. It’s not as in-depth as the document from the Basel Committee, but it serves as an indication of how third-party risk awareness is growing. It can also help people from that industry to find resources to start this conversation with company leadership, as the National Motor Freight Traffic Association is holding a cybersecurity event in October.
Finally, Sophos published a blog post with a reminder that the Network and Information Systems (NIS) 2 directive comes into effect in October, in less than two months. Their blog has a summary of what the directive is and its impacts, and they also have a whitepaper with more information. According to Sophos, covered companies have to make sure that service providers are also compliant with the directive.
This is all we have for this issue of Alice in Supply Chains… well, aside from three more bonus links below!
See you again next month.
CocoaPods, an open-source dependency manager used in over three million applications coded in Swift and Objective-C, left thousands of packages exposed and ready for takeover for nearly a decade – thereby creating opportunities for supply chain attacks on iOS and macOS apps, according to security researchers.
What if there was a supply chain attack that could provide an attacker with direct access to core infrastructure within thousands of companies worldwide. What if that attack required no social engineering and could be executed within a few hours?
Between April 2nd, 2024 and May 21st, 2024 that attack would have been possible, and the only prerequisite would be signing up for an account on GitHub.
A threat actor on BreachForums is selling an unverified npm vulnerability for account takeover, but npm has not officially confirmed the existence of this security concern.