CrowdStrike Strikes Out

Oh, somewhere in this favored land the sun is shining bright,

The band is playing somewhere, and somewhere hearts are light;

And somewhere men are laughing, and somewhere children shout,

But there is no joy in Mudville - mighty [CrowdStrike] had struck out.

Casey at the Bat, Ernest Thayer

I know what you are thinking: “Another CrowdStrike article?! Give it a rest already!”

And I did almost skip writing this because of all the other great content already out there analyzing the CrowdStrike incident. However, this incident has brought out all of the Monday morning quarterbacks doing horrible post-analysis that needs to be addressed. Moreover, we have now had the time and space to see CrowdStrike's response, understand the full impact of the issue, and settle down to discuss next steps in a rational manner.

Before we get to the post-analysis and full impact of CrowdStrike's issue, though, let's do a quick recap on what happened.

What Happened

On July 19 just after midnight, CrowdStrike released a faulty configuration update for its Rapid Response Content.

The flawed update caused the CrowdStrike program to crash. Because CrowdStrike integrates into Windows with the highest possible privileges, CrowdStrike's crash caused a similar crash in the Windows host that put it into an unrecoverable state (often called the “Blue Screen of Death”).

CrowdStrike rolled back the update just an hour and a half later, but the damage had already been done. Any Windows host that was online and got the update was now stuck on a blue screen. For a more technical breakdown, see this excellent article: https://weekendbyte.com/p/crowdstrike-took-85-million-systems-offline.

Why Did it Happen

The CrowdStrike program has lots of moving parts and the Rapid Response Content is designed to update quickly and regularly to keep the overall sensor up-to-date on the latest attack patterns and malware.

CrowdStrike has a robust quality assurance testing methodology to ensure that all updates are safe. However, this particular update was erroneously marked safe due to a bug in the internal Content Validator program. As a result, the flawed update wasn't caught.

CrowdStrike has (appropriately) taken full responsibility for the issue and has made a plan for how they will improve their product and internal development.

The $10 UberEats gift cards though was definitely a mistake.

What was the Impact

CrowdStrike is an industry leader, so it makes sense that a lot of big-name companies and important industries use CrowdStrike as part of their security program.

Microsoft estimates that 8.5 million Windows devices were affected. While that is less than 1% of Microsoft's market overall, the small percentage of hosts were also some of the most critical - after all, it was the servers that were running in the middle of the night that were affected.

As a result, more than 10,000 flights around the world had significant delays or were canceled, public transit in many American cities was disrupted, hospitals and healthcare clinics faced delays and cancellations in appointments, financial systems went offline causing delays in people's paychecks, and multiple media outlets were taken off the air.

According to insurance firm Parametrix, the total financial impact to the biggest U.S. based companies alone is over $5.4 billion.

What is the Lesson Learned

This incident had massive impact to affected companies, costing huge amounts of money and time in recovery and lost business. As a result, it has garnered huge amounts of Monday Morning quarterbacking, where experts and media attempt to explain what should have happened to avoid the problem.

Most of this advice is wrong.

Some of the worst offenders I have seen are:?

• Don't enable automatic updates and/or test all tool updates in a dedicated test environment
• Don't let security tools have so much power in your environment
• Obtain more transparency from vendors about how their tools work
• Have 100% backup coverage and recovery plans

All of these can be true in a vacuum, but don't take into account the reality for most businesses in that each of these takes a tremendous amount of time and resources to manage without compromising security in the process.

The real lesson learned is to refocus on risk management. This CrowdStrike incident had a huge impact, but the likelihood of it happening in the first place was very low. Most businesses therefore won't, and shouldn't, prioritize this sort of incident in their security program. And no, the fact that it happened doesn't change the validity of the initial risk assessment - just as the fact that the shot goes in doesn't make a guarded shot from mid-court a good shot.

We don't have time to mitigate/fix every possible risk. We HAVE to make risk-based decisions and prioritize. Moreover, often mitigating one risk will increase a different risk (such as disabling automatic updates).

There are four ways to mitigate risk:

• Mitigate/Fix it (such as patching a vulnerability)
• Accept it (essentially ignore it)
• Transfer (use contracts and legal methods to push the risk to someone else… such as insurance or a vendor)
• Avoidance (don't engage in the vendor/tool/process that is risky)

As sensational as it was, this CrowdStrike incident is just like any other that involves risk to our environment.

Don't let the sensationalism override the fundamentals.

Security News

• Cybersecurity endpoint giant CrowdStrike says almost all of the computers disrupted by a faulty software update on July 19 have now been fixed.
• CrowdStrike is alerting about an unfamiliar threat actor attempting to capitalize on the Falcon Sensor update fiasco to distribute dubious installers targeting German customers as part of a highly targeted campaign.
• An overwhelming backlog of unanalyzed vulnerabilities at the National Institute of Standards and Technology threatens to extend into 2025 unless the agency dramatically accelerates its processing operations, a new analysis reveals.
• U.S. hospital chain Ascension has filed its first breach report to federal regulators on its May 8 ransomware attack, which involved the theft of data from seven servers and disrupted patient care services at facilities across several states for weeks.
• Apple is the latest tech giant to sign onto a list of voluntary commitments for artificial intelligence development pushed by the Biden administration. Fifteen technology heavyweights have already pledged they will follow the guidance.
• Attackers can bypass the Secure Boot process on millions of Intel and ARM microprocessor-based computing systems from multiple vendors, because they all share a previously leaked cryptographic key used in the device startup process.
• A security firm recently hired a software engineer for its internal AI team that turned out to be a North Korean threat actor, who immediately began loading malware to his company-issued workstation.
• Researchers have come across a rather odd Python code package online that aims to steal Google Cloud Platform credentials from a very limited set of macOS victims.
• The remote access trojan known as Gh0st RAT has been observed being delivered by an "evasive dropper" called Gh0stGambit as part of a drive-by download scheme targeting Chinese-speaking Windows users.
• A threat actor known as Stargazer Goblin has set up a network of inauthentic GitHub accounts to fuel a Distribution-as-a-Service (DaaS) that propagates a variety of information-stealing malware and netting them $100,000 in illicit profits over the past year.

Until next time,

The Craft Compliance Team