Did a Single Bad Code Update Just Break a Billion Windows PCs?

If you woke up today, you should probably just go back to sleep because your Windows work computer will likely have a blue screen of death today. Millions, if not billions, of Windows computers got instab-bricked around the world thanks to an update pushed by enterprise cybersecurity firm CrowdStrike , and it's bad. Airports are shutting down, hospitals are unable to treat patients, banks can't access your money, and even the Arby's drive-through window went down, forcing people to hunt stray cats in the street just for food.

What's hilarious, though, is that a top cybersecurity firm just messed up the global economy in a way that the evil hackers they protect you from could only dream of. In today's video, we'll take a look at the technical side of this disaster and find out how such a catastrophic mistake can even happen in the modern world. It is July 20th, 2024, and you're reading The Code Report.

Corporate America is in panic mode right now because everybody's work computers are bricked, and that means the hamsters can't keep spinning the wheels. A huge number of Fortune 500 companies use CrowdStrike for cybersecurity. It's got over 500 clients on the Fortune 1000 list. Its primary product is called Falcon, a tool that provides endpoint protection using artificial intelligence and analytics to detect threats in real-time. It's publicly traded, and its stock is down right now for good reason because everybody's blaming them for causing Windows to deliver its blue screen of death. Luckily, macOS users and Linux chads are unaffected.

To understand why, we first need to understand how CrowdStrike's Falcon sensor actually works. It's installed just like regular software but integrates with the operating system at a low level, often using kernel-mode drivers, and basically just sits there in the background looking for anomalies. It collects telemetry data, produces reports, and offers a bunch of other incomprehensible tech nonsense products to justify multi-million dollar enterprise contracts. But the bottom line is that it's third-party software that sits in the critical path of a computer, which means if it fails, the entire computer might fail, and that's exactly what happened here.

Apparently, an automated software update last night had some bad code in it, and every computer that got that update is now dead. Now, part of the reason this is really bad—oh my God, is this bad—is that it's not just a regular outage, but every affected computer needs to be rebooted in fail mode so the driver can be removed manually, and most employees don't have access to do that on their own. That means IT guys are going to be really busy today. It's the IT guy equivalent of being a surgeon in World War I.

The consequences are real. The London Stock Exchange was disrupted, most Indian airports went down, causing them to write boarding passes by hand, along with a ton of other issues. To CrowdStrike's credit, they were quick to point out that it's not a security incident or cyberattack and explained it this way: "Yeah, listen, uh, we messed up." But they were quick to fix it, and the fix is really easy. All you have to do is detach the operating system disk, create a snapshot or backup of the disk, mount a volume to a new virtual server, navigate to the Windows drivers directory, locate the file c291.sys, and delete it. Then detach the volume from the new virtual server and reattach the fixed volume to the impacted virtual server. Piece of cake.

Option two is to go to Home Depot, buy a sledgehammer, and use it to uninstall Microsoft Windows and switch to Linux (I use arch btw). I do feel really bad for the programmer who updated this driver, though, because the tech lead's about to run git blame and blame them for this whole mess. Not only is this person about to get fired, but they also have blood on their hands for shutting down hospitals, transportation networks, and Arby's family restaurants that we need to survive. If you're that guy and you're watching this, don't feel too bad. What we have here is a situation where the cure is more harmful than the disease. Public mega-corporations are under a ton of pressure to secure their computer systems, and they're constantly audited by third parties. A company like Macy's is going to go out and hire a team of 100 cybersecurity weirdos. Instead, they'll pay a company like CrowdStrike a few million dollars a year to figure out cybersecurity for them, giving them someone else to blame when their system gets hacked.

What everyone failed to realize, though, is that giving one company kernel access to the computers of most Fortune 500 companies might actually be a bad idea because it only takes one automatic update with a misplaced zero to nearly destroy the entire world. This has been The Code Report. Thanks for reading, and I will see you in the next one.