Simplified Insights on the Recent CrowdStrike Incident Root Cause Analysis

Recently, CrowdStrike encountered an issue with its Falcon sensor software that led to system crashes on some Windows computers. While the official incident report dives deep into technical details, I wanted to share a streamlined version that’s easier to understand, especially for those who aren’t deeply involved in the technical aspects.

What Happened?

CrowdStrike's Falcon sensor is a powerful tool that uses advanced technology, including AI, to detect and prevent cyber threats in real-time. This software is continually updated to ensure it can protect against the latest threats.

In February 2024, CrowdStrike introduced a new feature designed to detect certain sophisticated hacking techniques, specifically those that manipulate a part of the Windows operating system known as "named pipes." However, a coding error meant the software was supposed to check 21 different pieces of information, but it only had 20. This mismatch went unnoticed during testing.

When the software attempted to use the missing piece of data, it caused the system to look for information that wasn’t there, leading to crashes on some computers.

Key Findings and Fixes

1. Data Field Mismatch: The software was designed to process 21 fields but was only given 20. This issue wasn’t caught during the initial development phase. Solution: CrowdStrike released a patch to ensure the software checks for the correct number of fields moving forward. (Most machines were not able to receive this update as they had already been impacted by the "Blue Screen of Death" and were not functional.)
2. Missing Safety Checks: The software lacked necessary safeguards to prevent it from accessing non-existent data, leading to crashes. Solution: These safety checks have now been added to the software.
3. Inadequate Testing: The testing process didn’t cover all possible scenarios, allowing this issue to slip through. Solution: CrowdStrike has expanded its testing procedures to include a broader range of scenarios.
4. Validation Errors: The software that checks for errors in updates before they’re released didn’t catch this issue. Solution: Additional validation checks have been implemented to prevent this from happening again.
5. Need for More Comprehensive Testing: The initial tests weren’t thorough enough to identify the problem. Solution: CrowdStrike has enhanced its testing procedures to ensure all potential issues are caught before updates are released.
6. Controlled Rollout of Updates: The problematic update was rolled out to all systems at once, which exacerbated the impact. Solution: Future updates will be released in stages, allowing issues to be identified and resolved before they affect everyone. (This is often referred to as a Canary Deployment, and it is worth noting that this is an industry best practice, and very common for software companies to do. It is unknown by CrowdStrike chose not to employ this practice.)

CrowdStrike’s Proactive Measures

In addition to addressing the specific issue, CrowdStrike has engaged independent experts to review their software and processes from development to deployment. This external review will help ensure that their security products continue to meet the highest standards and that any vulnerabilities are swiftly addressed. (External reviews and validations are a common practice in the software world. Had CrowdStrike performed this level of external review / validation before or during the launch of this new feature, it's very likely that the issue would have been proactively caught and addressed. It is unknown why CrowdStrike chose not to utilize third-party review or validation before a major feature release.)

In Summary

While the technical details behind this incident are complex, the core issue was a coding error that led to system crashes. For all of the faults and blatant mistakes that CrowdStrike made that led them to this point, the company has been transparent in their response, taking immediate steps to fix the problem and improve their processes.

For those of us in the cybersecurity space, this incident serves as a reminder of the importance of rigorous testing and validation. Even the most advanced systems can have vulnerabilities, and it's important to be consistent with employing industry best practices, even if it means delaying feature releases.