The CrowdStrike outage: Just another day at the office?
It’s every CEO’s nightmare scenario: Millions of customers locked out of their workstations. Billions of dollars in economic losses. Share prices plummeting. Government demanding answers.
We're talking of course about the July 19 2024 CrowdStrike incident, in which a faulty update to the company’s Falcon security sensor sent 8.5 million Windows endpoints around the world into a BSOD crash loop, disrupting critical infrastructure and requiring manual fixes on affected machines to restore operations.?
If you follow tech news, you likely know the basics. Perhaps you even experienced the fallout firsthand, facing travel disruptions, missed deliveries, or healthcare and banking delays. But beyond these widely reported disruptions, there’s another side of the story that has yet to be explored: how this massive outage affected software delivery patterns around the world.
Given the scale of the outage, you might expect to see development grinding to a halt across affected organizations. After all, how do teams maintain productivity when their workstations are crashing, their deployment targets are down, and their databases are inaccessible? But the data tells a different story, one with important insights about the resilience of modern DevOps practices.
In this newsletter, we’ll explore CI/CD pipeline data from across industries to uncover the challenges software teams faced and the strategies they used to keep their workflows on track in the midst of the largest IT outage in history.
TL;DR
How the CrowdStrike incident unfolded
Like many software incidents, the trouble began with a routine update. In February 2024, CrowdStrike launched a new version of its Falcon security driver to better detect attacks exploiting Windows named pipes. This update, along with several follow-up improvements in March and April, operated smoothly in production environments.
The critical failure occurred on July 19, 2024, at 04:09 UTC. CrowdStrike deployed channel file 291 globally, containing what appeared to be a minor update to threat definitions. Within minutes, Windows systems worldwide began to experience severe failures, affecting an estimated 8.5 million endpoints across industries.
By 05:27 UTC, less than 90 minutes after the initial deployment, CrowdStrike had identified the issue and deployed a fix. However, affected systems remained stuck in a crash loop that made remote remediation impossible. Instead, IT teams had to manually access each affected machine, boot into safe mode, and remove the corrupted file.
The scale of the incident prompted a coordinated response. The US government’s Cybersecurity and Infrastructure Security Agency (CISA)? issued a national alert by 15:30 UTC, working directly with CrowdStrike to support critical infrastructure recovery. By evening, CrowdStrike's CEO had publicly addressed the situation, emphasizing that the failure stemmed from an internal error rather than a cyberattack.
Recovery efforts continued through the following days. Microsoft released specialized recovery tools for affected systems and CrowdStrike published comprehensive remediation instructions. By July 25, CrowdStrike reported that virtually all affected systems had been restored to normal operation.
Technical deep dive: Understanding the CrowdStrike failure
According to CrowdStrike’s public incident report, the root cause of the incident lay in the interaction between CrowdStrike's Falcon sensor and its dynamic configuration system. The Falcon sensor operates through a combination of core software (the sensor itself) and dynamic configuration files called channel files. These files contain important information about threat definitions and detection rules, allowing CrowdStrike to respond to new security threats without requiring full updates to the Falcon sensor software.
Channel file 291's fatal flaw was surprisingly simple: a mismatch between expected and provided input fields. The file's template expected 21 distinct input values, but the update only contained 20. When the sensor attempted to read the missing 21st value, it triggered an out-of-bounds memory access, causing a system crash.
What made this error particularly devastating was the Falcon sensor's privileged position within the Windows operating system. To achieve the system access necessary for comprehensive security monitoring, the Falcon sensor operates on the kernel level rather than the user level, where most applications run. When a user-level driver or application experiences an error, it becomes unresponsive or closes. When a kernel-level driver crashes, it brings down the entire operating system with it.
The crash loop became self-perpetuating. Because the Falcon driver loads automatically during startup, affected systems would crash before reaching a state where remote fixes could be applied. This forced organizations to physically access each affected machine, boot into safe mode, and manually remove the corrupted channel file.
Why didn’t CrowdStrike catch the error?
CrowdStrike’s testing process failed to catch the misconfigured definition file because of a subtle flaw in how they tested definition files. While they validated the parser and the definition files separately, they did not validate the specific combination of inputs under production-like conditions.
Specifically, the discrepancy between expected and provided inputs didn’t trigger a fault in testing because the test cases used regex wildcard matching in the 21st field. This allowed the 21st field to match any input (including a null or missing input). In production, the Windows boot process required the exact input structure to function correctly.
Why didn’t Windows software policies block the faulty update?
In Windows, drivers and other critical components that interact closely with the operating system must be signed to prevent unauthorized or malicious code from running. However, code signing validates only that a file hasn't been tampered with after it was signed—it does not check the actual contents of the file, meaning it would not flag the configuration error that caused the incident.
Windows Hardware Quality Labs (WHQL) certification adds an additional layer of assurance for kernel-mode drivers. WHQL testing ensures drivers are evaluated for compatibility and reliability with Windows. Kernel drivers like the Falcon sensor go through this stringent process to prevent serious security and stability flaws from being introduced to Windows hosts.
Yet in the case of the CrowdStrike outage, Microsoft’s WHQL requirements applied only to the Falcon driver itself, not to the configuration files that control how the driver operates. While the core Falcon driver passed WHQL tests, its configuration files were not subjected to the same level of scrutiny. This allowed the faulty update in channel file 291 to slip through Windows’ safety net.
What about IT phased rollout policies?
Typically, IT departments use phased rollouts and delay policies (such as “N-1” or “N-2”) to confirm stability before full deployment. However, CrowdStrike’s urgent approach to critical updates bypassed these safeguards, pushing the update to all hosts at once. This decision, combined with kernel-level access, insufficient production-like testing, and an unvalidated configuration file, created a perfect storm that amplified the outage’s impact across every endpoint running Falcon.
How did the outage affect software delivery patterns?
Given the widespread infrastructure disruptions—from grounded flights to healthcare delays—you might expect to see similar chaos in software delivery metrics. After all, development and deployment processes rely heavily on the same computing environments that were crashing worldwide.
This expectation makes our findings particularly interesting. By examining CI/CD data during the incident, we can understand how organizations responded in real-time, adjusted their priorities, and leveraged their DevOps practices to handle the fallout. Were certain industries more resilient? Did companies pull back on feature development to focus on stability? The answers provide valuable insights about which practices best support continuity during major incidents.
Methodology
To answer these questions, we analyzed software delivery performance on our platform from July 1 to August 3, 2024. We focused on four key metrics that would indicate disruption to normal development patterns:
We segmented this data by industry and execution environment (Linux, macOS, and Windows) to identify any patterns in how different types of teams were affected.
Since the incident occurred on a Friday, and development teams are notoriously hesitant to deploy on Fridays — CrowdStrike now being a high-profile example of why — we focused our comparison on other Fridays in July and early August. This allowed us to identify deviations from typical Friday behaviors and performance levels. We excluded July 5 from the dataset, as it fell immediately after a major U.S. holiday, when many companies were either closed or had a high number of employees on vacation.
Join the Ranks of Software Champions
Looking to achieve elite delivery performance? Join the ranks of resilient, high-performing teams who rely on CircleCI to keep software delivery on track, even during industry-wide disruptions. Visit CircleCI.com to learn more and see how your team can lead with confidence.
Results
What does pipeline data tell us about the impact of the CrowdStrike outage? To understand the extent of the disruption, we first looked at performance across all teams building on CircleCI. This broad view helped us establish whether there were any significant disruptions to normal development patterns before diving into specific sectors and workflows where we expected to see the greatest impact.?
The following chart shows performance across metrics as a percentage of their expected value. We dig into the data more in the sections below.
Global throughput impact: Did activity levels deviate from the norm?
On the day of the incident, activity across all development branches held relatively steady at 97% of the Friday average. This suggests that most teams found ways to continue development work despite the infrastructure challenges.?
However, on main branches specifically, activity dropped to 93% of the average, suggesting that some teams may have paused or scaled back deployments during the outage—either because production machines were less available or due to extra caution around deploying to production during the incident. We’ll look more closely at this in subsequent sections.?
Global duration impact: Did pipelines run faster or slower than usual?
Pipeline duration on the day of the incident remained largely consistent with typical Friday activity, reaching 98% of the July average across all branches and 101% on main branches. While there may have been slight shifts in focus, this suggests that most teams continued their usual workflow patterns without significant delays.
Global success rate impact: Did pipelines fail at an unusual rate?
Of all key metrics, success rates were the least affected by the outage, coming in at 100% and 101% of expected levels for all pipelines and main branch pipelines specifically.
This stability is particularly noteworthy because outages typically lead to cascading failures, incomplete runs, or increased error rates. The normal success rates suggest either that teams quickly adapted their processes to account for the outage, or that CI/CD infrastructure remained largely isolated from the affected systems.
Global MTTR impact: Was recovery slower than usual?
We’ve seen that there was no measurable increase in the number of failed pipelines on the day of the incident. But is there any evidence that the types of failures that happened during this incident were any more complex or difficult to resolve, as measured by mean time to recovery (MTTR)?
In fact, recovery times were faster than average during the incident: teams resolved errors across all branches 16% faster than the typical Friday and 3% faster on main branches. While it’s difficult to attribute this improvement to any one factor, faster recovery times may reflect a heightened sense of urgency to fix build issues and ship updates quickly, even among teams who were not directly affected by the incident.
Industry impact: Were certain sectors more affected than others?
Next, we examined the effect of the outage on software delivery patterns in specific industries to better understand if certain sectors experienced greater disruption or were better equipped to maintain stability and efficiency under challenging conditions.
These are the industries we checked:?
First, we analyzed which industries had the most significant deviations from their expected throughput, offering insights into whether specific sectors faced blockages in their delivery pipelines or ramped up productivity in response to the outage.
Utilities, government, and airlines showed the highest increases in activity on July 19, reflecting efforts to address widespread disruptions and support critical operations. Airlines — which faced some of the most visible service interruptions — saw throughput rise to 114% above baseline across all branches on Friday, followed by a surge of activity (+129%) on main branches by Saturday. This suggests that airline software teams spent Friday developing and testing updates to support critical systems overwhelmed by unprecedented service demands, then deployed them over the weekend.
In contrast, banking and pharmaceuticals saw the most significant reductions in activity, with workflows on the main branch falling by 22% and 20%, respectively. This more conservative approach likely reflects these industries' emphasis on risk management over rapid response.
Next we looked at success rates to assess whether the outage had an impact on the reliability of software workflows across industries.?
Here again, airlines, utilities, and government fared the worst, with success rates on all branches falling between 11 and 21 percent. At the same time, these three sectors were all at or above their typical performance on the main branch.?
Combine these results with what we know about throughput in these three industries and you get an interesting story: IT teams in the most heavily affected industries increased development activity in an attempt to address operational issues or adapt to surges in demand, suffering higher rates of build failures as they experimented with updates and configuration changes. The elevated success rates on main branches indicate that only the most thoroughly validated changes made it to production, effectively shielding end users from the trial and error phase of the recovery effort.
Project impact: Did the outage hit Windows-based workflows hardest?
Finally, we looked at a breakdown of throughput by pipeline execution environments. CircleCI offers hosted compute across a number of operating systems and environments, including? Docker, Linux VM, macOS, Windows, GPU, and Arm. Given that this incident exclusively affected Windows machines, we wanted to see whether there were any noticeable disruptions for Windows workflows specifically.
On the day of the incident, throughput on the main branch of Windows projects fell by approximately 33%—the largest decline we observed in any segment of our analysis. However, feature branches maintained normal levels of activity. By the next day, the pattern had reversed dramatically: main branch workflows surged to 60% above normal levels, while feature branch activity dipped only slightly (9% below average).
The sharp drop in main branch throughput on the day of the incident likely reflects teams dealing with disruptions to Windows-based build environments or pausing deployments to troubleshoot issues in production systems.
By the following day, the surge in main branch workflows points to a concerted push to deploy critical fixes, as teams refocused on stabilizing production systems. This trend was especially notable in industries like airlines, where urgent updates were likely prioritized. Meanwhile, feature branch activity dipped slightly, implying that new development paused while immediate issues took precedence.
Summary of results
Our analysis reveals three key insights about how development teams weathered the CrowdStrike incident:
It's important to note, though, that our sample population consists entirely of teams using CI/CD practices. These teams likely had advantages—in terms of automation, process consistency, and feedback loops—that teams using manual processes did not. The impact of the outage on the software industry as a whole may have been far more severe than shown here. Our data represents the best-case scenario: teams who already had robust automation, feedback loops, and process controls in place.
What strategies could prevent similar incidents?
The CrowdStrike outage is a reminder of just how complex and risky it can be to manage large-scale updates on mission-critical systems. It also shows why it’s so important to build strong safeguards into deployment workflows.?
For teams navigating high-stakes deployments, there are several proven strategies to boost resilience and keep small problems from causing big disruptions:
To their credit, CrowdStrike conducted an extensive root cause analysis identifying many of these opportunities along with several others more targeted to their specific development and delivery processes. In response, they introduced a range of mitigation strategies, including:
In the weeks that followed, CrowdStrike unveiled a Resilient by Design framework focused on making resilience fundamental to the company’s identity, adapting quickly to customer needs, and maintaining continuous feedback loops to drive learning and improvement throughout the delivery lifecycle.
Conclusion
The July 2024 CrowdStrike incident demonstrates both the fragility and resilience of modern technical infrastructure. While the incident's impact varied significantly across sectors, global software delivery metrics remained surprisingly stable — at least among CI/CD users.
Ironically, the very practices that helped teams manage this crisis—comprehensive testing, staged rollouts, and automated checks—might have prevented it altogether if CrowdStrike had fully implemented them. This highlights an important point: CI/CD is not only about speed; it’s about ensuring stability, control, and resilience,? especially in challenging situations.?
Want to ensure your team is ready for the next industry-wide incident? Visit CircleCI.com and join thousands of high-performing software delivery teams who use CI/CD to turn potential crises into controlled responses.
Further Reading
Fascinating insights into how critical industries respond under pressure! The data highlights the importance of agility and risk management, especially in industries like banking and pharma. DevOps truly proves its value in enabling resilience during such crises.