The World is Shaking! CrowdStrike Update Pushing Windows Machines Into a Blue Screen Of Death (BSOD) Loop - "Massive Outage in IT History"
A recent update to the CrowdStrike Falcon sensor is causing major issues for Windows users worldwide. This update leads to blue screen of death (BSOD) loops and makes systems inoperable.
The issue, which began on July 19, 2024, affects Windows 10 and 11 systems running CrowdStrike’s endpoint security software.
Users report experiencing repeated BSODs with the error message “DRIVER_OVERRAN_STACK_BUFFER,” which prevents normal system boot and operation.
CrowdStrike has acknowledged the problem , stating they are “aware of reports of crashes on Windows hosts related to the Falcon Sensor” and that their engineering teams are working to resolve the issue.
The company advises affected users not to open individual support tickets now. This update’s impact has been particularly severe for enterprise customers, with some organizations reporting that thousands of devices, including critical production servers and SQL nodes, have been affected.
IT departments are scrambling to mitigate the damage, with some resorting to removing CrowdStrike-related files from affected systems to restore functionality.
This incident highlights the potential risks associated with automatic updates for security software, especially in enterprise environments.
Many affected users are now calling for more rigorous testing procedures and the implementation of staged rollout policies to prevent similar incidents in the future.
"The US Aviation Authority has required all flights to land due to a technical computer glitch."
Major services like banks, media, Airlines, Microsoft services & stock exchanges were affected.
As the situation develops, CrowdStrike is expected to provide further updates and a permanent fix for the issue.
In the meantime, affected users are advised to monitor official CrowdStrike communication channels for guidance on recovery procedures and temporary workarounds.
Microsoft has confirmed that it is investigating an “issue” affecting its 365 apps and operating systems, cautioning users to anticipate “service degradation.
“U.S. cybersecurity firm CrowdStrike has acknowledged responsibility for the error, stating they are “working on it.” Experts suggest that a “buggy” security update may have triggered the problem, though they caution that it is too early to “rule out” the possibility of a cyberattack.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” says CrowdStrike CEO George Kurtz in a post on X .
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack."
"The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website."
"We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers."
Below is a detailed table listing the affected countries and services as of July 19, 2024.
"Outage affecting 911 lines in multiple states. If you have an emergency, call the 10-digit number for your local police or fire department."
Here's the impact of IT disruptions listed as bullet points for each country:
- Australia: Media, airlines, supermarkets, banks, and hospitals impacted.
- Belgium: Train ticket sales, digital announcements, media, banks, airports, and government services disrupted.
- Canada: TD Canada Trust app and Vancouver International Airport affected.
- China: Widespread blue screens, some businesses allowed early dismissal.
- Croatia: Health information system and air traffic control issues.
领英推荐
- Czech Republic: Prague Airport affected.
- France: TV channels and Paris Olympics systems disrupted.
- Germany: Berlin Airport halted flights, Lufthansa affected, hospital operations canceled.
- Hungary: Budapest Airport issues.
- Hong Kong: Airport check-in delays, airline booking systems down.
- India: Major airlines and IT firms impacted.
- Israel: Emergency services, hospitals, and banks affected.
- Japan: Spring Japan airline experiencing issues.
- Malaysia: KTMB railway ticketing system issues.
- Netherlands: Schiphol airport, banks, and medical services disrupted.
- New Zealand: Banks, supermarkets, Auckland Transport, and Christchurch Airport were affected.
- Philippines: Major banks, telecommunications, airlines, and government websites down.
- South Africa: Banking issues.
- South Korea: Jeju Air is experiencing issues.
- Singapore: Changi Airport delays, various service disruptions.
- Spain: National airport traffic control IT outage.
- Switzerland: Zurich Airport halted landings.
- United Kingdom: News channels, airports, rail companies, NHS, and various services disrupted.
- United States: Airline ground stops, 911 service disruptions, Microsoft and CrowdStrike shares dropped.
An update from Crowdstrike is below;
Identifying Affected Systems
For those concerned about whether their systems are affected, there are several ways to check:
While CrowdStrike works on a permanent fix, some users have reported success with the following workaround:
It’s important to note that this workaround has not been officially verified. Users should proceed with caution and await official guidance from CrowdStrike.
As the situation develops, CrowdStrike is expected to provide further updates and a permanent fix. In the meantime, affected users are advised to monitor official CrowdStrike communication channels for guidance on recovery procedures and temporary workarounds.
This incident serves as a stark reminder of the delicate balance between security and system stability, especially in the context of widely deployed enterprise software. It underscores the need for robust testing procedures and carefully managed update processes in the cybersecurity industry.
As organizations worldwide grapple with the fallout from this update, the incident is likely to spark discussions about best practices for software updates in critical systems and the need for failsafe mechanisms to prevent such widespread disruptions in the future.
Senior Infrastructure Engineer - MMC Network - JPMC
3 个月About that
Infrastructure Engineer
3 个月This outage should not happened if the “vendor” test their software update first before releasing. And from company side done test or pilot of the software updates first before mass deployment. We should hate when P1 major incident need to raise for something preventable. Doesn’t look good at all.
Senior Systems Engineer/ Telephony Administrator . Over 39 years of IT experience.
4 个月Hmm. So codes were written same day and pushed to the world? Was there any testing and more testing in a control lab environment? ??
MASTER TECHNICAL WRITER / EXPERT CYBER SECURITY TRAINER Diversity-Inclusive Advocate | Trusted Team Builder | NIST-ISO Expert | Cyber Security Compliance Auditor | CompTIA Security+ | GISP | FedRAMP | HITRUST | M-21-31 |
4 个月This is why a micro-segmented network along with a patch and software deployment plan that supports this, so unforeseen and unanticipated impacts can be quickly managed. The implementation plan would roll out the change in phases on a few network segments at a time, monitor for any impacts, and etc.
MASTER TECHNICAL WRITER / EXPERT CYBER SECURITY TRAINER Diversity-Inclusive Advocate | Trusted Team Builder | NIST-ISO Expert | Cyber Security Compliance Auditor | CompTIA Security+ | GISP | FedRAMP | HITRUST | M-21-31 |
4 个月The cost of an outage can be measured against the cost of the implemented change. The total cost of this incident will have impacts even several days and weeks afterwards that will need to be factored in that outage cost.