Global IT Outage: Microsoft and CrowdStrike's Update Issues

Mustafa Saidalavi, CEO EBITA Ai

What Happened? On 19 July 2024, a CrowdStrike Falcon update caused widespread Blue Screen of Death (BSOD) issues on Windows hosts, leading to severe disruptions in various sectors including travel, banking, and healthcare.

Why Did It Happen? The outage was triggered by a flawed update involving the file “C-00000291*.sys” in CrowdStrike's antivirus software, which led to system crashes.

What Is the Global Impact Now?

• Travel: Major airports experienced long queues, delays, and flight cancellations as systems went offline and manual check-ins were required.
• Banking: Financial transactions faced significant delays due to system outages.
• Healthcare: Medical services were disrupted, affecting patient care and hospital operations.

How Did It Happen?

• The defective update from CrowdStrike led to BSOD issues, primarily impacting Windows PCs.
• This caused system crashes on boot, complicating network-based remediation efforts.
• The issue arose from a problematic file in the update, which when deleted, resolved the BSOD problem.

Other Similar Incidents:

• Cloudflare Outage: On 4 July 2024, a bad software deployment by Cloudflare caused a global outage, affecting websites and internet applications. The issue was quickly identified and resolved by rolling back the offending ruleset and making changes to the deployment process to prevent future incidents.
• Microsoft Azure Outage: A faulty systemd update on Microsoft Azure caused DNS errors, taking Ubuntu 18.04 VMs offline. This impacted various Azure services, including Azure Kubernetes Service and Azure Monitor.
• McDonald's Outage: A configuration change by a third-party service provider caused a global outage affecting McDonald's restaurants. This resulted in POS systems going down, forcing manual order taking and payment processing.

How to Resolve?

1. Immediate Update: Deploy the reverted content update from CrowdStrike.
2. Manual BSOD Fix:Boot into Safe Mode.Delete the file “C-00000291*.sys” from “C:\Windows\System32\drivers\CrowdStrike”.
3. Cloud Systems and Virtual Servers: Follow specific steps to detach, fix, and reattach affected volumes.
4. AWS EC2 Instances: Relaunch from a pre-issue snapshot or follow detailed steps to remove the file.

How to Prevent?

1. Enhanced Update Testing:Rigorous pre-deployment testing and staged rollouts to monitor for issues.
2. Improved Monitoring and Incident Response:Utilize real-time monitoring systems and dedicated response teams.
3. Compatibility Checks:Ensure updates are compatible with all operating systems and configurations.
4. Regular Security Audits and Penetration Testing:Conduct audits and testing to identify vulnerabilities.
5. User Training:Educate employees on update procedures and issue handling.
6. Collaboration with Cybersecurity Firms:Engage in partnerships for shared intelligence and best practices.
7. Robust Backup and Recovery Solutions:Maintain regular backups and updated disaster recovery plans.

By adopting these measures, organizations can strengthen their defenses against similar incidents in the future, ensuring smoother operations and enhanced cybersecurity resilience. For further details, consult the full advisory document from KPMG on CrowdStrike's update issue.

#GlobalITOutage #MicrosoftBSOD #CrowdStrikeIssue #TechFail #ITChaos #SystemCrash #BlueScreenOfDeath #BSOD #Cybersecurity #TechUpdate #ITDisaster #SystemFailure #WindowsCrash #TechSupport #TechNews #CyberAttack #DataSecurity #ITManagement #TechResolution #DisasterRecovery #ITOutage #SystemUpdate #TechError #ITInfrastructure #SystemMaintenance #TechTroubles #ITIncident #CyberIncident #GlobalOutage #TechCrisis #SecurityBreach