A Guide to the Windows BSOD Crisis, Following CrowdStrike's Update

On Friday, July 19, 2024, the world experienced a significant outage across various industry sectors due to what initially appeared to be a minor software update to the CrowdStrike Falcon product on Microsoft system endpoints. This update introduced a corrupt system file, which led to a "blue screen of death" when booted by a Microsoft operating system, rendering hundreds of thousands of endpoints unusable and in need of direct human intervention for recovery.

As a partner to CrowdStrike, we share concerns for both our customers and our partner regarding this issue. We want to provide a guide to help mitigate and recover impacted devices in this situation.

Summary

• CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Full Statement is covered here.

Details

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
• This issue does not impact Mac- or Linux-based hosts.
• Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Current Action

• CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
• If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

1. Workaround Steps for individual hosts:

Reboot the host to give it an opportunity to download the reverted channel file.? If the host crashes again, then:?

• Boot Windows into Safe Mode or the Windows Recovery Environment
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
• Locate the file matching “C-00000291*.sys”, and delete it
• Boot the host normally

Note:? Bitlocker-encrypted hosts may require a recovery key.

2. Workaround Steps for public cloud or similar environment:

• Detach the operating system disk volume from the impacted virtual server
• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
• Attach/mount the volume to to a new virtual server
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
• Locate the file matching “C-00000291*.sys”, and delete it.?
• Detach the volume from the new virtual server
• Reattach the fixed volume to the impacted virtual server