Blue screen of death

Crowd strike release update cause Outrages to Microsoft Windows

Summary:

Crowd Strike is a cybersecurity technology company that provides endpoint security solutions to protect computers against cyber incidents.

On Friday (19/07/2024), Crowd Strike released a software update to its solution that protects against hackers and online intruders. However, machines running Microsoft Windows started crashing when the software update began.

When a single flawed piece of software is released on the internet, it can almost instantly harm countless companies and organizations that rely on those software providers to conduct everyday business.

This outage has affected airports, trains, businesses, broadcasters, and critical infrastructure.

Bit more technical:

Crowd strike provide next-generation end point protection solution named “Cobalt Strike Falcon”.

Users need to install Falcon sensor in their system. Once installed there is no need to reinstall it as long as account is active. Also note that this software has privilege access.

Due to the flawed software update of Falcon sensor windows hosts start crashing and showing blue screen, called as “Blue screen of death” (BSOD).

How to Identify faulty machines:

Run the below query:

C-00000291* |in(field="#event_simpleName", values=[AgentOnLine, LFODownloadConfirmation]) | groupBy([aid,ComputerName], function=[max(@timestamp, as=lastSeen),max(@timestamp, as=lastSeenForCalculation) ,collect([FileName])], limit=max) | lastSeen:=formatTime(field=lastSeen, format="%Y/%m/%d %H:%M:%S") // 4:09 to 5:27 | lastSeenForCalculation >= 1721362140000 AND lastSeenForCalculation <= 1721366820000        

How to fix it :

Windows Host Machine:

Steps:

·?????? Boot Windows into Safe Mode or the Windows Recovery Environment

·?????? Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

·?????? Locate the file matching “C-00000291*.sys”, and delete it.

·?????? Boot the host normally.

?

For Bitlocker based machines:

·?????? Use the advanced restart options to launch the command prompt

·?????? skip the bitlocker key ask

·?????? which then brought us to drive X Run "bcdedit /set {default} safeboot minimal"

·?????? which let us boot into safemode and

·?????? delete the sys file causing the bsod

?

For Cloud based systems:

1.????? For Hyper V VMs:

Steps:

·?????? Create a Windows VM.

·?????? Detach the OS disk from the affected VM

·?????? Attach this disk as a data disk to new VM

·?????? Access the below path from the new VM on the attached disk of the affected VM

o?? a. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

o?? b. Locate the file matching “C-00000291*.sys” and delete it

·?????? Detach the disk from new VM & attach to existing VM

·?????? Boot the machine normally

?2.????? For Azure:

Steps:

·?????? Detach the OS disk from existing VM.

·?????? Create a new VM which is bootable.

·?????? Attach the OS disk to the new VM.

·?????? Access the below path from the new VM on the attached disk of the affected VM

o?? a. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

o?? b. Locate the file matching “C-00000291*.sys” and delete it.

·?????? Detach the disk from new VM & attach to existing VM

·?????? Boot the machine normally.

?

3.????? AWS:

Steps:

·?????? Detach the EBS volume from the impacted EC2

·?????? Attach the EBS volume to a new EC2

·?????? Fix the Crowdstrike driver folder

·?????? Detach the EBS volume from the new EC2 instance

·?????? Attach the EBS volume to the impacted EC2 instance

?

How to avoid and catch issues before it causes such disruption:

Security Audit: Regularly review the system and privilege application. Follow least privilege rule.

Employee Training: Ensure that all employees, especially those with privileged access, are trained in security best practices and are aware of the potential risks and how to avoid them.

Monitor and Log: Continuously monitor and log activities related to your software updates to detect any unusual or suspicious activities.

First test into lower environment: For business-critical applications, test the update in a pre-production environment before deploying it to production.

Incident Response Plan: Develop and maintain a robust incident response plan that includes steps to quickly mitigate any outrage or DoS attacks and restore normal service.

Vulnerability Scanning: Use automated vulnerability scanning tools to continuously check for security weaknesses in your software.

?