Skip to content

Instantly share code, notes, and snippets.

@whichbuffer
Last active August 28, 2024 09:26
Show Gist options
  • Save whichbuffer/7830c73711589dcf9e7a5217797ca617 to your computer and use it in GitHub Desktop.
Save whichbuffer/7830c73711589dcf9e7a5217797ca617 to your computer and use it in GitHub Desktop.
Automated CrowdStrike BSOD Workaround in Safe Mode using Group Policy

Automated Workaround in Safe Mode using Group Policy

You can set up a GPO to run a script during Safe Mode. Here’s how you can do this:

  1. Create the PowerShell Script

    Create a PowerShell script that deletes the problematic CrowdStrike driver file causing BSODs and handles the Safe Mode boot and revert:

    # CrowdStrikeFix.ps1
    # This script deletes the problematic CrowdStrike driver file causing BSODs and reverts Safe Mode
    
    $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"
    $files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue
    
    foreach ($file in $files) {
        try {
            Remove-Item -Path $file.FullName -Force
            Write-Output "Deleted: $($file.FullName)"
        } catch {
            Write-Output "Failed to delete: $($file.FullName)"
        }
    }
    
    # Revert Safe Mode Boot after Fix
    bcdedit /deletevalue {current} safeboot
    
  2. Create a GPO for Safe Mode

    • Open the Group Policy Management Console (GPMC).
    • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here....
    • Name the GPO, for example, CrowdStrike Fix Safe Mode.
  3. Edit the GPO

    • Right-click the new GPO and select Edit.
    • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
    • Double-click Startup, then click Add.
    • In the Script Name field, browse to the location where you saved CrowdStrikeFix.ps1 and select it.
    • Click OK to close all dialog boxes.
  4. Force Safe Mode Boot Using a Script

    Create another PowerShell script to force Safe Mode boot and link it to a GPO for immediate application:

    # ForceSafeMode.ps1
    # This script forces the computer to boot into Safe Mode
    
    bcdedit /set {current} safeboot minimal
    Restart-Computer
    
  5. Create a GPO to Apply the Safe Mode Script

    • Open the Group Policy Management Console (GPMC).
    • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here....
    • Name the GPO, for example, Force Safe Mode.
    • Right-click the new GPO and select Edit.
    • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
    • Double-click Startup, then click Add.
    • In the Script Name field, browse to the location where you saved ForceSafeMode.ps1 and select it.
    • Click OK to close all dialog boxes.
  6. Apply the GPOs

    • Make sure the Force Safe Mode GPO is applied to the affected computers first.
    • The computer will boot into Safe Mode and execute the CrowdStrikeFix.ps1 script.
    • Once the issue is fixed, the script will revert the boot settings to normal mode.
@pablodz
Copy link

pablodz commented Jul 19, 2024

Huge day for Linux

@ryox82
Copy link

ryox82 commented Jul 19, 2024

LiNuX wInS. That isn't how real life works.

@ephraimduncan
Copy link

linux <3

@ntwrite
Copy link

ntwrite commented Jul 19, 2024

Updating based on hashes from good Crowdstrike patch for accuracy on detection of bad driver

# CrowdStrikeFix.ps1
# This script checks for a specific driver file, deletes it if it meets criteria, and reverts Safe Mode

$filePath = "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"
$goodHash = "7224125E3B3899DDE946D9C9872FD07201B83389"

if (Test-Path $filePath) {
    $file = Get-Item $filePath
    $sha1Hash = (Get-FileHash $file.FullName -Algorithm SHA1).Hash

    if ($sha1Hash -ne $goodHash) {
        Write-Output "[MATCH] BAD. C-00000291*.sys found. Hash does not match known good file."
        Write-Output "SHA1: $sha1Hash"
        exit 1
    } else {
        Write-Output "[NONMATCH] GOOD. C-00000291*.sys found. Hash matches known good file."
        Write-Output "SHA1: $sha1Hash"
        exit 0
    }
} else {
    Write-Output "[NONMATCH] C-00000291*.sys not found."
    exit 0
}

# Revert Safe Mode Boot after Fix
bcdedit /deletevalue {current} safeboot

@cloudynetwork
Copy link

cloudynetwork commented Jul 19, 2024

Great work. For those who think this couldn't have happened to Linux endpoints if they had something similar pushed to them...you're sense of wellbeing is misplaced 😟

@jeroenh
Copy link

jeroenh commented Jul 19, 2024

I'm just gonna leave this one here:
do not tell me you don't have this problem

@zakig7
Copy link

zakig7 commented Jul 19, 2024

Does this work on machines that are in the blue screen loop or is this just corrective for machines that have the errored file but are booting fine?

Crowdstrike already released a new channel file version to overwrite the one that borked everything. This solution would have worked if:

  1. The machines booted beyond BSOD long enough for a GPO or MDM script to run.
  2. Crowdstrike didn't release a fix already, which made this solution redundant. If the machine boots properly, it will get the new sensor fix they released.

@tcxseo
Copy link

tcxseo commented Jul 19, 2024

CrowdStrike’s ‘BSOD’ Outage: ‘No evidence that this is a Cybersecurity Incident,’ Says Cybersecurity Minister

This is a technical issue, caused by a Crowdstrike update to its customers. The company has informed us that most issues should be resolved through the fix they have provided, but given the size and nature of this incident it may take some time to resolve," Clare O'Neil, Minister of Cyber Security said.

Read More: (https://thecyberexpress.com/blue-screen-of-death-not-a-cyber-incident/)

@miqueet
Copy link

miqueet commented Jul 19, 2024

# CrowdStrikeFix.ps1
# This script deletes the problematic CrowdStrike driver file causing BSODs and reverts Safe Mode

$filePath = "C:\Windows\System32\drivers\Crowdstrike\C-00000291*.sys"
$files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue

foreach ($file in $files) {
    try {
        Remove-Item -Path $file.FullName -Force
        Write-Output "Deleted: $($file.FullName)"
    } catch {
        Write-Output "Failed to delete: $($file.FullName)"
    }
}

# Revert Safe Mode Boot after Fix
bcdedit /deletevalue {current} safeboot

I believe this is the proper file path. This what i have on my systems.

@erkexzcx
Copy link

lmao

@winguy253
Copy link

Hi, Does the GPO still apply on a machine that is in the BSOD state? How exactly does this work?

@miqueet
Copy link

miqueet commented Jul 19, 2024 via email

@winguy253
Copy link

Since this GPO applies as a startup script. If you can’t startup it will never apply.

-Mike Wheat
On Fri, Jul 19, 2024 at 9:12 AM winguy253 @.> wrote: @.* commented on this gist. ------------------------------ Hi, Does the GPO still apply on a machine that is in the BSOD state? How exactly does this work? — Reply to this email directly, view it on GitHub https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217797ca617#gistcomment-5126742 or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIZH752AG7SWAGULIQQW4DZNEGE3BFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTGMJUG42TGMJSU52HE2LHM5SXFJTDOJSWC5DF . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

Ok. I was wondering how that would work since we need to acquire a bitlockr key first to boot into safemode.

Anyone else come up with any way to remotely execute a fix with a machine in a BSOD state?

@Brown-Dog-Soup
Copy link

You would need to boot into safe mode with networking. Has anyone used this and had success?

@zakig7
Copy link

zakig7 commented Jul 19, 2024

You would need to boot into safe mode with networking. Has anyone used this and had success?

Yes the solution works if you boot in safe mode and either connect network, or manually delete the channel file, but this is fairly manual and does not scale.

@winguy253
Copy link

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?

I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

@winguy253
Copy link

Can someone explain how a GPO startup script works if the device can't boot into Windows?

@iz0t0pe101
Copy link

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?

I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

@winguy253
Copy link

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?
I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

So there's no way to securely unlock a device automatically with a script? So this outage is only manually fixable if you have MDE implemented?

@iz0t0pe101
Copy link

Can someone explain how a GPO startup script works if the device can't boot into Windows?

good point, this method would work if the device stays online enough time for the GPO to be deployed, but the devices are throwing BSOD couple of seconds after they reached the windows logon screen. this loop is made several time until the device stops booting and you need to take manual steps to boot again.

@HotCakeX
Copy link

This is a problem caused by CrowdStrike 3rd party software, has nothing to do with Windows.

@viktorurukhai
Copy link

How this solution is suposed to solve the issue if the computers are not booting because of the BSOD? They don't have a network connection to reach de domain and apply the GPO...

The only way the GPO can be applied is entering into safe mode with networking computer by computer.

@iz0t0pe101
Copy link

iz0t0pe101 commented Jul 19, 2024

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?
I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

So there's no way to securely unlock a device automatically with a script? So this outage is only manually fixable if you have MDE implemented?

i usually work with Intune managed devices, and from there i can extract for each device all the bitlocker recovery keys for each drive that they have, using Graph API calls and some powershell scripting... but this would require an EntraID to be created with an appId and appSecret/certificate, you have to give specific ReadAll permissions to the app to a specific endpoint from Graph API, then using powershell you need to iterate through all device Ids and use them in the API calls to extract the bitlocker recovery keys from another API endpoint, then use those keys to somehow unlock the devices, you need to match the device with it's own recovery key for the specific drive on which windows is installed and do some scripting magic to decrypt the device, remove the dreaded sys file and then reboot it.
i would not do it like this, as i don't want to risk having client devices unencrypted ;)

on SCCM you also have queries that you can run on SCCM SQL server to find the recovery keys for each device...

this being said, we still don't have enough online time for the device to be able to receive GPOs or scripts regardless of the management infrastructure SCCM/Intune, as as soon as the device hits the logon..boom BSOD, reboot loop and then you stuck at the manual recovery :)

now we don't know if this was a glitch from CrowdStrike or a targeted attack through CrowdStrike (one that CrowdStrike wouldn't acknowledge anyway :) ) right before the Olympic games :D

@ThoughtContagion
Copy link

ThoughtContagion commented Jul 19, 2024

Assuming you can access a Server OS (preferably a DC), this automates the entire thing and fixes an issue I had in a labbed environment where the reboot GPO caused my VM to just keep rebooting.

Edit - It does not help with systems that cannot get past BitLocker.

Import-Module GroupPolicy

$domain = Get-ADDomain

$gpoNames = @("Crowdstrike Fix $(Get-Date -Format MMddyyyy)", "Forced Reboot - Crowdstrike Fix $(Get-Date -Format MMddyyyy)")


Function Create-GroupPolicy {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [string]
        $GPOName,
        [Parameter(Mandatory = $true)]
        [string]
        $scriptPath
    )

    If ($GPOName -like "CrowdStrike Fix *") {
        $StartupScriptPath = $scriptPath
        $StartupScriptName = "Crowdstrike-Fix.ps1"
    }
    ElseIf ($GPOName -like "Forced Reboot *") {
        $StartupScriptPath = $scriptPath
        $StartupScriptName = "Forced-Reboot.ps1"
    }

    $GPO = New-GPO -Name $GPOName

    $ScriptRegistryPath = "HKLM\Software\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup\0"
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "Script" -Type String -Value $StartupScriptPath
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "Parameters" -Type String -Value ""
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "ExecTime" -Type DWord -Value 0
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "GPO-ID" -Type String -Value $GPO.Id.ToString()
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "DisplayName" -Type String -Value $StartupScriptName

    New-GPLink -Name $GPOName -Target "$($domain.DistinguishedName)"
}

Function Create-RebootScript {
    $script = @'
  $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"

  If (Get-ChildItem -Path $filePath){
    bcdedit /set {current} safeboot networking
    Restart-Computer
  }
  Else {
    Write-Output "No files found. Hit any key to exit."
    Pause
    Exit
  }
'@

    $script | Out-File -FilePath "$PSScriptRoot/Forced-Reboot.ps1"
}

Function Create-CSFixScript {
    $script = @'
  $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"
  $files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue

  foreach ($file in $files) {
    try {
      Remove-Item -Path $file.FullName -Force
      Write-Output "Deleted: $($file.FullName)"
    } catch {
      Write-Output "Failed to delete: $($file.FullName)"
    }
  }

  bcdedit /deletevalue {current} safeboot
'@

    $script | Out-File -FilePath "$PSScriptRoot/Crowdstrike-Fix.ps1"
}

Create-CSFixScript
Create-RebootScript

ForEach ($gpName in $gpoNames) {
    Create-GroupPolicy -GPOName $gpName -scriptPath $PSScriptRoot
}

@tig0ss
Copy link

tig0ss commented Jul 19, 2024

Someone managed to use this automation to fix the problem in servers?

@sanjay7178
Copy link

Just use Linux nerds <3

image

btw I use arch

@percu
Copy link

percu commented Jul 20, 2024

@Inoriol
Copy link

Inoriol commented Jul 20, 2024

Just use Linux nerds <3

Well...
https://access.redhat.com/solutions/7068083

@Grime121
Copy link

Grime121 commented Jul 20, 2024

Didn’t work for me. Computers reboot too quickly for the GPO/script to run. Also, I had to change the “bcdedit” lines to use Start-Process. Running them directly from PowerShell resulted in not all the arguments getting passed to it (bcdedit failed with “bad parameters” error).

@Grime121
Copy link

Grime121 commented Jul 20, 2024

Didn’t work for me. Computers reboot too quickly for the GPO/script to run. Also, I had to change the “bcdedit” lines to use Start-Process. Running them directly from PowerShell resulted in not all the arguments getting passed to it (bcdedit failed with “bad parameters” error).

Actually, it's not that the system crashes before the script can run. It's that the boot config simply cannot be modified by a startup script. I even tried using Start-Process to call bcdedit, and specifying credentials rather than using the local system account permissions. It simply will not work. I was finally able to get the error that it is throwing when the script runs at startup. It is:

The boot configuration data store could not be opened.
A required privilege is not held by the client.

I don't know of any way to get around this. This simply won't work as a startup script. In fact, even the command to delete the file doesn’t seem to work when you manually boot the computer to safe mode w/ networking. I tried a startup script that had a single line in it (the Remove-Item line), applied it to computers that I manually rebooted into safe mode, and it was unable to delete the file.

At this point, we are pretty much stuck with booting from WinPE images to delete the file. We’re having to walk users through this process on 7k+ systems, not including the 1k servers that we have already remediated. CrowdStrike should die a very slow and painful death for this…..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment