Automated CrowdStrike BSOD Workaround in Safe Mode using Group Policy

CRWD-GPO.md

Automated Workaround in Safe Mode using Group Policy

You can set up a GPO to run a script during Safe Mode. Here’s how you can do this:

1. Create the PowerShell Script

   Create a PowerShell script that deletes the problematic CrowdStrike driver file causing BSODs and handles the Safe Mode boot and revert:

   # CrowdStrikeFix.ps1
   # This script deletes the problematic CrowdStrike driver file causing BSODs and reverts Safe Mode
   
   $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"
   $files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue
   
   foreach ($file in $files) {
       try {
           Remove-Item -Path $file.FullName -Force
           Write-Output "Deleted: $($file.FullName)"
       } catch {
           Write-Output "Failed to delete: $($file.FullName)"
       }
   }
   
   # Revert Safe Mode Boot after Fix
   bcdedit /deletevalue {current} safeboot
   

2. Create a GPO for Safe Mode

   • Open the Group Policy Management Console (GPMC).
   • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here....
   • Name the GPO, for example, CrowdStrike Fix Safe Mode.

3. Edit the GPO

   • Right-click the new GPO and select Edit.
   • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
   • Double-click Startup, then click Add.
   • In the Script Name field, browse to the location where you saved CrowdStrikeFix.ps1 and select it.
   • Click OK to close all dialog boxes.

4. Force Safe Mode Boot Using a Script

   Create another PowerShell script to force Safe Mode boot and link it to a GPO for immediate application:

   # ForceSafeMode.ps1
   # This script forces the computer to boot into Safe Mode
   
   bcdedit /set {current} safeboot minimal
   Restart-Computer
   

5. Create a GPO to Apply the Safe Mode Script

   • Open the Group Policy Management Console (GPMC).
   • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here....
   • Name the GPO, for example, Force Safe Mode.
   • Right-click the new GPO and select Edit.
   • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
   • Double-click Startup, then click Add.
   • In the Script Name field, browse to the location where you saved ForceSafeMode.ps1 and select it.
   • Click OK to close all dialog boxes.

6. Apply the GPOs

   • Make sure the Force Safe Mode GPO is applied to the affected computers first.
   • The computer will boot into Safe Mode and execute the CrowdStrikeFix.ps1 script.
   • Once the issue is fixed, the script will revert the boot settings to normal mode.

Can someone explain how a GPO startup script works if the device can't boot into Windows?

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?

I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?
I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

So there's no way to securely unlock a device automatically with a script? So this outage is only manually fixable if you have MDE implemented?

Can someone explain how a GPO startup script works if the device can't boot into Windows?

good point, this method would work if the device stays online enough time for the GPO to be deployed, but the devices are throwing BSOD couple of seconds after they reached the windows logon screen. this loop is made several time until the device stops booting and you need to take manual steps to boot again.

This is a problem caused by CrowdStrike 3rd party software, has nothing to do with Windows.

How this solution is suposed to solve the issue if the computers are not booting because of the BSOD? They don't have a network connection to reach de domain and apply the GPO...

The only way the GPO can be applied is entering into safe mode with networking computer by computer.

Anyone know how to incorporate this: ( https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically ) with Bitlockr?
I was wondering if someone could write a script to grab the recovery key and somehow deploy this package?

that would be a security issue in itself ... as you will expose the key, and that is not ok for several reasons :)

So there's no way to securely unlock a device automatically with a script? So this outage is only manually fixable if you have MDE implemented?

i usually work with Intune managed devices, and from there i can extract for each device all the bitlocker recovery keys for each drive that they have, using Graph API calls and some powershell scripting... but this would require an EntraID to be created with an appId and appSecret/certificate, you have to give specific ReadAll permissions to the app to a specific endpoint from Graph API, then using powershell you need to iterate through all device Ids and use them in the API calls to extract the bitlocker recovery keys from another API endpoint, then use those keys to somehow unlock the devices, you need to match the device with it's own recovery key for the specific drive on which windows is installed and do some scripting magic to decrypt the device, remove the dreaded sys file and then reboot it.
i would not do it like this, as i don't want to risk having client devices unencrypted ;)

on SCCM you also have queries that you can run on SCCM SQL server to find the recovery keys for each device...

this being said, we still don't have enough online time for the device to be able to receive GPOs or scripts regardless of the management infrastructure SCCM/Intune, as as soon as the device hits the logon..boom BSOD, reboot loop and then you stuck at the manual recovery :)

now we don't know if this was a glitch from CrowdStrike or a targeted attack through CrowdStrike (one that CrowdStrike wouldn't acknowledge anyway :) ) right before the Olympic games :D

Assuming you can access a Server OS (preferably a DC), this automates the entire thing and fixes an issue I had in a labbed environment where the reboot GPO caused my VM to just keep rebooting.

Edit - It does not help with systems that cannot get past BitLocker.

Import-Module GroupPolicy

$domain = Get-ADDomain

$gpoNames = @("Crowdstrike Fix $(Get-Date -Format MMddyyyy)", "Forced Reboot - Crowdstrike Fix $(Get-Date -Format MMddyyyy)")


Function Create-GroupPolicy {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [string]
        $GPOName,
        [Parameter(Mandatory = $true)]
        [string]
        $scriptPath
    )

    If ($GPOName -like "CrowdStrike Fix *") {
        $StartupScriptPath = $scriptPath
        $StartupScriptName = "Crowdstrike-Fix.ps1"
    }
    ElseIf ($GPOName -like "Forced Reboot *") {
        $StartupScriptPath = $scriptPath
        $StartupScriptName = "Forced-Reboot.ps1"
    }

    $GPO = New-GPO -Name $GPOName

    $ScriptRegistryPath = "HKLM\Software\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup\0"
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "Script" -Type String -Value $StartupScriptPath
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "Parameters" -Type String -Value ""
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "ExecTime" -Type DWord -Value 0
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "GPO-ID" -Type String -Value $GPO.Id.ToString()
    Set-GPRegistryValue -Name $GPOName -Key $ScriptRegistryPath -ValueName "DisplayName" -Type String -Value $StartupScriptName

    New-GPLink -Name $GPOName -Target "$($domain.DistinguishedName)"
}

Function Create-RebootScript {
    $script = @'
  $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"

  If (Get-ChildItem -Path $filePath){
    bcdedit /set {current} safeboot networking
    Restart-Computer
  }
  Else {
    Write-Output "No files found. Hit any key to exit."
    Pause
    Exit
  }
'@

    $script | Out-File -FilePath "$PSScriptRoot/Forced-Reboot.ps1"
}

Function Create-CSFixScript {
    $script = @'
  $filePath = "C:\Windows\System32\drivers\C-00000291*.sys"
  $files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue

  foreach ($file in $files) {
    try {
      Remove-Item -Path $file.FullName -Force
      Write-Output "Deleted: $($file.FullName)"
    } catch {
      Write-Output "Failed to delete: $($file.FullName)"
    }
  }

  bcdedit /deletevalue {current} safeboot
'@

    $script | Out-File -FilePath "$PSScriptRoot/Crowdstrike-Fix.ps1"
}

Create-CSFixScript
Create-RebootScript

ForEach ($gpName in $gpoNames) {
    Create-GroupPolicy -GPOName $gpName -scriptPath $PSScriptRoot
}

Someone managed to use this automation to fix the problem in servers?

Just use Linux nerds <3

btw I use arch

Just use Linux nerds <3

https://forums.rockylinux.org/t/crowdstrike-freezing-rockylinux-after-9-4-upgrade/14041

Just use Linux nerds <3

Well...
https://access.redhat.com/solutions/7068083

Didn’t work for me. Computers reboot too quickly for the GPO/script to run. Also, I had to change the “bcdedit” lines to use Start-Process. Running them directly from PowerShell resulted in not all the arguments getting passed to it (bcdedit failed with “bad parameters” error).

Didn’t work for me. Computers reboot too quickly for the GPO/script to run. Also, I had to change the “bcdedit” lines to use Start-Process. Running them directly from PowerShell resulted in not all the arguments getting passed to it (bcdedit failed with “bad parameters” error).

Actually, it's not that the system crashes before the script can run. It's that the boot config simply cannot be modified by a startup script. I even tried using Start-Process to call bcdedit, and specifying credentials rather than using the local system account permissions. It simply will not work. I was finally able to get the error that it is throwing when the script runs at startup. It is:

The boot configuration data store could not be opened.
A required privilege is not held by the client.

I don't know of any way to get around this. This simply won't work as a startup script. In fact, even the command to delete the file doesn’t seem to work when you manually boot the computer to safe mode w/ networking. I tried a startup script that had a single line in it (the Remove-Item line), applied it to computers that I manually rebooted into safe mode, and it was unable to delete the file.

At this point, we are pretty much stuck with booting from WinPE images to delete the file. We’re having to walk users through this process on 7k+ systems, not including the 1k servers that we have already remediated. CrowdStrike should die a very slow and painful death for this…..