Billions of Computers Not Working | Unraveling the Issue Behind the Global CrowdStrike Chaos

Written by dhanushnehru | Published 2024/07/23
Tech Story Tags: microsoft | crowdstrike-outage | crowdstrike-update-outage | crowdstrike-it-outage | crowdstrike-incident-report | crowdstrike | global-it-outage | crowdstrike-outage-explained

TLDRCrowdStrike Falcon update accidentally turned a critical driver file into a series of zeros. As a result, computers running Falcon couldnโ€™t boot up, leading to blue screens across the globe. Fixing this issue isn't simple. Users need to boot in safe mode, use command prompts, and manually delete the corrupted driver file.via the TL;DR App

Introduction:

Imagine waking up to find that every PC around the world, especially those in businesses, is suddenly unusable. Business meetings are falling apart, news networks are in disarray, and flights are grounded. The chaos is beyond anything imaginable. What happened? Letโ€™s break down the CrowdStrike blue screen chaos.

The Issue:

The issue began with a bad update from CrowdStrike Falcon. This update, intended to enhance security, accidentally turned a critical driver file into a series of zeros. As a result, computers running Falcon couldnโ€™t boot up, leading to blue screens across the globe.

You might wonder why only Windows was affected. Itโ€™s because Windows, despite being popular, has more security vulnerabilities compared to other operating systems like macOS or Linux. Falconโ€™s deep integration with Windows, designed to bolster security, ironically made it a single point of failure when things went wrong.

The Fix:

Fixing this issue isnโ€™t simple. Users need to boot in safe mode, use command prompts, and manually delete the corrupted driver file. For systems with BitLocker encryption, the process is even more complicated, requiring additional steps to decrypt the hard drive.

The various information received detail the fix below ๐Ÿ‘‡

๐—ฃ๐—ต๐˜†๐˜€๐—ถ๐—ฐ๐—ฎ๐—น ๐—–๐—ผ๐—บ๐—ฝ๐˜‚๐˜๐—ฒ๐—ฟ๐˜€:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment

  2. Navigate to the ๐—–:\๐—ช๐—ถ๐—ป๐—ฑ๐—ผ๐˜„๐˜€\๐—ฆ๐˜†๐˜€๐˜๐—ฒ๐—บ๐Ÿฏ๐Ÿฎ\๐—ฑ๐—ฟ๐—ถ๐˜ƒ๐—ฒ๐—ฟ๐˜€\๐—–๐—ฟ๐—ผ๐˜„๐—ฑ๐—ฆ๐˜๐—ฟ๐—ถ๐—ธ๐—ฒ ๐—ฑ๐—ถ๐—ฟ๐—ฒ๐—ฐ๐˜๐—ผ๐—ฟ๐˜†

  3. Locate the file matching โ€œ๐—–-๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฎ๐Ÿต๐Ÿญ*.๐˜€๐˜†๐˜€โ€ ๐—ฎ๐—ป๐—ฑ ๐—ฑ๐—ฒ๐—น๐—ฒ๐˜๐—ฒ ๐—ถ๐˜ (I would rename it to be safe).

  4. Boot the host

๐—™๐—ผ๐—ฟ ๐—”๐—ช๐—ฆ (๐—”๐—บ๐—ฎ๐˜‡๐—ผ๐—ป ๐—ช๐—ฒ๐—ฏ ๐—ฆ๐—ฒ๐—ฟ๐˜ƒ๐—ถ๐—ฐ๐—ฒ๐˜€), ๐—ณ๐—ผ๐—น๐—น๐—ผ๐˜„ ๐˜๐—ต๐—ฒ๐˜€๐—ฒ ๐˜€๐˜๐—ฒ๐—ฝ๐˜€:

  1. Detach the EBS volume from the impacted EC2 instance.

  2. Attach the EBS volume to a new EC2 instance.

  3. Fix the CrowdStrike driver folder.

  4. Detach the EBS volume from the new EC2 instance.

  5. Attach the EBS volume back to the impacted EC2 instance.

๐—™๐—ผ๐—ฟ ๐—”๐˜‡๐˜‚๐—ฟ๐—ฒ, ๐—ณ๐—ผ๐—น๐—น๐—ผ๐˜„ ๐˜๐—ต๐—ฒ๐˜€๐—ฒ ๐˜€๐˜๐—ฒ๐—ฝ๐˜€:

  1. Log in to the Azure console.
  2. Go to Virtual Machines and select the affected VM.
  3. In the upper left of the console, click โ€œConnectโ€.
  4. Click โ€œMore ways to Connectโ€ and then select โ€œSerial Consoleโ€.
  5. Once SAC has loaded, type in โ€˜cmdโ€™ and press Enter.
  6. Type โ€˜ch -si 1โ€™ and press the space bar.
  7. Enter Administrator credentials.
  8. Type the following commands:
  9. โ€˜bcdedit /set {current} safeboot minimalโ€™
  10. โ€˜bcdedit /set {current} safeboot networkโ€™
  11. Restart the VM.
  12. To confirm the boot state, run the command: โ€˜wmic COMPUTERSYSTEM GET BootupStateโ€™.

Checkout this video for a more detailed explanation ๐Ÿ‘‡

https://youtu.be/GgP0EyuN8GA?si=F7WBzG8rvOld4qdh&embedable=true

Conclusion:

This incident underscores the importance of robust cybersecurity practices and the potential risks of deep system integrations.


Thanks for reading; please give a like as a sort of encouragement and also share this post on socials to show your extended support.

Follow for more โฌ

Twitter / Instagram / Github / Youtube / Newsletter / Discord


Written by dhanushnehru | R&D Engineer
Published by HackerNoon on 2024/07/23