CrowdStrike blames test software for its BSOD issue

      GG

      By Surakshya Acharya

      Content Writer

      Updated on Jul 26, 2024

      CrowdStrike blames test software for its BSOD issue
      In an unparalleled event, Windows users face security issues worldwide as their systems are stuck at the dreaded Blue Screen of Death (BSOD). The root cause? A content update from CrowdStrike's Falcon, a widely used cybersecurity software. This article will go through the details of this global outage, its impact, and the solution.

      CrowdStrike Falcon update causes BSOD

      CrowdStrike, a major player in the cybersecurity industry, recently pushed an update to its Falcon software. One of the files in this software was corrupted which was sent to a huge number of users. This update, intended to enhance security measures, has accidentally caused a widespread crash of Microsoft Windows systems. Well, the issue affected less than 1% of all Windows machines, but in number that counts to a whopping 8.5 million devices. So, how did this happen?
      CrowdStrike Falcon issue
      The main issue was with the C-00000291*.sys file which should've contained device drivers and hardware configurations necessary for the system's operation. However, during the update, CrowdStrike inadvertently replaced the driver's data with null characters. Due to this failure in their build mechanism, the essential content was missing. Consequently, they released an update with a non-functional driver, leading to system instability. That being said, it's also important to note that the CrowdStrike Falcon update which caused BSOD is not a cyber-attack, but rather an unfortunate technical glitch. The crash has affected a wide range of CrowdStrike's clients across various industries. While not all customers are publicly disclosed, some high-profile organizations like Sony, Dell, Hyundai, FedEx, Uber, etc are some of them.
      Airlines resorted to handwriting boarding pass numbers, healthcare services were disrupted, and hospitals had to shut down electronic medical records and what else not.
      Blue screens showing error messages at Chicago O'Hare Airport
      Blue screens showing error messages at Chicago O'Hare Airport. Pic: AP[/caption]
      It is unlikely for a big company like CrowdStrike to make a blunder like this, but everyone makes mistakes, you know!

      Solutions

      While the issue continues, some users have found temporary relief by booting Windows machines into safe mode, deleting a specific system file, and rebooting. However, this was not a long-term solution, and many affected organizations were eagerly waiting for a more permanent fix. Just a few hours ago, CrowdStrike acknowledged the issue and provided a manual solution to its customers. To fix the CrowdStrike Windows 10 BSOD issue, follow these steps:
      1. Boot Windows into Safe Mode or WRE
      2. Navigate to C:/Windows/System32/drivers/CrowdStrike
      3. Locate and delete the file matching "C-000000291*.sys"
      4. Boot normally
      Although the steps look easy, the manual fix is tedious and time-consuming, requiring users to navigate system directories and delete specific files. This makes it a challenging and complex process for many affected users. [caption id="attachment_190411" align="alignnone" width="768"]
      Microsoft facing issue
      Microsoft facing issue

      Looking Ahead

      As cybersecurity keeps changing, this incident shows how important it is to balance security and system stability. While big companies should have a cyber security department of their own, trusting a third-party company with their essential data is risky. This global outage underscores the critical role of cybersecurity software in our increasingly digital world and the potential widespread impact of even minor glitches in such systems.

      CrowdStrike's Response and Impact

      Today CrowdStrike released a preliminary post-incident review addressing the actual cause behind the issue. The company will upload a detailed Root Cause Analysis soon. Here’s what they have got to say about it.

      CrowdStrike's reason behind the issue

      CrowdStrike provides security updates in two ways i.e. Sensor Content and Rapid Response Content. Sensor Content is usually an AI and Machine Learning model and is tested before release. Before becoming available to customers, they go through testing, and even after the availability customers can control when to install these updates. The buggy update on July 19 was not caused by Sensor Content. The main reason behind this cause was the Rapid Response Content, which CrowdStrike uses to perform behavioural pattern-matching to detect threats.
      “These updates are a regular part of the dynamic protection mechanisms of the Falcon platform. The problematic Rapid Response Content configuration update resulted in a Windows system crash,” says CrowdStrike.
      Rapid Response Content updates how the sensor behaves to detect malware. It is quickly deployed from the cloud and doesn't require changes to the sensor's code. Unfortunately, the update that caused the BSOD contained an error that was not detected during validation. Testing and Deployment Process of Rapid Response Content Rapid Response Content is created, validated, and deployed through a comprehensive system that includes:
      • Content Configuration System: This system is responsible for creating and validating updates.
      • Content Interpreter: This component on the sensor reads the updates.
      • Sensor Detection Engine: This engine uses updates to detect threats.
      The update caused crashes because there was a bug in the system that checks for errors, which allowed the faulty update to be released.

      CrowdStrike's opinion

      CrowdStrike promises to improve Rapid Response Content testing by incorporating local developer testing, content update and rollback testing, stress testing, fuzzing, and fault injection. However, this incident highlights the importance of balancing rapid updates and system stability. CrowdStrike is working to prevent similar issues in the future and will release a detailed Root Cause Analysis soon.
      • Meanwhile, watch our Galaxy M35 Review video:

      How did we do with this article?

      Conversation

      We’d love to hear your thoughts! Let's keep it respectful and on-topic. Any inappropriate remarks may be removed. Happy commenting! Privacy Policy

      Be the first to share your thoughts—start the conversation!

      Related News

      More Articles