An Overview of the CrowdStrike Outage & Proactive Strategies for Mitigating IT Disruption

If your company is still working to recover from the CrowdStrike outage, please reach out to success@zipsec.com for assistance. We’re happy to provide help and advice to impacted companies.

CrowdStrike made headlines early Friday morning as a routine content upgrade pushed bad code to an estimated 8.5 million Windows devices worldwide. A .sys file with problematic content was automatically pushed to Windows PCs running the CrowdStrike Falcon security software, causing systems to display the Blue Screen of Death (BSOD) and enter a boot loop once downloaded. While CrowdStrike was quick to correct the .sys file, the initial error affected the kernel code at the core of operating systems, crashing devices and leaving them unable to reconnect to the internet. This prevented devices from accessing the update automatically through the network, making it necessary for users to manually apply a fix. 

‍For more information, we recommend CrowdStrike’s technical overview published on 7/20 and their preliminary post incident review published on 7/24. 

CrowdStrike bug impacted business across sectors

Experts have placed early estimates on impact as a result of the bug as high as $5.4 billion as disruption continued through early this week. Businesses of all sizes and sectors were affected by the bug, including hospitals, banks, factories, airlines, retail stores, and more, highlighting how software dependent the economy is. As of the stock market close on Wednesday, 7/24, CrowdStrike stock prices had dropped 27%.

Remediation recommendations were shared across security communities

CrowdStrike support and cybersecurity and IT communities across the globe joined together to troubleshoot in the wake of the outage. There were three remediation recommendations that we posted in full on our website on Friday morning: 

1. Rebooting up to 20 times. When connected to the internet via ethernet cable, computers were able to connect to the internet to pull updated sys files around 5% of the time.
2. Booting in Safe mode and deleting file. For devices encrypted with Bitlocker (a common industry setting), the BitLocker key was required to complete this step. 
3. Reinstalling Windows.

Zip Security worked with customers early Friday morning to get affected devices back online, closing out most active tickets by 2pm ET. Bitlocker keys managed by Zip were sent to system administrators and employees to ease remediation. 

CrowdStrike has since created a Remediation & Guidance Hub where you can find the most up-to-date remediation information.

CrowdStrike vows to amend update policies

All eyes are on CrowdStrike over the next few weeks as they complete a root cause analysis and businesses recover and assess overall impact. CEO George Kurtz has been asked to testify in Congress, and it is likely that CrowdStrike will have to pay some fees and refunds. The company has been quick to accept responsibility for their role with senior leadership taking to LinkedIn to apologize. 

CrowdStrike is widely considered to be the gold standard when it comes to Endpoint Detection & Response (EDR) products. While it is clear quality assurance policies need an overhaul, it is unlikely that customers will migrate away from the platform. A few updates CrowdStrike named in the Preliminary Post Incident report include:

• New controls to allow users to control the rate at which they receive content updates
• New release processes to bake content updates, similar to the way sensor updates work today

Proactive strategies for mitigating future IT disruption

On the tactical side, this is a reminder to businesses of the impact IT disruptions can have. It is important that businesses proactively and regularly assess their infrastructure. We recommend companies create cyber-resilience plans, including playbooks to address common scenarios to ensure business disruption is minimized in the event they occur. A few scenarios all businesses should have plans for include what to do if:

• Devices simultaneously become inaccessible
• Company domain is not renewed
• There is a remote takeover of a device
• Credentials are compromised
• Device needs to be restored from backups

It is also important that businesses invest in tooling that automates IT workflows and enables teams to work more efficiently and effectively. Today’s basic enterprise security tech stack has multiple layers requiring different software solutions. For resource constrained IT teams, it's a mountain of responsibility with a steep learning curve and very little room for error. 

Zip enables companies to manage all their security tools in one easy-to-use platform. We’ve built in automated workflows to simplify tasks while increasing security posture visibility. Our opinionated software and white-glove customer support ensures companies have comprehensive coverage. Because we’re a software company at heart, our customers pay a fraction of what they would for similar coverage from an MSP.