CrowdStrike 2024 Incident Summary

On July 19, 2024, CrowdStrike caused a global IT outage which affected over 8.5 million Windows computers. This outage caused airports to delay/cancel flights, hospitals to defer patients, and banking systems to go down, among many more.

It has been described as being the most impactful IT outage of the 21st century, but what caused the outage?

One of the main things to note is that July 19th was a Friday, usually the last day of the week that most people work.

What was the issue?

The issue was that Falcon, one of CrowdStrike's suite of security products had bad code that caused it to crash. Falcon is a kernel-level access program, meaning that it runs super deep inside Windows. Anything that crashes with kernel-level access will likely crash the entire operating system.

Falcon is also a program that runs as soon as Windows boots up. So if Falcon or any kernel-level load-on-startup program were to fail, it would cause a boot-crash loop.

Unfortunately, that is what happened with Falcon.

According to CrowdStrike, the channel file, C-00000291, responsible for evaluating pipe executions, crashed because of an update pushed to production.

According to Tavis Ormandy on X, formerly known as Twitter, the root cause was dereferencing a null pointer. I don't know C++ that well, but I know that doing anything with null values except for handling "do this if something is null" is probably a bad idea. This is banner.ts from my discord bot, Asomataru. Note that this screenshot may not apply to the current releases of Asomataru.

In this code, userBanner.banner could be null, so I return a warning so that the bot doesn't potentially crash if the selected user doesn't have a banner (e.g. non-nitro account).

What has been done since then?

CrowdStrike has officially released a "how-to-fix" video to fix this issue on Falcon. From July 19-26, many companies have recovered from the issue, but some have not. CrowdStrike's stock has dropped by almost half since then.

Fun fact: it dropped by 2 cents right after I took this screenshot.

Ending

Pro tip: Don't push to production on a Friday.