What happened
On 19 July 2024 CrowdStrike pushed a content update (Channel File 291) to its Falcon sensor on Windows. The update had a mismatch between the fields the sensor expected and the data it got, which led to an out-of-bounds memory read in kernel code. Around 8.5 million Windows machines crashed and kept crashing on boot. Airlines, hospitals and banks were hit. Many machines needed someone to fix them by hand.
The published manual fix was to boot into safe mode or recovery and delete the file:
del %WINDIR%\System32\drivers\CrowdStrike\C-00000291*.sys
CrowdStrike's hub: Falcon content update remediation and guidance.
What I took from it
It was "just a content update." Content updates went out fast and to everyone, because their job was to respond to new threats quickly. Code releases had more gates. The content ran in the kernel all the same.
It's the same lesson as Cloudflare's 2019 regex outage, five years later, at a much bigger scale. Anything that changes what production does needs a staged rollout, whatever you call it.
How I'd check a team for this
# which deploy paths skip the canary stage? (e.g. grep pipelines for direct prod targets)
grep -rnE "target[:=] *(prod|all)" .github/workflows/ jenkins/ 2>/dev/null
# what in our estate updates itself? (agents, sensors, auto-updating packages)
systemctl list-timers --all | grep -iE "update|upgrade"
What I do now
- Treat config, rules, feature flags and content like code: staged rollout, with automatic stop on errors.
- Know which agents in your estate update themselves, and on whose schedule.
- Have a recovery path for "every machine won't boot" that doesn't rely on the machines.
stage 0 looks at how your changes reach production → get the free audit