What happened
On 2 July 2019 Cloudflare deployed a new WAF rule. Its regular expression could backtrack badly on some inputs and drove CPU to 100% on machines across their network. The rule went everywhere at once, because WAF rules were deployed globally to respond quickly to new attacks. Traffic dropped for about half an hour until they turned the rules off.
Cloudflare's write-up: Details of the Cloudflare outage on July 2, 2019.
They changed the rollout to go in stages and moved to a regex engine with guaranteed run times.
How you'd spot it
# one process eating every core on every host, at the same time
top -o %CPU
# where the CPU goes inside it
sudo perf top -p <pid>
And before shipping, test a regex against bad input with a time limit:
timeout 2 python3 -c '
import re, time
p = re.compile(r"(a+)+$")
s = "a" * 28 + "!"
t = time.time(); p.match(s); print(f"{time.time()-t:.2f}s")
' || echo "too slow: catastrophic backtracking"
What I took from it
Config is code when it changes what production does. Most of us had canary stages for application code and pushed config straight to everything.
In a Jenkins pipeline the shape is simple:
stage('canary') { steps { sh './deploy.sh --target canary' } }
stage('verify') { steps { sh './check-error-rate.sh --target canary --max 0.5' } }
stage('rollout') { steps { sh './deploy.sh --target all' } }
What I do now
- Send config and rules through the same staged rollout as code.
- Keep a fast global kill switch, and test it.
- Test regexes against malicious input, or use an engine with linear run time.
stage 0 checks how your changes reach production → get the free audit