What happened
On 18 November 2025 Cloudflare had a major outage. A permissions change in a ClickHouse database made the query that generates the Bot Management feature file return duplicate rows, so the file roughly doubled in size. That file is regenerated every few minutes and pushed to the whole network. The proxy code had a hard limit on the number of features, and the bigger file went over it. The proxy crashed and returned 5xx errors across a large part of the internet for around three hours.
Cloudflare's write-up: Cloudflare outage on November 18, 2025.
How you'd catch it before it ships
The file was generated, so nobody reviewed it. The check has to be automatic:
# fail the pipeline if a generated file changes size more than 20% from the last good one
OLD=$(stat -c %s last-good/features.json)
NEW=$(stat -c %s build/features.json)
python3 -c "import sys; o,n=$OLD,$NEW; sys.exit(0 if abs(n-o)/o < 0.2 else 1)" \
|| { echo "features.json changed by more than 20%, holding rollout"; exit 1; }
# and check it against the consumer's hard limits
jq '.features | length' build/features.json # must stay under the proxy's limit
jq '[.features[].name] | group_by(.) | map(select(length>1)) | length' build/features.json # duplicates
What I took from it
It's the third time in this series that a config push, not a code push, took something huge down (Cloudflare 2019, CrowdStrike 2024, Cloudflare 2025). This one adds a twist: the config was generated by a machine, so the "review" step nobody skips for code didn't exist at all.
What I do now
- Validate generated config against the consumer's limits before it ships.
- Compare against the last good version. Big jumps hold the rollout.
- Make consumers fail safe on bad config: keep the last good file and alert, don't crash.
stage 0 checks for the changes nobody reviews → get the free audit