Cloudflare Global Outage From WAF Regular Expression CPU Exhaustion
- Organization:
- Cloudflare
- Date:
- 2019-07-02
- Severity:
- sev1
- Failure domain:
- cpu-exhaustion
Summary
A new Web Application Firewall managed rule containing a regular expression with excessive backtracking was deployed globally. It exhausted CPU on machines handling HTTP traffic and produced 502 errors worldwide until the WAF was disabled with a global kill.
Architecture context
No content available
Customer impact
Cloudflare-proxied sites returned 502 errors worldwide for roughly half an hour.
Causal factors
- unsafe-regexfact
A regular expression with excessive backtracking consumed CPU on every request path that evaluated it.
- unstaged-rolloutfact
The rule change was released worldwide at once without a gradual, canaried rollout.
Recovery steps
No content available
Lessons learned
Rule and configuration changes deserve the same staged rollout and resource limits as code changes.
Recommended control: Stage rule releases and use regex engines with bounded execution time.
Timeline
- fact
The new WAF rule is deployed globally and CPU usage spikes on edge machines.
Evidence: Cloudflare
- inference
A global kill of the WAF managed rulesets is decided and executed.
Evidence: Cloudflare
Sources
- tier1 — Cloudflare ()