Skip to main content
OpsAtlas
Educational Reconstruction — not a verbatim account

Cloudflare Global Outage From WAF Regular Expression CPU Exhaustion

Organization:
Cloudflare
Date:
2019-07-02
Severity:
sev1
Failure domain:
cpu-exhaustion

Summary

A new Web Application Firewall managed rule containing a regular expression with excessive backtracking was deployed globally. It exhausted CPU on machines handling HTTP traffic and produced 502 errors worldwide until the WAF was disabled with a global kill.

Architecture context

No content available

Customer impact

Cloudflare-proxied sites returned 502 errors worldwide for roughly half an hour.

Causal factors

  • unsafe-regexfact

    A regular expression with excessive backtracking consumed CPU on every request path that evaluated it.

  • unstaged-rolloutfact

    The rule change was released worldwide at once without a gradual, canaried rollout.

Recovery steps

No content available

Lessons learned

  • Rule and configuration changes deserve the same staged rollout and resource limits as code changes.

    Recommended control: Stage rule releases and use regex engines with bounded execution time.

Timeline

  1. fact

    The new WAF rule is deployed globally and CPU usage spikes on edge machines.

    Evidence: Cloudflare

  2. inference

    A global kill of the WAF managed rulesets is decided and executed.

    Evidence: Cloudflare

Sources