Skip to main content
OpsAtlas
Educational Reconstruction — not a verbatim account

Google Cloud Network Congestion From a Misapplied Configuration Change

Organization:
Google Cloud
Date:
2019-06-02
Severity:
sev1
Failure domain:
network-config

Summary

A configuration change intended for a small number of servers in one region was applied to a larger set of servers across several neighboring regions. This reduced network capacity and caused significant congestion and elevated latency for Google Cloud and some Google services.

Architecture context

No content available

Customer impact

Elevated latency and packet loss for services in affected regions over multiple hours; some Google services were also affected.

Causal factors

  • configuration-scope-errorfact

    The change was applied to more servers and regions than intended, reducing network capacity.

  • recovery-tooling-impactinference

    Congestion also hampered the network used by engineers to diagnose and repair the problem.

Recovery steps

No content available

Lessons learned

  • Configuration changes need validated scope limits so a mistake cannot span many regions.

    Recommended control: Enforce scope checks and keep an independent management path for recovery.

Timeline

  1. inference

    The configuration change takes effect and network congestion begins.

    Evidence: Google Cloud

  2. unknown

    Engineers diagnose the issue and restore capacity; exact times are not stated here.

    Evidence: Google Cloud

Sources