Google Cloud Network Congestion From a Misapplied Configuration Change
- Organization:
- Google Cloud
- Date:
- 2019-06-02
- Severity:
- sev1
- Failure domain:
- network-config
Summary
A configuration change intended for a small number of servers in one region was applied to a larger set of servers across several neighboring regions. This reduced network capacity and caused significant congestion and elevated latency for Google Cloud and some Google services.
Architecture context
No content available
Customer impact
Elevated latency and packet loss for services in affected regions over multiple hours; some Google services were also affected.
Causal factors
- configuration-scope-errorfact
The change was applied to more servers and regions than intended, reducing network capacity.
- recovery-tooling-impactinference
Congestion also hampered the network used by engineers to diagnose and repair the problem.
Recovery steps
No content available
Lessons learned
Configuration changes need validated scope limits so a mistake cannot span many regions.
Recommended control: Enforce scope checks and keep an independent management path for recovery.
Timeline
- inference
The configuration change takes effect and network congestion begins.
Evidence: Google Cloud
- unknown
Engineers diagnose the issue and restore capacity; exact times are not stated here.
Evidence: Google Cloud
Sources
- tier1 — Google Cloud ()