AWS US-EAST-1 Kinesis Overload and Cascading Failure
- Organization:
- Amazon Web Services
- Date:
- 2021-12-07
- Severity:
- sev1
- Failure domain:
- retry-storm
Practice this incident in the simulator: Kinesis Cascade: Diagnosing a Control Plane Failure
Summary
A capacity management error in AWS US-EAST-1 caused Amazon Kinesis to become overloaded, triggering a cascading failure across multiple dependent AWS services including Cognito, CloudWatch, and Lambda. The incident lasted approximately seven hours and affected a large number of customers.
Architecture context
No content available
Customer impact
Widespread degradation of AWS services in the US-EAST-1 region for approximately 7 hours. Services affected included Amazon Kinesis, AWS Lambda, Amazon Cognito, Amazon CloudWatch, AWS EventBridge, and others relying on internal control planes.
Causal factors
- capacity-misconfigurationfact
An automated capacity management activity added capacity in a way that overloaded the front-end Kinesis servers beyond their thread limits.
- hidden-dependencyinference
Multiple internal AWS services used Kinesis as a control plane channel, causing cascading errors when Kinesis degraded.
Recovery steps
No content available
Lessons learned
Hidden internal dependencies on shared infrastructure should be explicitly tracked and visible in service maps.
Recommended control: Maintain a live service dependency graph and run regular dependency audits on critical infrastructure.
Services recovering from a dependency outage should use exponential backoff with jitter to prevent thundering herd re-triggers.
Recommended control: Review retry and reconnect logic across all services that depend on shared control-plane infrastructure.
Timeline
- fact
Automated activity triggered a capacity management error causing front-end server overload in the Kinesis data plane.
Evidence: Amazon Web Services
- fact
AWS engineers identify the root cause and begin scaling the Kinesis front-end fleet.
Evidence: Amazon Web Services
- fact
Kinesis and most dependent services recover to full operation.
Evidence: Amazon Web Services
Sources
- tier1 — Amazon Web Services ()