Skip to main content
OpsAtlas
Educational Reconstruction — not a verbatim account

AWS US-EAST-1 Kinesis Overload and Cascading Failure

Organization:
Amazon Web Services
Date:
2021-12-07
Severity:
sev1
Failure domain:
retry-storm

Practice this incident in the simulator: Kinesis Cascade: Diagnosing a Control Plane Failure

Summary

A capacity management error in AWS US-EAST-1 caused Amazon Kinesis to become overloaded, triggering a cascading failure across multiple dependent AWS services including Cognito, CloudWatch, and Lambda. The incident lasted approximately seven hours and affected a large number of customers.

Architecture context

No content available

Customer impact

Widespread degradation of AWS services in the US-EAST-1 region for approximately 7 hours. Services affected included Amazon Kinesis, AWS Lambda, Amazon Cognito, Amazon CloudWatch, AWS EventBridge, and others relying on internal control planes.

Causal factors

  • capacity-misconfigurationfact

    An automated capacity management activity added capacity in a way that overloaded the front-end Kinesis servers beyond their thread limits.

  • hidden-dependencyinference

    Multiple internal AWS services used Kinesis as a control plane channel, causing cascading errors when Kinesis degraded.

Recovery steps

No content available

Lessons learned

  • Hidden internal dependencies on shared infrastructure should be explicitly tracked and visible in service maps.

    Recommended control: Maintain a live service dependency graph and run regular dependency audits on critical infrastructure.

  • Services recovering from a dependency outage should use exponential backoff with jitter to prevent thundering herd re-triggers.

    Recommended control: Review retry and reconnect logic across all services that depend on shared control-plane infrastructure.

Timeline

  1. fact

    Automated activity triggered a capacity management error causing front-end server overload in the Kinesis data plane.

    Evidence: Amazon Web Services

  2. fact

    AWS engineers identify the root cause and begin scaling the Kinesis front-end fleet.

    Evidence: Amazon Web Services

  3. fact

    Kinesis and most dependent services recover to full operation.

    Evidence: Amazon Web Services

Sources