GitHub Database Failover After Brief Network Partition
- Organization:
- GitHub
- Date:
- 2018-10-21
- Severity:
- sev1
- Failure domain:
- database-failover
Summary
A brief loss of connectivity between GitHub's US East Coast network hub and its primary East Coast data center led automated MySQL failover to promote West Coast primaries. Writes then existed in both coasts, and reconciling the data led to degraded service for roughly a day.
Architecture context
No content available
Customer impact
Degraded GitHub service, with stale or inconsistent data shown and delayed webhook and Pages builds, until data consistency was restored.
Causal factors
- automated-failoverfact
Automated topology management reacted to a transient partition by promoting primaries in another region.
- cross-region-replicationinference
Cross-coast replication could not carry writes made on the original primaries, creating divergent data.
Recovery steps
No content available
Lessons learned
Automated failover should tolerate transient network partitions rather than act on a single brief signal.
Recommended control: Add damping and cross-region guardrails to automated promotion decisions.
Timeline
- fact
Connectivity between the East Coast network hub and the primary East Coast data center is lost for a short period.
Evidence: GitHub Engineering
- inference
Automated orchestration promotes West Coast database primaries, so each coast holds writes the other lacks.
Evidence: GitHub Engineering
- unknown
Restoration of data consistency continues over many hours before full service returns.
Evidence: GitHub Engineering
Sources
- tier1 — GitHub Engineering ()