Skip to main content
OpsAtlas
Educational Reconstruction — not a verbatim account

GitHub Database Failover After Brief Network Partition

Organization:
GitHub
Date:
2018-10-21
Severity:
sev1
Failure domain:
database-failover

Summary

A brief loss of connectivity between GitHub's US East Coast network hub and its primary East Coast data center led automated MySQL failover to promote West Coast primaries. Writes then existed in both coasts, and reconciling the data led to degraded service for roughly a day.

Architecture context

No content available

Customer impact

Degraded GitHub service, with stale or inconsistent data shown and delayed webhook and Pages builds, until data consistency was restored.

Causal factors

  • automated-failoverfact

    Automated topology management reacted to a transient partition by promoting primaries in another region.

  • cross-region-replicationinference

    Cross-coast replication could not carry writes made on the original primaries, creating divergent data.

Recovery steps

No content available

Lessons learned

  • Automated failover should tolerate transient network partitions rather than act on a single brief signal.

    Recommended control: Add damping and cross-region guardrails to automated promotion decisions.

Timeline

  1. fact

    Connectivity between the East Coast network hub and the primary East Coast data center is lost for a short period.

    Evidence: GitHub Engineering

  2. inference

    Automated orchestration promotes West Coast database primaries, so each coast holds writes the other lacks.

    Evidence: GitHub Engineering

  3. unknown

    Restoration of data consistency continues over many hours before full service returns.

    Evidence: GitHub Engineering

Sources