A 43-second network partition occurred on October 21, 2018, at 22:52 UTC, during routine maintenance to replace optical equipment. This brief loss of connectivity between GitHub’s US East Coast network hub and its primary data center triggered an automated MySQL master failover managed by Orchestrator. Due to cross-continent latency, the new primaries in the US West Coast data center did not receive several seconds of writes that had occurred on the East Coast, leading to divergent database states.

Upon restoration of network connectivity, applications began directing write traffic to the new West Coast primaries. However, the East Coast databases contained unique writes not replicated to the West Coast, preventing a safe failback. Engineers quickly identified the unexpected database topologies and, by 23:13 UTC, decided to prioritize data integrity over immediate service restoration, accepting an extended period of degradation.

To prevent further data corruption, GitHub paused webhook delivery and GitHub Pages builds at 23:19 UTC. The remediation plan involved restoring affected MySQL clusters from backups, synchronizing replicas, and re-establishing a stable serving topology. This process was lengthy, as it required restoring multiple terabytes of data from remote cloud storage, which took many hours despite daily testing of the backup procedure.

By October 22, 11:12 UTC, all database primaries were re-established in the US East Coast, significantly improving site responsiveness. However, many read replicas remained hours behind the primary, causing users to experience inconsistent data. The incident resulted in degraded service for 24 hours and 11 minutes, impacting various GitHub services, though no user data was ultimately lost.