On November 12, 2021, Google Cloud experienced networking issues beginning at 00:30 US/Pacific. The incident was fully mitigated by 02:14 US/Pacific, lasting approximately 1 hour and 44 minutes.

The core of the problem was a failure in Google’s Front End load balancing service. Preliminary analysis indicated that a new infrastructure feature inadvertently triggered a latent issue within the internal network load balancer code.

This disruption led to significant customer impact across multiple Google Cloud services and regions. Customers in Europe faced issues with Google Cloud Console (unable to load, timeouts), Container Registry (connection failures in europe-west1), Cloud Bigtable and Cloud Spanner (unavailable errors and latency in Europe), and Cloud Endpoints (unavailable in europe-west1 and europe-west4). Additionally, Firebase Messaging users in asia-southeast1 experienced notification issues, while Cloud SQL saw probe creation request failures in europe-west1, europe-west4, and europe-west5. App Engine and Cloud Functions also reported elevated errors and traffic drops in europe-west1 and us-central1.

Automated safety systems detected the error within four minutes, slowing its spread. Google’s engineering team successfully rolled out a fix to disable the vulnerable code path, fully mitigating the issue. A permanent root cause fix is being implemented to prevent recurrence.