On June 8, 2021, Fastly experienced a global disruption beginning at 09:47 UTC. The company’s monitoring identified the issue within a minute, and a status post was published by 09:58 UTC. Fastly Engineering identified the triggering customer configuration by 10:27 UTC, leading to service recovery starting at 10:36 UTC, with the majority of services recovered by 11:00 UTC. The incident was fully mitigated by 12:35 UTC.

The outage was caused by an undiscovered software bug introduced during a deployment that began on May 12. This bug could be triggered under specific circumstances by a valid customer configuration. On June 8, a customer pushed a configuration change that met these specific conditions, activating the bug.

This activation resulted in 85% of Fastly’s network returning errors, leading to a broad and severe global outage. The disruption significantly impacted customers and all those who rely on Fastly’s services.

Fastly’s immediate response involved detecting, identifying, and isolating the cause, then disabling the problematic configuration. Following mitigation, a permanent fix for the bug was created and deployment began at 17:25 UTC on the same day. The company is also conducting a full post-mortem to understand why the bug wasn’t detected during QA, improve remediation times, and enhance platform safety through technologies like WebAssembly and Compute@Edge for greater resiliency.