{"UUID":"513195e1-adfd-4d8d-ad19-2e723988356d","URL":"https://status.cloud.google.com/incidents/X8SNkK2BPyCrc1sveeiu","ArchiveURL":"","Title":"Cloud Filestore ListInstances API failed with error code 429 globally","StartTime":"2022-09-13T14:38:00Z","EndTime":"2022-09-13T17:42:00Z","Categories":["automation","cloud","time"],"Keywords":["cloud filestore","api","429 error","global","throttling","listinstances","gcp"],"Company":"Google","Product":"Cloud Filestore","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"2026-05-04T17:47:59.372026Z","Summary":"Filestore enforces a global limit on API requests to limit impact in overload scenarios. The outage was triggered when an internal Google service managing a large number of GCP projects malfunctioned and overloaded the Filestore API with requests, causing global throttling of the Filestore API. This continued until the internal service was manually paused. As a result of this throttling, read-only API access was unavailable for all customers. This affected customers in all locations, due to a global quota that applies to Filestore. Console, gcloud and API access (List, GetOperation, etc.) calls all failed for a duration of 3 hours, 12 minutes. Mutate operations (CreateInstance, UpdateInstance, CreateBackup, etc.) still succeeded, but customers were unable to check on operation progress.","Description":"On Tuesday, September 13, 2022, Google Cloud Filestore experienced an incident where all read-only API requests, specifically the ListInstances API, failed with error code 429 globally. This issue began at 07:38 US/Pacific and was fully mitigated by 10:42 US/Pacific, lasting for 3 hours and 4 minutes.\n\nThe root cause was identified as an internal Google service, managing a large number of GCP projects, malfunctioning and overloading the Filestore API with requests. Filestore's global API request limit, intended to prevent overload, was triggered, leading to global throttling of the API.\n\nAs a result of this throttling, customers globally were unable to access read-only API functions for Cloud Filestore. This impacted Console, gcloud, and direct API access for operations like List and GetOperation. While mutate operations (e.g., CreateInstance, UpdateInstance) still succeeded, customers could not check their progress.\n\nGoogle engineers were alerted at 07:47 US/Pacific. Once the nature and scope of the issue were understood, engineers began shutting down the overloading internal service at 10:33 US/Pacific, with the issue fully resolved by 10:42 US/Pacific.\n\nTo prevent recurrence, Google plans to remove the global quotas that caused a local overload to have a global impact. They will also improve alerting and monitoring to detect and mitigate issues more promptly when internal limits are approached."}