{"UUID":"50a576ca-3171-417a-b001-50093be2d910","URL":"https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/","ArchiveURL":"","Title":"15 seconds of API downtime during PostgreSQL migration","StartTime":"0001-01-01T00:00:00Z","EndTime":"0001-01-01T00:00:00Z","Categories":["config-change"],"Keywords":["postgres","postgresql","migration","database","api","downtime","lock","foreign key"],"Company":"GoCardless","Product":"API","SourcePublishedAt":"2019-07-08T13:42:49Z","SourceFetchedAt":"2026-05-04T17:47:54.124528Z","Summary":"All queries on a critical PostgreSQL table were blocked by the combination of an extremely fast database migration and a long-running read query, causing 15 seconds of downtime.","Description":"In late January, GoCardless experienced approximately 15 seconds of unexpected API downtime during a planned database migration. Shortly after the schema migration began, API requests started timing out, indicating a service disruption.\n\nThe incident was triggered by an ALTER TABLE statement attempting to add a foreign key constraint. This operation required an AccessExclusive lock on the referenced parent table. Unfortunately, a long-running read query was simultaneously holding an AccessShare lock on the same parent table.\n\nThe AccessExclusive lock, unable to be acquired immediately, entered a queue. Because AccessExclusive locks conflict with all other lock types, subsequent read/write queries from the API also queued up behind it, leading to widespread timeouts and the 15-second API outage.\n\nTo prevent similar issues, GoCardless identified several best practices. These include eliminating long-running queries, running analytical queries on replicas, setting lock_timeout in migration scripts, splitting schema changes into smaller transactions, and keeping PostgreSQL updated to benefit from locking code improvements."}