{"UUID":"2552ccf5-4e03-40b1-a1fa-354a34bfa5fe","URL":"https://m.signalvnoise.com/postmortem-on-the-read-only-outage-of-basecamp-on-november-9th-2018/","ArchiveURL":"https://web.archive.org/web/20220529044310if_/https://m.signalvnoise.com/postmortem-on-the-read-only-outage-of-basecamp-on-november-9th-2018/","Title":"","StartTime":"0001-01-01T00:00:00Z","EndTime":"0001-01-01T00:00:00Z","Categories":null,"Keywords":null,"Company":"Basecamp","Product":"Basecamp","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"0001-01-01T00:00:00Z","Summary":"","Description":"In November 2018 a database hit the integer limit, leaving the service in read-only mode while the team migrated the affected `id` column from a 4-byte to an 8-byte integer.\n\nA subsequent [follow-up post](https://web.archive.org/web/20220530044506/https://m.signalvnoise.com/update-on-basecamp-3-being-stuck-in-read-only-as-of-nov-8-922am-cst/) explains that the affected table held the `events` log used by every Basecamp 3 page render, so once writes were paused the rest of the app degraded into a read-only mode for users. Restoring write capacity required altering the column on the live primary, which took several hours of downtime because Postgres has to rewrite the table to widen an integer column."}