{"UUID":"b4b5957d-2c5f-4db4-b866-8d82da25bd93","URL":"https://keepthescore.co/blog/posts/deleting_the_production_database/","ArchiveURL":"","Title":"Leaderboarded production database accidental deletion","StartTime":"2020-10-17T21:45:00Z","EndTime":"2020-10-17T22:15:00Z","Categories":["config-change"],"Keywords":["database","deletion","leaderboarded","digitalocean","flask","python","production","data loss"],"Company":"Keepthescore","Product":"Leaderboarded","SourcePublishedAt":"2020-10-17T00:00:00Z","SourceFetchedAt":"2026-05-04T17:52:25.267157Z","Summary":"Engineers deleted the production database by accident. Database is a managed database from DigitalOcean with backups once a day. 30 minutes after the disaster, it went back online, however 7 hours of scoreboard data was gone forever.","Description":"On October 17, 2020, at approximately 10:45 PM CET, the production database for Leaderboarded.com was accidentally deleted. This incident resulted in the immediate loss of over 300,000 scoreboards and their associated data.\n\nThe root cause was a misconfiguration in the application's environment. A function intended to wipe a local development database was inadvertently executed against the live production database. Although the function's code hardcoded 'localhost' for the database host, an incorrect `FLASK_ENV` setting caused the application to connect to the production environment.\n\nThe customer impact was significant data loss. Any scoreboards created or scores added between 3:47 PM CET and 11:21 PM CET on October 17, 2020, were permanently lost. The service was put into maintenance mode shortly after the incident began.\n\nWithin 30 minutes, by 11:15 PM CET, the service was brought back online using a daily backup provided by DigitalOcean. While this allowed for quick recovery, the data created during the 7-hour window between the last backup and the incident was irrecoverable. The incident highlighted the danger of having database-deleting functions and the critical importance of reliable, quick-recovery backups. The developer plans to implement measures to prevent this specific error from recurring."}