{"UUID":"3ca9020c-e384-4964-98b6-68268d9e42ec","URL":"https://blog.heroku.com/how-i-broke-git-push-heroku-main","ArchiveURL":"https://web.archive.org/web/20240920174200/https://blog.heroku.com/how-i-broke-git-push-heroku-main","Title":"How I Broke `git push heroku main`","StartTime":"0001-01-01T00:00:00Z","EndTime":"0001-01-01T00:00:00Z","Categories":["cloud","config-change"],"Keywords":["heroku","git","deployment","configuration","401","kernel app","manual process","incident"],"Company":"Heroku","Product":"Git server component","SourcePublishedAt":"2020-10-01T20:30:00Z","SourceFetchedAt":"2026-05-04T17:46:58.272857Z","Summary":"An incorrect deployment process caused new config variables not to be used when the code required them.","Description":"On September 3, an incident occurred with Heroku's Git server component, which handles `git push heroku main` operations. The issue arose during a deployment that required a new configuration variable for an internal API endpoint.\n\nThe deployment process for this \"kernel app\" component involved several manual steps: setting the config variable, transmitting it to instances, and then deploying the new code. The author correctly set the variable for production but failed to transmit it to the production instances before initiating the code deployment.\n\nThis oversight led to a batch of production instances running new code that attempted to make unauthenticated calls to an internal API, resulting in HTTP 401 errors for users performing `git push heroku main`. An alert was triggered, and the problem was quickly identified.\n\nThe immediate resolution involved transmitting the correct configuration variables to the affected instances and restarting their processes. To prevent recurrence, Heroku plans to merge the steps of updating configuration variables and deploying code into a single, atomic action, or implement a safeguard warning if configuration variables are out of sync during a deployment."}