On September 3, an incident occurred with Heroku’s Git server component, which handles git push heroku main operations. The issue arose during a deployment that required a new configuration variable for an internal API endpoint.

The deployment process for this “kernel app” component involved several manual steps: setting the config variable, transmitting it to instances, and then deploying the new code. The author correctly set the variable for production but failed to transmit it to the production instances before initiating the code deployment.

This oversight led to a batch of production instances running new code that attempted to make unauthenticated calls to an internal API, resulting in HTTP 401 errors for users performing git push heroku main. An alert was triggered, and the problem was quickly identified.

The immediate resolution involved transmitting the correct configuration variables to the affected instances and restarting their processes. To prevent recurrence, Heroku plans to merge the steps of updating configuration variables and deploying code into a single, atomic action, or implement a safeguard warning if configuration variables are out of sync during a deployment.