{"UUID":"7f0cc221-46c7-485e-bf6c-1719dc4c4e49","URL":"https://blog.npmjs.org/post/74949623024/2014-01-28-outage-postmortem.html","ArchiveURL":"","Title":"NPM Fastly VCL misconfiguration outage on 2014-01-28","StartTime":"2014-01-28T00:00:00Z","EndTime":"0001-01-01T00:00:00Z","Categories":["cascading-failure"],"Keywords":["npm","fastly","varnish","vcl","manta","couchdb","outage","403"],"Company":"NPM","Product":"npm registry","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"2026-05-04T17:49:58.204687Z","Summary":"Fastly configuration change caused backend routing issue. To be exact, the issue is that we were setting the req.backend in a vcl_fetch function, and then calling restart to re-parse the rules. However, calling restart will reset the req.backend to the first backed in the list, which in this case happened to be Manta, rather than the load balanced CouchDB servers.","Description":"On January 28, 2014, npm experienced an outage due to a configuration change made to the Varnish VCL on Fastly. This change inadvertently misrouted all incoming requests, including those intended for CouchDB, to Manta.\n\nAs Manta was not equipped to handle these requests, users received 403 Forbidden responses. Fastly's configuration, which does not cache error codes, exacerbated the issue by creating a \"thundering herd\" effect, prolonging the incident.\n\nThe root cause was identified as a bug in the VCL configuration. Specifically, `req.backend` was set within a `vcl_fetch` function, and a subsequent `restart` call reset `req.backend` to the default first backend (Manta) instead of maintaining the intended CouchDB routing.\n\nTo prevent recurrence, future Fastly VCL configuration changes will ensure `req.backend` is explicitly set within the `vcl_recv` function, even after a `restart`. Additionally, npm plans to implement a separate Varnish instance for staging VCL changes to catch errors before deployment."}