{"UUID":"32b081c5-bfd7-4986-82e6-9e9cd7740c95","URL":"https://blog.cloudflare.com/partial-cloudflare-outage-on-october-25-2022/","ArchiveURL":"https://web.archive.org/web/20221112015610if_/https://blog.cloudflare.com/partial-cloudflare-outage-on-october-25-2022/","Title":"Partial Cloudflare outage on October 25, 2022","StartTime":"2022-10-25T08:40:00Z","EndTime":"2022-10-25T18:04:00Z","Categories":null,"Keywords":["cloudflare","tiered cache","cdn","outage","530 error","rollback","distributed tracing","dns","530 errors","control headers"],"Company":"Cloudflare","Product":"Tiered Cache","SourcePublishedAt":"2022-10-26T03:02:50Z","SourceFetchedAt":"2026-05-04T18:12:35.803594Z","Summary":"A change to our Tiered Cache system caused some requests to fail for users with status code 530. The impact lasted for almost six hours in total. We estimate that about 5% of all requests failed at peak. Because of the complexity of our system and a blind spot in our tests, we did not spot this when the change was released to our test environment.","Description":"On October 25, 2022, a software release for a Cloudflare CDN component began rolling out at 08:40 UTC, leading to a partial outage. The issue caused HTTP 530 errors, affecting approximately 5% of all HTTP requests at its peak at 17:28 UTC. Customers using Tiered Cache, Cloudflare Images, or Bandwidth Alliance were impacted if their requests required an origin fetch through the tiered cache hierarchy.\n\nThe problem stemmed from new distributed tracing code introduced into the Tiered Cache logic. This code inadvertently cleared control headers, which are crucial for routing requests and performing internal DNS lookups for origin server IP addresses. Consequently, essential hostname information was missing, resulting in the 530 DNS errors observed by clients.\n\nCloudflare initiated a rollback of the faulty release, confirming the root cause at 17:03 UTC. An accelerated global rollback followed, with impact ending once the reversion was complete in all upper tier data centers by 18:04 UTC.\n\nTo prevent similar incidents, Cloudflare plans to include larger Tiered Cache upper tier data centers in earlier release stages, expand acceptance test coverage for various Tiered Cache topologies, implement more aggressive alerting for requests lacking full context, and ensure systems fail fast during development and testing."}