{"UUID":"91f1928a-c9cd-4473-a761-dc0b07914b2e","URL":"https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/","ArchiveURL":"https://web.archive.org/web/20211029020126if_/https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/","Title":"Cloudflare parser bug causes memory leak","StartTime":"2016-09-22T00:00:00Z","EndTime":"2017-02-18T01:19:00Z","Categories":["config-change","security"],"Keywords":["cloudflare","parser","memory leak","buffer overrun","ragel","project zero","security","nginx","parser bug"],"Company":"Cloudflare","Product":"Cloudflare parser","SourcePublishedAt":"2017-02-23T23:01:06Z","SourceFetchedAt":"2026-05-04T18:16:24.007535Z","Summary":"A parser bug caused Cloudflare edge servers to return memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data.","Description":"On February 18, 2017, Google's Project Zero reported a security vulnerability in Cloudflare's edge servers. This bug, present in Cloudflare's HTML parser, caused servers to return memory containing private information such as HTTP cookies, authentication tokens, and HTTP POST bodies. While the bug had been latent for years, it became active with the rollout of certain features, with the earliest potential leak date being September 22, 2016. The period of greatest impact was from February 13 to February 18, 2017.\n\nThe issue stemmed from a latent buffer overrun bug in Cloudflare's Ragel-based HTML parser. The root cause was a pointer error, specifically a missing `fhold` in an error handler, combined with an equality check (`==`) instead of a greater-than-or-equal check (`\u003e=`) for buffer boundaries. This allowed the parser to read past the end of a buffer. The bug was exposed when the introduction of a new parser, `cf-html`, subtly changed the internal buffer handling within NGINX, causing `last_buf` to be set to 1 in specific scenarios, which triggered the vulnerable error path.\n\nThe leaked memory could contain sensitive customer data, including HTTP headers, POST data, JSON for API calls, URI parameters, cookies, API keys, and OAuth tokens. Cloudflare customer SSL private keys were not affected, but an internal private key used for machine-to-machine encryption was leaked. During the peak impact period (February 13-18, 2017), approximately 1 in every 3.3 million HTTP requests through Cloudflare potentially resulted in memory leakage. Additionally, some of this leaked data was cached by search engines.\n\nCloudflare quickly mitigated the issue by disabling three features that used the vulnerable parser chain: email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites. The initial mitigation took 47 minutes from receiving the report, with global resolution achieved within 7 hours. Cloudflare also collaborated with Google, Yahoo, and Bing to identify and purge 770 unique URIs containing leaked memory from search engine caches before public disclosure.\n\nThe incident highlighted how a latent bug in older software could be exposed by new system interactions. As a result, Cloudflare's infosec team initiated a project to fuzz older software for other potential security problems. They also added explicit pointer checks to generated code to prevent future buffer overruns and implemented logging for such errors."}