Around May 4th, 2019, most Firefox add-ons stopped functioning due to an expired certificate. The incident was triggered when a certificate used for signing add-ons expired, preventing Firefox from validating their authenticity.

The root cause was identified as a misunderstanding within the team responsible for the signing system, who incorrectly believed Firefox ignored certificate expiration dates. This misconception was exacerbated by a previous incident where end-entity certificate checking was disabled, causing confusion regarding intermediate certificate validation. Furthermore, Firefox’s QA plan lacked testing for certificate expiration or future date behaviors, preventing early detection of the problem.

To remediate the issue, a fix was delivered via the Studies system (internally known as Normandy), which injected a valid certificate. This method was chosen for its speed, despite requiring some users to enable Telemetry, leading to temporary over-collection of data. Over the subsequent weeks, numerous fixes and dot releases were deployed to address various deployment targets and defects in initial patches.

Lessons learned from the incident include the need for improved communication and documentation regarding system components, better integration of this information into engineering and QA processes, and the development of a faster, independent update mechanism not tied to Telemetry or the Studies system. The importance of dedicated QA resources during incident response was also highlighted.