Postmortem Index

Explore incident reports from various companies

Chrome SyncDataType parsing crash

Google · Chrome Sync

2012-12-10 – 2012-12-11 config-change

On December 10, 2012, Google Chrome users experienced widespread browser crashes. Initially, these crashes were mistakenly linked to a Gmail outage, but investigations quickly identified the Chrome Sync service as the trigger.

The incident stemmed from a faulty load balancing configuration change within a core Google infrastructure component. This change severely impacted a quota service, which Chrome Sync servers relied upon to manage per-datatype sync traffic.

In response to the quota service’s failure, Chrome Sync servers conservatively instructed clients to throttle “all” data types. This included data types that older Chrome client versions did not yet understand or support. The client-side logic, specifically a function converting a protobuf field number to an enum, returned an UNSPECIFIED value, causing an out_of_range error when stored in a bitset, leading to the browser crash.

Customer impact was significant, with users experiencing unexpected browser terminations.

Remediation involved a swift server-side fix to prevent Sync servers from throttling unknown data types, which immediately stopped the crashes. Additionally, a client-side fix (r172232) was developed and merged into Chrome on December 11, 2012, to enhance client robustness by gracefully handling invalid specifics field numbers.

Keywords

chromesynccrashquotaload balancingclientserverparsingbrowserprotobuf2012