Postmortem Index

Explore incident reports from various companies

Healthcare.gov launch failure

Centers for Medicare & Medicaid Services (CMS) · Healthcare.gov

2013-10-01 – 2013-12-31 cloud automation

Healthcare.gov went live on October 1, 2013 and was immediately non-functional. About 2.8 million visitors attempted to access the site on day one; 6 of them completed registration. Users typically saw “The system is down at the moment. Please try again later.” For the first week the site stayed effectively unusable through a combination of crashes, errors and extreme slowness.

The most acute capacity bottleneck was the identity-verification subsystem, which had been load-tested for roughly 2,000 concurrent users and was hit by tens or hundreds of thousands at once. Databases locked up, network connections saturated, and front-end servers were overwhelmed. Page loads exceeded 8 seconds; some registration steps took over 60 seconds. Insurers receiving downstream enrollments saw duplicate records and incorrect applicant data due to data-integrity glitches. A pre-launch government memo had warned of “high” security risk; the site shipped with known vulnerabilities anyway. Independent assessments — including a McKinsey report — and contractor warnings flagging roughly 45 critical and 300+ severe defects in the final weeks were not acted on, in favor of meeting the statutory October 1 launch date.

Contractually the project was split across more than 60 contracts and 33 vendors with no single systems integrator. CMS, which owned the program, did not have in-house capacity to lead delivery at this scale and did not designate a lead contractor. Leadership was distributed across multiple agencies and committees, leaving no single accountable owner for the end-to-end system.

The “tech surge” — a team of senior industry engineers brought in shortly after launch — fixed more than 400 defects over six weeks, raised concurrent-user capacity to roughly 25,000, and reduced page latency. By late December 2013 the site was functioning acceptably and year-end enrollments reached nearly one million.

Initial development was budgeted at roughly $300 million. An August 2014 HHS Inspector General report put website cost at $1.7 billion; Bloomberg later estimated total cost across all related systems at about $2.1 billion. In response, the federal government created the U.S. Digital Service and 18F to provide in-house digital-delivery capacity.

Keywords

healthcare.govacacmsidentity verificationload testingtech surgeu.s. digital service18f