Postmortem Index

Explore incident reports from various companies

Northeast blackout of 2003

FirstEnergy / General Electric · XA/21 energy management system

2003-08-14 – 2003-08-16 cascading-failure hardware security

The Northeast blackout of 2003 began on August 14, 2003, just after 4:10 p.m. EDT, affecting parts of the Northeastern and Midwestern United States and most of Ontario, Canada. The incident started with local failures in the FirstEnergy system, where transmission lines contacted untrimmed foliage. This led to a cascading failure across the regional electricity distribution system.

The primary technical cause was a software bug, specifically a race condition, in General Electric Energy’s Unix-based XA/21 energy management system used by FirstEnergy. This bug stalled the control room’s alarm system for over an hour, rendering operators unaware of critical system changes and the need to redistribute electrical load. Contributing factors included FirstEnergy’s failure to adequately manage tree growth and assess system vulnerabilities.

The widespread outage affected an estimated 55 million people, including 10 million in Ontario and 45 million in eight U.S. states. Customer impact included disruptions to transportation (Amtrak, airports), communication (overloaded cellular networks), and water supply (loss of pressure, boil advisories). The incident occurred during high temperatures, increasing energy demand and exacerbating the initial strain on power lines.

Most affected areas saw power restored within hours, with some as early as 6 p.m. on August 14. However, full restoration for major cities like New York and parts of Toronto took until August 16. A joint U.S.-Canada task force investigated the incident, highlighting FirstEnergy’s operational failures and the lack of enforceable reliability standards in the U.S. at the time. The event underscored the need for substantial investment in grid reliability.

Keywords

blackoutpower outagefirstenergygeneral electricxa/21alarm systemnortheastontario