{"UUID":"2487d400-146b-4eba-a72a-9d715fc2128f","URL":"https://stackstatus.net/post/96025967369/outage-post-mortem-august-25th-2014","ArchiveURL":"https://web.archive.org/web/20220928194609if_/https://www.stackstatus.net/post/96025967369/outage-post-mortem-august-25th-2014","Title":"Stack Exchange Network outage due to HAProxy iptables misconfiguration on August 25, 2014","StartTime":"2014-08-25T19:26:00Z","EndTime":"2014-08-25T19:32:00Z","Categories":null,"Keywords":["stack exchange","haproxy","iptables","firewall","puppet","network","outage","new york","load balancer","datacenter"],"Company":"Stack Overflow","Product":"HAProxy","SourcePublishedAt":"2014-08-28T20:53:00-04:00","SourceFetchedAt":"2026-05-04T18:12:12.299459Z","Summary":"A bad firewall config blocked stackexchange/stackoverflow.","Description":"On August 25, 2014, the entire Stack Exchange Network, including Q\u0026A sites and Careers, experienced an outage lasting approximately 6 minutes, from 7:26 pm to 7:32 pm UTC. The incident began when an incorrect network firewall configuration change was applied to the primary HAProxy load balancer in the New York data center.\n\nThe root cause was a harmful change made to the iptables configuration on the HAProxy load balancers. A misleading comment in the existing iptables configuration led to a modification that prevented HAProxy systems from completing connections to the IIS web servers, specifically blocking SYN/ACK packets for response traffic.\n\nThe change was pushed to Git at 19:01 UTC, and the outage commenced at 19:26 UTC when Puppet applied the change to the active load balancer. Staff were alerted immediately, and a revert was pushed to Git within two minutes of the outage start, at 19:27 UTC. The outage was resolved at 19:32 UTC after Puppet applied the revert to the primary load balancer.\n\nCustomer impact included a complete outage across all Stack Exchange sites for the duration of the incident. The immediate remediation was the application of a revert to the misconfigured load balancer.\n\nCorrective actions include cleaning up misleading comments in the iptables configuration. Future plans involve implementing a mechanism to apply and test changes on secondary/inactive load balancers first, converting existing static iptables configurations to use the Puppet Labs firewall module, and conducting a full audit for readability and cleanup to simplify understanding and modification of rules."}