{"UUID":"f98d88c2-febd-4e0d-a4c0-06547bfc3e65","URL":"http://www.bbc.co.uk/blogs/internet/entries/a37b0470-47d4-3991-82bb-a7d5b8803771","ArchiveURL":"","Title":"BBC Online outage on Saturday 19th July 2014","StartTime":"2014-07-19T08:30:00Z","EndTime":"2014-07-21T07:30:00Z","Categories":null,"Keywords":["bbc online","iplayer","iplayer radio","metadata","database","caching","outage","july 2014"],"Company":"BBC Online","Product":"BBC iPlayer, BBC iPlayer Radio, bbc.co.uk audio and video playback, BBC homepage","SourcePublishedAt":"2014-07-22T18:12:07Z","SourceFetchedAt":"2026-05-04T17:55:39.570956Z","Summary":"In July 2014, BBC Online experienced a very long outage of several of its popular online services including the BBC iPlayer. When the database backend was overloaded, it had started to throttle requests from various services. Services that hadn't cached the database responses locally began timing out and eventually failed completely.","Description":"On Saturday, July 19, 2014, at 9:30 AM, BBC Online experienced a serious incident affecting several key services. The initial problem involved a metadata service, which saw its database load spike, causing requests to application servers to fail. Simultaneously, a caching layer pool failed, rendering products like BBC iPlayer and the BBC homepage inaccessible.\n\nThe metadata service, comprising 58 application servers and 10 database servers, provides program and clip metadata for various BBC iPlayer applications and other BBC Online sites. The sudden increase in database load led to widespread request failures, particularly impacting older applications that did not cache metadata locally.\n\nConcurrently, a critical caching layer, which sits in front of most BBC Online products, suffered a complex failure. This failure directly impacted major services such as BBC iPlayer and the BBC homepage, making them inaccessible to users. The exact root cause of this caching layer failure was still under forensic investigation at the time of the report.\n\nRemediation efforts began with prioritizing the restoration of the caching layer. Significant remedial work was undertaken on the metadata service throughout Saturday afternoon and evening. While some services returned to a \"walking wounded\" state, allowing close to normal operation for much of the site, BBC iPlayer remained unavailable on several devices.\n\nThe services continued in a degraded state through the rest of the weekend. A full restoration was planned for Monday morning, July 21st, to avoid further risk during Sunday evening's peak usage. The incident resulted in reduced availability for users to watch or listen to programs within the normal seven-day catch-up window, particularly for content aired on July 12th and 13th."}