Google is apologizing following what started as a minor incident at one of their data centers when a CO2 handler caught fire.
Google returned 404 errors (page not found) for over an hour in many locations around the world after a fire in one of their data centers, but not because of the fire. According Google, the fire started as only a small blaze caused by systems designed to reduce CO2 emissions. Ironically, the worst damage was caused by the fire suppression sprinklers and not the fire.
Although the sprinkler systems put out the fire and saved Google from significant losses of data and equipment, one of their routing systems was affected by the water and was unable to properly re-route the center’s web traffic to other appropriate alternative data centers. The way Google tells the story, as the replacement router was installed and powered up, the backup of traffic caused four other Google data centers to crash.
Google Fire Bad News for Advertisers
While data loss was minimized by Google’s swift reaction with a replacement router, there were reportedly many advertisers who were falsely charged for AdWords. Google expects it to take some time to determine the advertisers and the amount of lost ads.
For more information, see Google’s full statement.
UPDATE: Read additional coverage on Mobile Local Social “Google Data Center Fire Returns Worldwide 404 Errors” which reports “A fire at a Seattle data center, nicknamed The Googtopia has affected over 42,000 servers, bringing down search engine results pages across the globe.”