Power Outages Are A Major Risk That Most Companies Overlook

Stephanie Balaouras

TechCrunchIT reported today that a Rackspace data center went down for several hours during the evening due to a power grid failure. Because Rackspace is a managed service provider (MSP), the downtime affected several businesses hosted in the data center.

When companies think of disaster recovery and downtime, they typically think of catastrophic events such as hurricanes, tornadoes, and earthquakes. What companies don't realize is that the most common cause of downtime is power failures. In a joint study by Forrester Research and The Disaster Recovery Journal of 250 disaster recovery decision-makers and influencers, 42% of respondents indicated that a power failure was the cause of their most significant disaster declaration or major business disruption.

To prevent power failures, businesses must ensure that they have multiple diverse connections to the power grid as well as install backup power generators and uninterruptible power supplies (UPS) at the data center. But it's not enough to have these preventative measures in place, businesses must test the ability to switch over to backup power must at least twice year. And if your business has a recovery data center, it's best if the recovery data center is on a different power grid and is also equipped with backup power generation.

But despite all these measures, failures might still happen, in the case of the Rackspace power failure, the company successfully failed over to its backup power generators but some of its chillers did not start up correctly.

In North America, the risk of power failures is likely to remain high for the foreseeable future. According to a 2007 report by the North American Electric Reliability Corporation (NERC), long-term capacity margins are still inadequate and significant investment in transmission is still required.

So businesses must not only invest in preventative measures such as backup power generators, they must think about where they locate their data centers. You must avoid areas that have clearly identified congestion issues and focus on areas that have access to cheap and abundant power. And, don't take it for granted that your service provider has effectively managed the risk of power failures.


re: Power Outages Are A Major Risk That Most Companies Overlook

Hi Stephanie, I definately agree, we have just moved to having dual A+B feed power meaning that it comes from two separate sources rather than only relying on generators. I think there are however a shortage of datacentres and providers offering this...

re: Power Outages Are A Major Risk That Most Companies Overlook

Cavalcade of Risk #56 now onlineCato Institute's Michael Cannon hosts his 2nd Cav, and your post is in it:http://www.cato-at-liberty.org/2008/07/16/cavalcade-of-risk-56/Please let your readers know.And if you'd like to host a Cavalcade, and you really should, just drop us a line. It's fun and easy, and a nice traffic bump.Thanks!Hank Stern

re: Power Outages Are A Major Risk That Most Companies Overlook

Hi Duncan, I think you might find the following post interesting. It appears that some UK data center operators are planning to generate their own electricity and that companies are moving their data centers out of regionas with power capacity constraints.http://www.datacenterknowledge.com/archives/2008/Jul/17/power_shortages_constrict_uk_data_centers.html

re: Power Outages Are A Major Risk That Most Companies Overlook

While some may have a comfortable feeling knowing they have n+1 generators, a UPS, and dual power feeds this belt and suspenders approach can still fail if the infrastructure is not maintained. Batteries must be placed on a maintenance schedule and replaced regularly, transfer switches must be serviced by qualified power technicians, and bypass procedures must be documented and key personnel must know them by heart when a catastrophic power event occurs. Finally, a contingency plan is still needed to answer the question "where am I going to get a 1.2 MW generator at 3am?". Processes for mobilizing electricians and other power vendors should be documented so if all else fails, a replacement generator can be hard wired into the input of the building. Extended run times of generators stretching into weeks during a major disaster can break down the generator leading to a failure well after you think you're out of the woods.It's certainly a good step forward to install redundant infrastructure and test it twice per year, but that is but the first step in assuring truly resilient power.