Many companies today came pretty close to needing to implement their DR plans yesterday, and many of them probably didn’t even realize it. In case you didn’t see what was going on in the Dallas area yesterday there was massive hail and several tornado’s toucheing down in the area. The Dallas Fort Worth (DFW) airport was shutdown for hours, hundreds of millions of dollars worth of airplanes were damaged, many homes were destroyed, etc.
What does all this have to do with companies DR plans? Well in the DFW area there is a little hosting company called RackSpace. RackSpace hosts a large percentage of their customers in the data center in the area which they actually call DFW (granted many companies in the Dallas area refer to that office DFW). In the case of RackSpace however the facility really is close to DFW, very close. In fact it’s at the end of Runway 13R/31L at the North/West end of the runway. The red mark at the top left is the RackSpace CoLo facility, the road at the bottom is runway 13R/31L.
So why I am picking on RackSpace right now? Because if you look at this map you’ll see several tornadoes which touched down not all that far away from RackSpace, just a few miles away in fact. This was a very close call.
If those tornadoes had touched down just a few miles to the west there would be a lot of companies would be in a really bad state at the moment as they try and figure out just how much data had been lost between the last tapes to be shipped from RackSpace to offsite storage and when the place was torn apart. Then there’s the problem of how long it’ll take RackSpace to get new servers delivered and racked in another data center (as it’ll probably take a while to get this one dried out and rebuilt).
Best case is that these companies would be looking at several days of downtime, worst case is weeks. The reality of the situation is that most of the smaller companies would be totally hosed as odds are that RackSpace would be focused on getting their largest clients online first, as the 30 largest clients probably bring in more revenue than the rest combined (I’ve done work for several of RackSpace’s larger clients so I know how much they are paying). Given that the major computer companies can only produce so many servers at a time, and RackSpace would pretty much need all of them for a couple of months as RackSpace would need probably thousands of servers and storage arrays to be delivered in order to get everything back up and running.
What would make this even worse is that companies that tried to move to another hosting provider to try to get online faster probably wouldn’t be able to. First they’d need to get their data from tape at RackSpace which would be a problem unto it self as there wouldn’t be anywhere for RackSpace to restore the data. Secondly the new hosting providers may not have been able to get new hardware delivered as RackSpace would be taking up all the production capabilities.
Now this hell could all be avoided by properly planning for this sort of disaster hitting the RackSpace hosting facility. RackSpace has several other data centers in the states that you could easily enough setup some DR machines at another facility and setup data replication between the facilities so that if one facility was taken offline you would be able to keep running at the second site.
But again this all requires planning this in advance. If you are a RackSpace customer I’d recommend talking to your sales team about getting a DR solution up and running within another of the RackSpace facilities.
If you need assistance with these conversations feel free to engage me and we can make sure that your systems are prepared for the next disaster that strikes near (or on top of) your data center.
Denny
