While preparing for a major disaster is what should be done ahead of time, that’s a little late for some people on the east coast of the US today. So if you find yourself (or your company or your servers) on the east coast, what are some things you can do today to help keep things running if things go very, very bad.
On the SQL Server side of things…
- Verify your most recent full backup by restoring it to another server (both user and system databases)
- Make sure that your backups are copied to another servers or USB Drive
- Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
- Generate some scripts to ensure that you can restore the databases to another machine when ready
- A way to get the transaction log backups off site, maybe FTP’ed to a personal website (after encrypting the files)
On the Hyper-V / VMware side of things…
- Do you have backups of critical VMs? If not take them now.
- Get a copy of the backups out of the facility (a USB drive is a great way to do this, the network will work if needed)
- Don’t forget to do a system state backup of Active Directory
If there is a disaster, the things that you’ll need at the new data center will include…
- Servers (get your order in as soon as you know that your data center doesn’t exist any more, you won’t be the only one ordering servers)
- Switches, Routers, etc.
- Lots of USB Hard Drives (The SAN won’t be available, and it’ll take weeks or months to get a new one)
- Zip ties/cable ties
- Network cables
- All the backups that you took from above
- Installation media for all your OSes and databases servers
- As lots of things
As you are getting the installations all setup, some things to keep in mind…
- You’ll want to enable Instant File Initialization
- You’ll want to restore system databases, then user databases
- Restore AD from the system state backup, otherwise all the Windows authentication SIDs will be useless within SQL Server
If your data center looses power and it’ll be out for a while, and you don’t have a big expensive generator to run the data center know where you can get a generator. A few small ones from Lowes, Home Depot, etc. will work as well for keeping the most critical systems online. Get some large box fans to keep the air in the data center moving. If you are using small generators, use a different one for the fans as they will have very dirty power, which could damage the power supplies of the servers. Plastic sheeting can be used to direct air in or out of the data center so that you get cool air coming in from outside and so you can direct the hot air back out (ask me some time about turning a 6 story building into a giant chimney for a data center).
Once we are in a disaster situation we are no longer worried about peak performance, we are worried about keeping the business running for our customers. Don’t worry about redundant power supplies, or passive nodes, reporting servers, etc. Just worry about getting the core systems online so that websites work, and so that customers can login to your applications. If your application supports it be sure to put up a message telling your customers that performance won’t be very good because of the weather. Just because you know that the weather is a mess doesn’t mean that your customers are aware that there’s a problem. Most people are pretty understanding and if you tell them that the weather is causing performance problems they will be fine with that. This will save you a lot of phone calls and a lot of headache.
No matter what, when getting things setup, rebuilt, brought back online, etc. do it safely. Don’t get hurt and risk hurting yourself or others to get things back up and running. It just isn’t worth it.
Denny
