Restoring Exchange Server Nightmares!
A while back, about 3 years ago, I wrote a white paper on restoring an Exchange 5.5 Server, and published it at Tek-Tips.com. Back then I used Veritas Backup Exec 8.x (Today, Symantec Backup Exec 10d is still my backup suite of choice.)
** Disclaimer: I'm in no way affiliated with Symantec, Veritas or Backup Exec! I am merely a customer who's set in using a product he's familiar with. There are other really great backup products out there such as: ArcServe, NetWorker, Retrospect Server Backup, and if you have the HUGE budget, Overland Data Neo. See options here for cost comparison and features
So, now that that's out of the way...
Ever since I wrote this white paper, I have received countless emails asking how to implement it properly. As concise as I'd like to think my white-paper is, there are so many variables and little details that can be overlooked in such a procedure, that sometimes the readers have to contact me directly. Just recently, I received a bottle of wine from a guy in Jersey for helping him restore his setup over a 3 day period (that was nice of him!).
With every email that I have received from these users, they all had made the same errors. And though the technology over the last 3 years has changed, the principals have not. So, I felt it may be helpful to post these insights here today!
If anyone has ever been a part of a Mock Disaster Recovery test with Exchange, he/she would know that the documentation provided with the products and on the subject to be incredibly confusing. Unfortunately, at the time that I wrote this White-Paper, it was not a Test - it was real life. Our Exchange server had tanked due to the fact that my apprentice at the time restored a single mailbox around 10 or 11 times without doing any sort of Logfile rollback (or just do a full back up to shrink the logs). In essence, the 20 gig volume for my logs filled up so much that the services shut down and left the server useless. We couldn't delete the logs, because we would loose data, so our solution was to restore the server to a Zero Data loss recovery. Great in theory; frustratingly difficult in reality.
The white-paper describes the issues we had (so if you read it, you will see all that had to be done), but two of the issues that I wanted to focus on were the restoration of the Directory Service (DS), and the naming of your Organizational Units within the rebuilt Exchange server. These are the two snags that all of the readers of my white-paper miss. They are small details, but they are very, very important ones, because if you miss them, restores fail and service restarts fail. I also found that these two things are not documented ANYWHERE that I could find.
It is very important to note that you must install the new server into the same state as your original server. Document all the paths to the MTA, Log Files, and to the Database files (PUB.EDB, PRIV.EDB). And make sure that the Site Name, Organizational Unit names and the Server name are typed identically to your production server. These values are Case-Sensitive (we found this out the hard way). Should you miss a capital letter anywhere in these values, the restores will not work....
....And I mean "identically"!
Directory Store - Restore procedure:
Delete everything within the DSADATA folder. This will effectively delete your current DS. This is a MUST. Without this step, upon restoration, the DS will fail to start. Then Restore a Directory Store that is OLDER than the Information Store you've just restored. Two weeks or two months, it does not matter how old it is.
This is a very important step! You must get rid of any files associated with the new installation of Exchange Server's Directory Store folder (DSAData). Allow the restore to replace it with an OLDER version of the DS (older as in, older than the Information Store). Why? Honestly, I am not 100% sure. But, I will theorize.
When the Exchange server is being backed up (a full backup), my observations have been that the Information Store gets backed up first. Usually, the Info-Store is larger than the DS and it takes alot more time to complete. And because I use ONLINE Backups (this is to say that the servers services are still running, and the Exchange Server continues to deliver mail and operate normally), the DS is still writing transaction logs and maintaining its own database during the IS backup. When all is said and done, the DS backups are always more up-to-date than the IS backups. If you restore both services with the backups from the same date, the DS service will not start citing that the logs are more up to date than the Information Store's and it fails. Hence, we came up with the idea to restore the DS using older backups, and voila it worked.
So, these are just two of the MANY confusing things that you can run into restoring an Exchange Server, arguably, one of the most imporant servers in an Enterprise. Hope you find it helpful; if not, I hope it was enlightening nonetheless!