[Geeks are Sexy] technology news

Wednesday, August 30, 2006

Restoring Exchange Server Nightmares!

A while back, about 3 years ago, I wrote a white paper on restoring an Exchange 5.5 Server, and published it at Tek-Tips.com. Back then I used Veritas Backup Exec 8.x (Today, Symantec Backup Exec 10d is still my backup suite of choice.)

** Disclaimer: I'm in no way affiliated with Symantec, Veritas or Backup Exec! I am merely a customer who's set in using a product he's familiar with. There are other really great backup products out there such as: ArcServe, NetWorker, Retrospect Server Backup, and if you have the HUGE budget, Overland Data Neo. See options here for cost comparison and features

So, now that that's out of the way...

Ever since I wrote this white paper, I have received countless emails asking how to implement it properly. As concise as I'd like to think my white-paper is, there are so many variables and little details that can be overlooked in such a procedure, that sometimes the readers have to contact me directly. Just recently, I received a bottle of wine from a guy in Jersey for helping him restore his setup over a 3 day period (that was nice of him!).

With every email that I have received from these users, they all had made the same errors. And though the technology over the last 3 years has changed, the principals have not. So, I felt it may be helpful to post these insights here today!

If anyone has ever been a part of a Mock Disaster Recovery test with Exchange, he/she would know that the documentation provided with the products and on the subject to be incredibly confusing. Unfortunately, at the time that I wrote this White-Paper, it was not a Test - it was real life. Our Exchange server had tanked due to the fact that my apprentice at the time restored a single mailbox around 10 or 11 times without doing any sort of Logfile rollback (or just do a full back up to shrink the logs). In essence, the 20 gig volume for my logs filled up so much that the services shut down and left the server useless. We couldn't delete the logs, because we would loose data, so our solution was to restore the server to a Zero Data loss recovery. Great in theory; frustratingly difficult in reality.

The white-paper describes the issues we had (so if you read it, you will see all that had to be done), but two of the issues that I wanted to focus on were the restoration of the Directory Service (DS), and the naming of your Organizational Units within the rebuilt Exchange server. These are the two snags that all of the readers of my white-paper miss. They are small details, but they are very, very important ones, because if you miss them, restores fail and service restarts fail. I also found that these two things are not documented ANYWHERE that I could find.

Organizational Units:

It is very important to note that you must install the new server into the same state as your original server. Document all the paths to the MTA, Log Files, and to the Database files (PUB.EDB, PRIV.EDB). And make sure that the Site Name, Organizational Unit names and the Server name are typed identically to your production server. These values are Case-Sensitive (we found this out the hard way). Should you miss a capital letter anywhere in these values, the restores will not work....

....And I mean "identically"!

Directory Store - Restore procedure:

Delete everything within the DSADATA folder. This will effectively delete your current DS. This is a MUST. Without this step, upon restoration, the DS will fail to start. Then Restore a Directory Store that is OLDER than the Information Store you've just restored. Two weeks or two months, it does not matter how old it is.

This is a very important step! You must get rid of any files associated with the new installation of Exchange Server's Directory Store folder (DSAData). Allow the restore to replace it with an OLDER version of the DS (older as in, older than the Information Store). Why? Honestly, I am not 100% sure. But, I will theorize.

When the Exchange server is being backed up (a full backup), my observations have been that the Information Store gets backed up first. Usually, the Info-Store is larger than the DS and it takes alot more time to complete. And because I use ONLINE Backups (this is to say that the servers services are still running, and the Exchange Server continues to deliver mail and operate normally), the DS is still writing transaction logs and maintaining its own database during the IS backup. When all is said and done, the DS backups are always more up-to-date than the IS backups. If you restore both services with the backups from the same date, the DS service will not start citing that the logs are more up to date than the Information Store's and it fails. Hence, we came up with the idea to restore the DS using older backups, and voila it worked.

So, these are just two of the MANY confusing things that you can run into restoring an Exchange Server, arguably, one of the most imporant servers in an Enterprise. Hope you find it helpful; if not, I hope it was enlightening nonetheless!


  • Yeah, I had to go through a partial restore of our exchange 2000 server information store and it was pretty nightmarish. Strangely, the IS corrupted itself after going through an offline store defrag, so I had to restore it from arcserve and use the ESEutil utility to resync the store with the E00 log file.. It took me a while to figure it out, but it worked.. Back then (2003), there was not documentation on the net about this procedure, so you can imagine how enjoyable this experience was for me..

    Strangely, 3 years later, typing "fix information store exchange 2000 tool" in google brings me to an MS knowledgebase article (dated: September 7, 2005)that precisely tells about what needs to be done..

    If anyone ever needs to have a look at it, here's the link:

    How to recover the information store on Exchange 2000 Server or Exchange Server 2003 in a single site

    By Blogger Kiltak, at 10:51 AM  

  • It shocks me, you know, how poorly this kind of thing is documented.

    Back in the day of Exchange 5.5 - the documentation was even more obscure and typing "Information Store Restart Fails" served up 1000's of articles, none of them giving you what you needed, but all of them giving you a different answer...

    It was a frustrating time.

    By Blogger AlRo, at 11:29 AM  

  • It would behoove any oranization from 200+ exchange server environment to one exchange server with 10 mailboxes on it, to do bii-annual DR testing.
    Get a box, max it out with memory and enough hard drive space - throw Vmware or vpc on it, create vm's for your dc's and an exchange mailbox server. Configuure the network to be in a private isoated ip range so you can use the same machine names. Test your DR process.
    This is hugely important - you can run a great service that never has problems for years but all it takes is one disaster that you cannot recoover from and your organization will never trust IT again and worse, your name is mud.
    If presented right with enough "fear" but not too much over the top, you can get your management to spend money on a DR environment.

    By Anonymous Anonymous, at 8:18 AM  

Post a Comment

Links to this post:

Create a Link

<< Home