Restoring an Exchange Server Computer

Last month, I wrote about backing up your Exchange Server computer. I discussed client-based versus server-based backup; which files you back up; backup types and strategies; and online, offline, and third-party backup applications. This month, I'll describe the restore process. I'll present several recovery scenarios, give you some disaster-recovery tips, and talk about a Microsoft utility that can prevent Exchange Server from automatically deleting Exchange objects after the restore process. Let's get started.

Server Restore
Restores come in several flavors, depending on what's crashed and what type of backup you have (offline or online). You might encounter any of these situations:

• Both Windows NT and Exchange Server are down as a result of catastrophic drive failure.
• NT is OK, but the Exchange Server system is in a failed state.
• NT is OK and log files are intact, but disk drive failure has destroyed database files.
• A user's PC hard disk failure has resulted in the loss of all the messages in the user's mailbox.

I'll walk you through the third scenario. Assume that NT is functional, but the disk where the \exchsrvr resides has failed. You have a single-site, single-server organization; you've disabled circular logging; and the transaction log files are on another hard disk that is still intact. After you've replaced the bad hard disk, you discover that the only backup tape you have is the Exchange online backup tape from yesterday. The backup type is normal (and you don't have a current NT backup tape of the entire server).

Your goal is to restore from the backup tape and then roll forward the transaction logs from yesterday's last backup to the time of the hard disk crash. Ideally, this procedure will rebuild the server up to the point of failure with almost no message loss. Here are the steps for restoring the Exchange Server system.

1. Run the Exchange Server setup routine with the Setup/r syntax. Setup, which is on the Exchange Server CD-ROM, copies over blank DIR.EDB, PRIV.EDB, and PUB.EDB files and doesn't start any services. Not starting the services prevents users from processing messages until you've completed the restore process. The /r switch also requires that you use a backup tape for the restoration.

   At the prompt, select Create a new site instead of Join an existing site. You can't join an existing site that has multiple servers, because other servers in the site don't know you've crashed, and you'll receive an error message saying that the computer name already exists.

2. Input the same Organization name, Site name, and Site Service account as before the crash. This information must be identical to the failed server's information so it will be a mirror image. You enter the information as a response to the Exchange Server installation wizard. Remember that the Organization name is case-sensitive.

   Later in the installation process, Exchange Server invokes the Performance Optimizer. During my test restoration, Performance Optimizer displayed several error messages, similar to the message you see in Screen 1; however, I ignored the messages with no ill consequences. Apparently, the optimizer wants to confirm that you have valid .edb files (database files for the Directory Service and Information Store), which I didn't have. The message appears because the /r switch creates skeleton .edb files, which the restore tape later fills with messages.

3. Restore the Information Store and Directory databases from the backup tape. Insert the normal backup tape, and from NT, select Start, Programs, Administrative Tools, Backup. Click on the Tapes window, and then Operations; choose Catalog and then the databases you want to restore, as Screen 2 shows. To my surprise, I discovered that the catalog doesn't select the Information Store by default.

   After you select both the Directory and Information Store backup sets, a dialog box similar to the one in Screen 3 lets you specify restore information. I think this screen is poorly designed, because it shows only the information for the Directory database and doesn't instruct you to scroll down to access the Information Store database screen, which Screen 4 shows. Another unintuitive step is that the Information Store window requires you to input a server name, but a pop-up window will appear if you don't enter a server name. I strongly recommend that Microsoft consider adding a wizard to make the restore process goof-proof.

   Select Verify After Restore and Start Service After Restore on both screens. Note that Exchange Server doesn't initialize the other Exchange services (e.g., the Message Transfer Agent or the System Attendant); you must start them manually after the restore. Also, request a Full Detail log file, so you can see exactly which files copied to the server. If you have both the log files and the database files on the same partition, don't check Erase all existing data, because you don't want to delete all the log files on the hard disk. Exchange Server plays back transaction logs from the backup tape and any transaction logs on the system's hard disk.

   After the restore, you will have a complete database, including all messages up to the crash. Note that if you enable circular logging, you can restore only up to the last backup tape.

4. Check the backup log. When Exchange has completed the restore process, look at the backup log, which you see in Figure 1, in the \\Winnt folder to check that the backup has copied the .edb file, then the transaction logs, and finally the .pat files.

5. Confirm that the restore is complete and clean. You need to confirm that the databases are complete by checking that mailboxes are functioning properly and that public folders still have their original postings. Also check the Global Address List (GAL) for completeness. Finally, verify all gateways and connectors by sending outbound mail messages.

   In offline backups, one step is different. After the restore, you need to run the ISINTEG -PATCH command to prepare the Information Store before you start the services. This step is necessary to replace globally unique identifiers (GUIDs) so that any new objects that users create in the Information Store don't conflict with other objects' GUIDs in the organization. Like NT security IDs (SIDs), each GUID must be unique. Because Exchange online backups capture the GUIDs, this step is unnecessary for online backups.

Disaster Recovery Tips
To improve your chances for a full recovery from a crash, you must store the Exchange database files on a different hard disk from the transaction logs. The Exchange Performance Optimizer places the transaction logs on the fastest hard disk it can find and, if possible, on a different partition from the .edb databases. However, this utility can't see physical partitions. Therefore, if you have two physical disks (and no disk striping) and the faster disk is split into two partitions (e.g., C and D), Performance Optimizer might place the .edb files on one of those partitions and the transaction logs on the other partition of the same physical disk and not place any files or logs on the other drive (E). This placement is undesirable, because both the databases and transaction logs are on the same spindle. If the disk crashes, you can restore only from the last backup and you lose all transactions between the last backup and when the crash occurred.

You can place these two files on different physical disks by rerunning Performance Optimizer and manually changing the paths, as you see in Screen 5, page 4, to store .edb files on a different disk. Then, if that disk crashes, you need only run the restore tapes and restore up to the last transactions recorded before the failure.

Here are some other disk administration tips:

• Install NT on a separate hard disk. If the Exchange Server hard disk crashes, you can concentrate on recovering just Exchange Server and not both NT and Exchange Server.
• House the database files on a hardware RAID 5 unit. That way, failure of one disk won't shut the stores.
• Ensure that ample free space is available when you run the defragmentation option of the EDBUTIL or the ESEUTIL program. You need at least twice as much disk space as the size of the database file you are defragmenting. The TEMPDFRG.EDB file will use this space to rebuild the original .edb. By default, the space allocated is on the same partition as the original .edb file, but you can reallocate the space with the /t switch.
• Because Exchange Server 5.5 has a virtually unlimited store size, backups can be time-consuming. Have the fastest available hard disks­usually SCSI hard disks­and state-of-the-art tape units to keep I/O rates from bogging you down.
• In Exchange Server 5.0, you can save time if you compact the Information Store before you use the EDBUTIL utility to back it up. However, Exchange 5.5 is more efficient in recovering deleted pages in the store and performs the function automatically. In either version, if you manually compact the PRIV.EDB or PUB.EDB files (using EDBUTIL in Exchange Server 5.0 or ESEUTIL in Exchange Server 5.5), you must shut down the Information Store service before you use the utility. If you shut down the Information Store service, you might want to shut down all the Exchange services and perform an offline backup of all partitions on the system. When the system is down, you can back up both NT and all Exchange Server databases at one time.
• Format your tapes before backing up. Most folks assume that because the tapes they buy are formatted, the tapes are ready to go. However, even if your tape drive doesn't encounter any errors during the write process, it might not be able to read the tapes during the restore. By formatting your tapes with the tape device hardware you're using, you're more likely to have a successful restore.

Microsoft has provided its customers with information about recovering Exchange. Two excellent white papers are MS Exchange Disaster Recovery Part I and Part II in TechNet, February 1998. These papers are required reading for any backup operator.

Finally, don't forget that Microsoft has a GUI-based command scheduler that can perform offline backups automatically at specified times. Microsoft introduced this utility (WINAT) in the NT 3.51 resource kit. You need to create a batch file to shut down the Exchange Server services before the backup routine runs. Part I of the disaster recovery white paper describes this process in detail.

Another Backup Tool
Sometimes you might need to restore a server because of operator error instead of as a result of hardware failure. For example, if the administrator inadvertently deletes objects in the Recipients container, users can't log on to Exchange because they no longer have a mailbox. In this case, you can restore the server with the populated Recipients container from an earlier backup tape and then replicate this good information from the restored server back to all the other servers. Use the Authoritative Restore tool for this restore.

Authoritative Restore (AuthRest.exe) is on the Exchanger Server CD-ROM, and you use it when the restored server is the directory database of choice. For example, if you delete data by mistake, Exchange Server will replicate this information to other servers in the site. If you later perform a restore with a tape that includes the old mailboxes, the other servers will tell the restored server to delete those mailboxes. Authoritative Restore forces the new restored server to replicate its directory to all other servers in the organization. Be careful: Don't use this utility without an experienced Microsoft consultant on hand, because little documentation about its use is available.

Practice Makes Perfect
Question: What do you call an administrator who doesn't back up? Answer: Unemployed. You can see why you must faithfully follow a well-documented disaster-recovery strategy. Microsoft has made backing up easy with tons of information to guide you in implementing a plan. However, Microsoft can't make you back up—doing so is up to you.

I was quite concerned about doing my backup and restore. I performed a simulated crash by using the uninstall routine to remove Exchange Server from my PC. The thought crossed my mind more than once that I might permanently destroy what had taken me months to fine-tune. However, nothing is more crucial than going through a trial restore. Microsoft recommends doing a mock restore on a test system to obtain experience so you'll be prepared when the time arrives for the real thing. Although the process was intimidating, I feel more confident that I could restore Exchange Server in a production environment if I had to. And remember: You don't have to go it alone. If a production system goes down, call a Microsoft Exchange consultant to work with you.

The bottom line is, how valuable is your information? If losing data would cause you pain, then back it up.