This month, I'll discuss some common questions about Microsoft Exchange Server's availability, fault tolerance, disaster recovery, and maintenance. I welcome your feedback about these topics and others you encounter day to day in the trenches of Exchange Server administration and support.
To what extent does Exchange Server support clustering?
In Exchange Server 5.5, Enterprise Edition (Exchange 5.5/E), Microsoft added the capability to cluster Exchange Server on Microsoft Cluster Server (MSCS). In brief, MSCS technology lets you join two Windows NT Server computers in a resource partnership called a cluster. These two computers (called nodes) cooperatively manage a set of shared resources, such as network names, IP addresses, and disk resources. When either node fails, the other node automatically takes control of the failed node's resources and makes them available to clients.
Future versions of MSCS will let more than two nodes participate in a cluster. In addition, application software will be able to use distributed processing capabilities, thus providing scalability as you add nodes to a cluster. (For a detailed explanation of clustering in Exchange, see Tony Redmond, "Exchange 5.5/E and Microsoft Cluster Server," Windows NT Magazine, February 1998, and http://www.microsoft.com/exchange/guide/papers/clusters.asp?A=2&B=6.)
How do I install Exchange 5.5/E in an MSCS environment?
To set up Exchange 5.5/E in an MSCS environment, you must create an Exchange Server resource group before you create the resources and install Exchange Server. In the following instructions for creating the resource group, I assume that you have already configured MSCS.
- Start Cluster Administrator and select the disk group that Exchange Server will use.
- Right-click the disk group to display the context-sensitive menu.
- Rename the group to Exchange and then create the Exchange group's three required resources: Physical Disk, IP Address, and Network Name.
Now you can begin installing Exchange Server. The installation process automates Exchange service and resource clustering. You must run the installation program first on the node you've designated for the Exchange resource group. The installation senses the MSCS status and configuration and installs the program and database files to the proper cluster locations. Next, run the setup program on the other cluster node. Again, the installation process senses the cluster configuration and updates only the necessary Registry settings and program files that Exchange will need to support the failover from the failed node to the live node. When you run setup on the second node, the setup dialog box presents another option, Update Node. Selecting this option will complete the Exchange Server clustered setup.
What is circular logging, and how do you configure it in Exchange Server?
One benefit of Exchange Server's transacted database storage architecture is the transaction log. To enhance performance and reliability, Exchange Server writes server transactions first to a sequential log file and then to the Exchange database. A log file contains a copy of every transaction written to the Exchange Server database. You can replay this copy against the database in case of a crash.
All Exchange Server transaction log files are a uniform size (usually about 5MB). When Exchange Server's active transaction log (which is always named EDB.LOG) reaches the 5MB limit, Exchange creates a temporary log file, closes the current log file, renames it with a numbered file name (e.g., EDB0001.LOG), and renames the temporary log file to EDB.LOG, which becomes the current log file. Each log file has a signature that matches the database file on the server. If these signatures don't match, the Exchange Server JET (or ESE97 in Exchange 5.5) database engine will fail to start.
To let the database process track which transactions Exchange has committed to the database, Exchange uses a mechanism called the checkpoint. Exchange creates a checkpoint file called EDB.CHK in the Exchange Server MDBDATA directory for the Information Store. (Note: Exchange also creates a checkpoint file for the Directory Service in the Exchange Server DSDATA directory.) Whenever Exchange commits a transaction to the database, Exchange updates the checkpoint file to reflect the last committed transaction.
During recovery of an Exchange Server, the Information Store scans EDB.CHK to find the last transaction that was committed to the Exchange Server database. Exchange Server then searches the transaction logs for transactions that have not been committed to the database and commits these transactions (or writes) to the database. This activity is automatic when you start Exchange Server and when you restore the server from a backup. If EDB.CHK doesn't exist, the ESE or JET engine can read through the transaction logs and determine where to begin the recovery process.
Circular logging is a very important feature that significantly affects Exchange Server's disaster recovery. Enabling circular logging causes Ex-change to overwrite the oldest log files instead of creating a new one. For example, when the EDB.LOG file reaches its 5MB limit after filling five stored log files (EDB0001.LOG, EDB0002.LOG, etc.), Exchange renames EDB.LOG to EDB0001.LOG and overwrites the original EDB0001.LOG.
Circular logging reduces the number of transaction logs by eliminating a continuous buildup of transaction log files in the MDBDATA directory. The effect on disaster recovery is that with circular logging enabled, incremental and differential backups cannot occur. By default, Exchange Server enables circular logging; therefore, if you want to perform incremental or differential backups, you must use Exchange Administrator to disable circular logging (on the Advanced Server Properties page for the Information Store or the Directory for each server). Microsoft recommends that you disable circular logging for all production Exchange servers. For servers functioning as routing and replication servers, disabling circular logging is less important.
How does Exchange use write-back caching?
To satisfy most client read requests to the Exchange Server Information Store, Exchange Server uses the Information Store buffer pool, a data structure that resides in memory. When a client requests a particular database element, such as a message text, folder, or attachment, the Information Store attempts to satisfy the request from the Information Store buffer pool before requesting the data from disk and incurring I/O overhead. This mechanism bypasses the NT system cache and provides substantial performance improvements for Exchange Server.
For write activity, Exchange Server uses a transacted mechanism that first applies the database write to the transaction log. This mechanism lets you recover the database or roll back transactions, depending on I/O success. The Exchange Server database engine then applies the writes to the database file (via the NT storage mechanisms) by moving through the transaction log, committing the transactions, and moving a pointer (the checkpoint) to identify the last transaction committed to the database.
Theoretically, if a write transaction to the database residing on disk is not successful, the Information Store must roll back the transaction and make the necessary alerts and log entries. Read and write activities occur in 4KB blocks, or database pages. A database engine must verify each page to ensure that a robust and recoverable database remains available to all clients.
On servers that use a hardware-based RAID controller with write-back caching, NT posts the Exchange Server engine's database write transactions to a write-back cache that the controller manages. The controller accepts responsibility for completing the write to the database and notifies the disk driver and operating system that it has completed the write. This process allows high-performance processing of disk write requests to the write-back cache and alleviates the need for the disk driver or OS to wait for the write operation to be completed to the slower mechanical rotating disk media. Even today's 10,000rpm drives can't match the speed of a write-back cache with 60nanosecond RAM. All these actions greatly improve the speed of write activity to the Information Store.
Why is using write-back caching on RAID disk controllers with Exchange Server controversial?
Although Exchange can survive a hardware or software event that causes the system to fail, I have seen cases in which the Exchange Server Information Store or the Directory Service has become corrupt. Microsoft Knowledge Base article "XADM: EDB Event ID 23 Error 1018" (http://support.microsoft.com/support/kb/ articles/q151/7/89.asp) discusses possible causes of Exchange Server database corruption.
This article suggests that write-back caching on hard disks and hardware-based RAID controllers is one possible cause for corruption of the Exchange Server database. The article offers no hard evidence or documentation for its claims. Microsoft Product Support Services wrote the article to address concerns about enabling write-back caching on hard disks and never intended to recommend against using hardware-based RAID controllers that use write-back caching.
In practice, hard disks that most server vendors ship with their products already have the write-back caching function disabled. Without the benefit of Error Checking and Correction (ECC) RAM or battery backup, write-back caching on hard disks is a dangerous feature to implement. Unfortunately, recommendations from Microsoft based on the Knowledge Base article have alarmed users about the use of write-back caching on hardware-based RAID controllers with Exchange Server.
The uncertainty around this issue stems from the confusion between write-back caching on hard disks and write-back caching on hardware-based RAID controllers. In short, don't enable write-back caching on hard disks. For hardware-based RAID controllers, write-back caching is safe if you have a controller with properly designed data integrity features.
What features does a RAID controller need for write-back caching?
A RAID controller must have a conservative design that includes ECC memory and battery-backup protection. You need to implement the cache memory architecture with some ECC mechanism that ensures protection of the cache data in the event of a RAM component failure. For example, the Compaq SMART-2/DH Array controller implements its cache in a RAID5 memory configuration. With parity information, the 20MB cache yields 16MB of usable cache.
In addition to protecting memory for the cache, a hardware-based RAID controller that supports a write-back cache must also protect that cache from power failure or interruption. In case of power loss, the controller must have battery power available to maintain posted writes that might be in the cache. When you restore power, the controller must ensure that it has applied the writes in the cache to the disk subsystem.
Other desirable features for RAID controllers include the ability to dynamically (i.e., while the server is online) configure the cache for read vs. write and the ability to designate which volumes to cache. Highly desirable is the ability to remove the cache with data intact (via a daughtercard) in the event of a controller failure, and to install the cache on a new controller and still maintain data integrity. In my group's performance testing, we have achieved the best results when we've configured controller cache for 100 percent write-back and cached only the Exchange Server Information Store volume.
Vendors that manufacture hardware-based RAID controllers contend that no evidence shows that these controllers cause database corruption with any Microsoft BackOffice products, including Exchange Server. These vendors recommend that companies use the technology to its fullest capability, but companies must follow the proper design criteria. Be careful, however, because not all hardware-based RAID controllers support a conservative design. Some use cache sizes as large as 64MB with no ECC or power protection. A thorough understanding of your hardware and of how Exchange Server interacts with it will reduce confusion surrounding this complex issue.
What is an easy way to change the location of my Exchange Server data files (i.e., .LOG, Message Transfer Agent—MTA, Directory Service, and Information Store files)?
You can move your Exchange Server files in several ways. You can manually shut down Exchange Server, edit the Registry, and move the files. Or you can reinstall Exchange Server and select other locations for the files. You can also use the Exchange Administrator program to modify the Database Paths property page in the Server Properties dialog box. The simplest and most error-free method I've found is to use the Exchange Performance Optimizer to move files.
You usually associate Exchange Performance Optimizer with the installation of Exchange Server, because the Exchange setup process launches Optimizer. Optimizer analyzes your hardware configuration and considers your Exchange Server topology and environment to adjust the parameters to run Exchange optimally for your hardware configuration. Exchange Server sets threads, buffers, and Registry settings according to how you define your environment. In addition, Optimizer tests disk reads and writes to determine optimal locations for Exchange Server's files and directories. By default, you answer the questions, and let Optimizer complete its work. However, you can use Optimizer as a handy tool for server maintenance.
Optimizer has a management, or verbose, mode that lets the Exchange Administrator view and edit the parameters that Optimizer configures. Executing Optimizer (Perfwiz.exe in the \EXCHSRVR\bin directory) from the command line with the -v switch enables the verbose mode, which lets you configure the location of the Exchange Server files and directories. (If no additional volumes are on the server, the Optimizer won't show an option to change Exchange Server file locations.)
You can input locations for the Information Store, Directory Service, Information Store log files, Directory Service log files, and MTA files. Optimizer then relocates the files to the locations you specified and restarts the Exchange Server services.
