Your Exchange servers are talking behind your back! Every day they send messages to one another to announce public folder replication, keep the directory up-to-date, and swap Address Book Views (ABVs). Under ideal circumstances, you don't notice these system messages. However, when your network connection is slow, congested, or unreliable, system messages can wreak havoc with message delivery. If you have a troublesome connection, you might want to reduce the noise. Let's see how you might reduce the volume of system messages in a large Exchange Server organization.
Observe the Problem
Your company, Bodge-it & Scarper (B&S), is rolling out a 51-site Exchange Server 5.5 organization. Each site has one server, and the company's sites are spread over a large geographical area. You've decided to use a hub-and-spoke topology with the hub situated at corporate headquarters. You must fit Exchange Server around B&S's existing WAN, whose connectivity ranges from T1 to a Public Switched Telephone Network (PSTN). You must plan for public folder replication from several sites to all other sites, for use of ABVs, and for updating each server's directory daily.
Halfway into the rollout, you begin to notice inexplicably large volumes of Directory Service (DS) and public Information Store messages in several Message Transfer Agent (MTA) queues, they have a backlog of DS and IS messages. These queues appear only on connectors to sites that have slow or unreliable network connectivity and occasionally grow to thousands of messages, most of which users didn't generate.
You try several fixes, such as using the Mtacheck utility to clear the queues so that the network can keep up with the number of messages, but the queues quickly build up again. (See the Microsoft article "XCON: When and How to Use the Mtacheck Utility" at http://support.microsoft.com/ support/kb/articles/ q148/2/84.asp for more information about Mtacheck.) The number and frequency of these messages appear to increase with the number of sites added to B&S's Exchange organization.
You try setting the public folder replication schedule to the troubled sites to a lower frequency and verifying that the directory replication schedule is set to once a day for 1 hour. Still, the queues grow, full of system messages. Users at these sites start to complain. You have to fix the problem quickly before your manager starts grumbling.
Analyze the Situation
Your first step is to understand why, how, and where the system is sending messages. You need to look at message connectors, directory replication connectors, public folders, ABVs and any activity that involves starting and stopping an Exchange service, such as performing offline backups and running the Mtacheck utility.
Connectors. B&S uses Site connectors for branches with good connectivity (i.e., permanent connections at 64Kbps or faster), X.400 connectors for branches with dial-on-demand connectivity of ISDN quality, and Dynamic RAS (DRAS) connectors to the branches with only dial-up modem connectivity to headquarters. You find problems most frequently on the DRAS connectors and a few heavily used ISDN lines.
Directory replication. The company set the directory replication schedule for 5:00 a.m. to 6:00 a.m. Monday through Sunday on all spoke servers and 4:00 a.m. to 5:00 a.m. Monday through Sunday on the hub server. This replication sequence is good practice because the hub requests updates from the spokes before the spokes update their directory from the hub; therefore, this sequence keeps all spoke directories consistent with one another within one replication schedule.
However, setting the replication schedule for more than one 15-minute interval isn't good practice. During the directory replication schedule, the DS creates two directory update request messages for every site in the organization every 15-minute replication period and sends the messages to the server's bridgehead server. One message requests an update to the configuration-naming context (e.g., for changes to connectors); the other message requests an update to the site-naming context (e.g., for changes to mailboxes). The bridgehead servers reply to these messages by sending all the directory information that's new since the spoke last requested and successfully received this data. With these settings, this schedule creates as much network traffic as completely syncing the directory four times a day.
Public folders. B&S has several public folders that Exchange replicates to all sites. The company uses the default public folder replication schedule (i.e., Always), which means that if a user changes the contents of a public folder, Exchange replicates the changes within 15 minutes. Because the company's folders aren't time-critical, this replication schedule is probably too frequent—does the next thread in the company's joke folder really need replicating every 15 minutes?
In addition to sending bona fide replication messages, Exchange produces public folder hierarchy status messages at least once every 24 hours, at either 12:15 a.m. Greenwich Mean Time (GMT) or 12:15 p.m. GMT, depending on the time you started the IS service. Every server that has a public IS sends a message containing the complete public folder hierarchy to every other server that has a public IS. These messages inform sites about public folders that users have created elsewhere in the organization but which Exchange hasn't necessarily replicated to all the sites.
ABVs replication messages. B&S uses ABVs to structure the Global Address List (GAL) to better suit the organization. ABVs carry a replication overhead. In Exchange Server 5.5, every server requests an update to its ABVs context by sending one message to and receiving one reply from its bridgehead server every 6 hours. In a hub-and-spoke environment, this process results in the hub sending four messages to and receiving four messages from every spoke and each spoke sending and receiving four messages every 24 hours.
Mtacheck. Because queues build up on various connectors, you occasionally run the Mtacheck utility on the hub and several of the troublesome spoke sites by entering
C:%exchange install directory%\bin\mtacheck.exe /rd /rp
The /rd switch clears directory replication messages, and the /rp switch clears public folder messages. Clearing out these excessive system messages can make transferring user messages over the troublesome links a little easier. However, although Microsoft suggests using Mtacheck as a way to clear out queues, it doesn't recommend using the utility as a regular maintenance tool.
A result of Mtacheck's deleting DS messages is that the destination's and the sender's directories are out of sync until the next directory replication occurs. However, the lack of synchronization generally isn't a serious problem.
Deleting public folder status messages is also relatively harmless in this environment. However, deleting public folder replication messages can cause Exchange to produce more messages in response to the deleted messages than Mtacheck deleted in the first place. For example, when B&S replicates its personnel guide from the human resources (HR) department's Exchange server to all other (i.e., 49) spoke sites, the system sends one replication message to the hub. The hub fans the message out to each individual server, as a message to a distribution list (DL) would fan out, for a total of 50 messages (1 + 49). If you delete the initial replication message, each receiving server eventually figures out that it has lost a message somewhere. Each server then sends a backfill request message (an extra 49 messages) to the originating server, which sends an individual reply message to each requesting server (another 49 messages). Therefore, the net effect of deleting the initial replication message is an extra 48 (98 - 50) messages that the system would not have otherwise created. For more information about this topic, see the Microsoft article "XADM: Repercussions of Deleting Public Folder Replication Messages" (http://support.microsoft.com/support/ kb/articles/q169/2/05.asp).
Stopping services. B&S backs up several servers in its Exchange organization offline because the branch systems administrator prefers to use a non-Exchange-aware backup program. The problem with offline backups is that every time you stop the DS or IS service, the services create system messages when they restart. The IS sends two public folder status messages to every other server with a public folder store in the organization. The DS sends the same number of messages as occur during one 15-minute cycle of the directory replication schedule.
Table 1 shows the number of system messages the hub and spokes sent and received in 7 days under the old system. The figures assume that the company didn't reboot any servers or restart the DS and IS services on any machines.
Reduce the Number of Messages
As Table 1 shows, B&S's original configuration can cause Exchange Server to generate an unacceptably large number of system messages. Your analysis has identified the primary sources of those messages. Here's what you can do to reduce the number of messages by about 80 percent.
Directory replication schedule. First, change the directory replication schedule on the hub server to one 15-minute interval on weekdays only. Scheduling replication for after working hours (e.g., at 8:00 p.m.) or during a quiet period (e.g., lunchtime) keeps directory replication traffic from adding to user traffic. Next, change all spoke directory replication schedules to one 15-minute interval, again on weekdays only (e.g., at 9:00 p.m. Monday through Friday). These simple changes instantly save the network and servers from transferring 234,600 directory replication messages every week, with no significant loss of functionality.
Registry key modifications. You also can modify several Registry keys to control the frequency and type of system messages. All the Registry keys I mention here are REG_DWORD type values, and the value names are case-sensitive. Warning: Be sure that you have a recent backup of your Registry before you modify it, or you might have to reinstall your system.
Add the HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\ MSExchangeDS\Parameters\Replicator inter site sync at Startup Registry key, and set the value to 0. This setting disables the creation of directory replication messages when the DS starts and makes rebooting less stressful for servers running Exchange Server.
Reduce the production of status messages to once a week by creating the HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\ MSExchangeIS\ ParametersPublic\Replication Send Status Timeout Registry key and setting the value to 604,800. This value represents the minimum number of seconds Exchange Server waits before sending a status message to the organization. You can choose as a value the amount of time you want to wait for Exchange to replicate changes to the public folder hierarchy throughout your organization.
Also, stop the IS from creating status messages on service startup. Create the HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\MSExchangeIS\ParametersPublic\ Disable Replication Messages at Startup Registry key, and set the value to 1. Stop and start the IS to force the changes to take effect. This change has the same effect on the public IS as the DS Registry key change does on the DS.
These Registry changes apply to Exchange Server 5.5 Service Pack 2 (SP2). The changes you can make depend on your Exchange version and service pack. Table 2 shows the minimum version and service pack for each change.
Finally, change the replication schedule of the public folders to more suitable intervals, depending on the content. To change the public folder replication schedule for all folders, go to the Advanced tab on the server's Public Information Store property sheet and enter the appropriate number of minutes in the Replicate always interval (minutes) window. To set the replication schedule folder by folder, open the Microsoft Exchange Administrator program and navigate to the appropriate folder. On the Replication Schedule tab of the folder's Properties page, choose Selected times and mark the replication times you want on the schedule, as Screen 1 shows. Modifying the schedule by folder is more work, but it lets you adapt the replication schedule to the importance of the folder's content. For more information about working with public folders and their schedules, see Mark Ott, "The Care and Feeding of Public Folders," June 1998.
ABVs. Update ABVs only once a day. Change the replication schedule to run from 10:00 p.m. to 10:15 p.m. Monday through Friday on the hub and from 11:00 p.m. to 11:15 p.m. Monday through Sunday on all the spokes. To make this change, run Exchange Administrator in raw mode. From a command line, type
%exchange install directory%\bin\admin.exe /r
and then complete the following steps. Select Address Book Views, File, Raw Properties. In the Object Attributes window, select Period-Rep-Sync-Times. (If this attribute isn't visible, change the List attributes of type: to All.) Click Editor, select Schedule, click OK, and then select the appropriate Schedule. Be sure to click Set before OK when you close the Raw Properties dialog box to save your changes. (To learn more about ABVs, see Mark Ott, "Using Address Book Views in Exchange Server," January 1999.)
Lean and Mean
As you see in Table 1, after you apply the changes, the company's Exchange Server organization has 254,500 fewer system messages on its network every week, excluding the extra messages that the system would have produced on service startup. In a network environment where connecting to a remote site costs money by the second, minimizing intersite traffic becomes very important. As these examples show, in a fairly large Exchange Server organization, a few Registry entries and schedule changes can save time and money. Even in an organization with few sites, several of these changes could be worth making. Companies don't need to have a server running flat out dealing with thousands of directory replication requests and responses that aren't needed when the server could be doing something more useful instead.
