Back Pressure Changes in Exchange 2007 SP1

Microsoft introduced a new transport service in Exchange Server 2007, so it's not surprising that the Exchange developers would have learned from experience in production and used the opportunity presented by SP1 to improve throughput and reliability in this area. One of the new features related to this transport service is back pressure, a technique that lets the transport service throttle back its ability to receive and process new messages when the server experiences some sort of resource constraint, such as low disk space or memory exhaustion. When a severe back pressure condition occurs, the server stops accepting incoming messages until all pending outgoing messages are processed.

The underlying idea is that the condition that provoked back pressure is temporary and while it persists the transport service should take care of messages that are en route instead of accepting new connections. When the condition—usually low memory or disk space—eases, back pressure lifts and normal processing resumes. Given the speed that the transport service processes messages, most users are unaware that a back pressure condition occurred—and why should they care?—unless the server is severely overloaded and the condition persists for several minutes.

The TechNet article "Understanding Back Pressure" describes the feature in detail. However, apart from saying that the free disk space threshold for the disk that holds the message queue was decreased from 4GB to 500MB in SP1, it doesn't say what other tuning changes Microsoft made to back pressure settings.

Before exploring further, it's important to know that Exchange stores back pressure settings in an XML format application configuration file called EdgeTransport.exe.config in the C:\Program Files\Microsoft\Exchange Server\Bin directory. The same file is used for both Hub Transport and Edge Transport servers. Figure 1 shows a sample file.

Microsoft developers didn't design this file to be changed by administrators unless absolutely necessary, and they don't provide a UI to view or change settings in the file. An Internet search reveals a lack of documentation about what all the settings mean and the effect that a change would have. In addition to the information in the TechNet article referenced earlier, the article "How to Change the Location of the Queue Database" explains how to change the location of the message queue database. Some information about experiences with EdgeTransport.exe.config is found in blogs written by Microsoft MVPs and other technologists, but that's about the extent of the available information.

You can edit EdgeTransport.exe.config with any text editor, but you shouldn't do so on your production servers because of the high risk of making a change that has unpredictable consequences. Simple text editors don't come with error checking for application configuration files! Making changes to application configuration files is much like running the old Exchange Server 5.5 admin.exe tool in raw mode or applying changes to Active Directory with ADSI Edit: You have total control over the data, but nothing prevents you from screwing things up.

Every time the transport service starts up, it reads the configuration file to know what thresholds to use for back pressure as well as other settings to control the efficient running of the service. Using an Extensible Storage Engine (ESE) database for the message queue is one of the big architectural changes in Exchange 2007—so, just like with the Information Store, it's clearly important to have effective caching behavior for optimum performance. For example, the DatabaseCheckPointDepthMax parameter controls the total allowed size of all uncommitted transaction logs for the message queue database. Its default value is 20,971,520 bytes, meaning that the ESE database used for the message queue can have a maximum of 20MB of uncommitted transactions in memory.

You can tune the performance of the transport service by making changes to the application configuration file, which you might want to do to improve the performance of an Edge Transport or Hub Transport server that handles a lot of traffic. Microsoft fundamentally overhauled the transport system in Exchange 2007; the transport system now uses an ESE database to capture messages as they pass through Hub Transport and Edge Transport servers. This process is different from Exchange 2003 and Exchange 2000, both of which rely on writing message files to NTFS. ESE is a full-fledged database engine, so it should come as no surprise that you can tweak its performance in different ways.

Before making changes even on a test server, you should review the information from the Exchange development team in "A study of Exchange 2007 SP1 Hub throughput with different message sizes." This post gives you some basic pointers about how to approach tuning to handle the workload generated by different average message sizes ranging from 25KB to 10MB.

My experience is that the average message size in large organizations ranges from 70KB to120MB, including attachments. The wide variation is explained by factors such as corporate culture and branding. For example, if a company insists that its employees post large files in repositories such as SharePoint and send only references to the files through email rather than attachments, the company's average message size will go down. You can increase your message size by including graphically rich corporate logos in documents and autosignature files.

The average message size in 1996 was around 4KB. You can expect the size to increase over time, even if the latest document formats introduced in Office 2007 take less space than their predecessors. "A study of Exchange 2007 SP1 Hub throughput with different message sizes." contains a lot of useful pointers about how to configure disks for the Exchange 2007 transport engine and how the engine scales up as message size increases, including basic tuning steps such as increasing the size of the transport cache. The Exchange team explores this aspect of tuning more comprehensively in "New maximum database cache size guidance for Exchange 2007 Hub Transport Server role," which explains when you might need to increase the database cache size on heavily trafficked Hub Transport servers.

So if you've updated to Exchange 2007 SP1, you should be happy, right? Well, maybe. The new settings are applied when you install Exchange 2007 SP1 on a new server, but they aren't applied when you update a server from Exchange 2007 RTM to SP1. This situation is understandable because Microsoft didn't want to overwrite any customized settings that might exist in the application configuration file. However, it's not beyond the wit of developers to merge new settings into an existing file on a selective basis and to let the system administrator know what the installation procedure has done. Unfortunately, with the way things work, there are plenty of Exchange 2007 Edge Transport and Hub Transport servers in production today that seem to run SP1 but are still using the RTM settings.

Of course, the problem crops up only if you operate heavily loaded Edge Transport or Hub Transport servers that report back pressure conditions or other performance issues. Like any database, the storage used for the disk that holds the message queue database has to perform, so we'll assume you've covered the basics and moved all the files used by the transport system, including the message queue database, off the C drive or any other drive that is hammered in normal use. After that, the indications that you have transport problems are found in event log entries 15006 (critically low disk space) or 15007 (critically low memory) from MSExchangeTransport; look for events 15002 and 15003 if you haven't upgraded servers to SP1.

If scans of the event log on your Hub Transport servers don't turn up any of these entries, and if you have no reported instances that mail isn't being delivered as fast as expected, and there are no indications of other performance problems on the servers, then the settings in the transport application configuration file are probably OK and you can concentrate on other things. All of this goes to prove that when Microsoft makes improvements in a service pack and you install that service pack, you might still have to check configuration details to ensure that your servers are tuned appropriately.

Comments

Plain text