Troubleshooting RPC over HTTP Connections

The traditional mechanism for Messaging API (MAPI) clients such as Outlook to access Exchange Server mailboxes has been to use remote procedure call (RPC) over TCP/IP connections. However, Exchange Server 2003 introduces a new mechanism: RPC over HTTP connections, which provides great flexibility for Microsoft Office Outlook 2003 clients. Using RPC over HTTP, Outlook 2003 clients can connect from virtually any network, across any firewall, provided that HTTP outbound access is available on that network.

The basic concept behind RPC over HTTP is straightforward. An Outlook 2003 client connects over HTTP to an RPC proxy server. Acting on behalf of the client, the RPC proxy server establishes RPC connections to the back-end Exchange server, which is the Exchange server that hosts the requesting client's mailbox. (If you'd like more background information about this type of connection, see the Windows & .NET Magazine article "Exchange 2003 RPC over HTTP Access," September 2003, InstantDoc ID 39770.)

Although the basic concept behind RPC over HTTP connectivity isn't difficult to grasp, this new form of access introduces new chances for connections to fail. To help you troubleshoot RPC over HTTP connections, I've created a checklist that targets the common causes of failures in RPC over HTTP connections. The items in the checklist fall into five broad categories:

Check the prerequisites.
Check the connection type.
Check the client's configuration.
Check the servers' configuration.
Check for connectivity.

Check the Prerequisites
RPC over HTTP connectivity has a number of prerequisites, and making sure that these prerequisites are met is the first task when you're troubleshooting an RPC over HTTP connection. The first prerequisite is that the client PC must be running Outlook 2003 and Windows XP Service Pack 2 (SP2) or XP SP1 with the appropriate patch—Quick Fix Engineering (QFE) patch 331320—which resolves a problem with client delays. (To learn more about the problem and its patch, see the Microsoft article "Outlook 2003 Performs Slowly or Stops Responding When Connected to Exchange Server 2003 Through HTTP" at http://support.microsoft.com/?kbid=331320.)

To make sure you have the correct OS patch version, you can check the version of the rpcrt4.dll file in the \%windir%\system32 directory on the client PC. In Windows Explorer, find the rpcrt4.dll file, right-click the file, and choose Properties. In the Properties dialog box, click the Version tab, then select File Version in the Item name list. You need to have version 5.1.2600.1142 or later.

The second prerequisite is that the RPC proxy server must be running Windows Server 2003, as must any domain controllers (DCs) or Global Catalog (GC) servers that the RPC proxy server will use to authenticate the client connection. In addition, any GC servers to which the user might be referred need to run Windows 2003. The back-end Exchange server or any other Exchange servers (e.g., public folder servers) must also be running Windows 2003 and Exchange 2003. So, the simplest rule of thumb is to make sure your entire infrastructure is using Windows 2003 and Exchange 2003.

Check the Connection Type
The goal of RPC over HTTP connectivity is to make connections between Outlook 2003 and Exchange 2003 as seamless as possible to the end user. That is, the user should detect no difference between an RPC over MAPI connection and an RPC over HTTP connection. For this reason, if Outlook 2003 can't make an RPC over HTTP connection, it automatically fails over to the TCP/IP protocol and tries to make an RPC over MAPI connection.

Although this behavior is desirable during the routine operation of the Exchange system, it's undesirable when you're first attempting to configure RPC over HTTP access. Because the failover is completely transparent, you won't know whether your RPC over HTTP connection worked or whether it failed and you're connected through an RPC over MAPI connection.

When you're on a client that has Outlook 2003 already running, there's a simple trick you can use to determine how the client is connected to the Exchange server. While holding down the Ctrl key, right-click the Outlook icon in the system tray and select Connection Status. This action yields a Connection Status dialog box that identifies the current connection type in use.

When you need to configure an RPC over HTTP connection, you can suppress this automatic failover feature by editing the registry on the Outlook 2003 client PC. In the HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\RPC subkey, create the DisableRpcTcpFallback entry. Set DisableRpcTcpFallback to the DWORD value of 1 to disable RPC/TCP fallback. After you know the system is running properly, set DisableRpcTcpFallback to 0 to allow failover.

Check the Client's Configuration
Microsoft has two implementations of RPC over HTTP—Version 1 and Version 2—that have different capabilities. The RPC over HTTP functionality implemented with Outlook 2003 uses Version 2, which requires that the client use Secure Sockets Layer (SSL) communication and that the RPC proxy server authenticate the connection. Thus, you need to check the configuration of the client MAPI profile to make sure that it's configured to use SSL communication and it accurately identifies the RPC proxy server.

To check these configurations, open Outlook 2003. On the Tools menu, select E-mail Accounts. After you make sure the View or change existing e-mail accounts option is selected, click Next. Select the appropriate account, then click Change, More Settings. In the Microsoft Exchange Server dialog box that appears, select the Connection tab and make sure the Connect to my Exchange mailbox using HTTP check box is selected.

Next, click Exchange Proxy Settings to open the dialog box that Figure 1 shows. Make sure the Mutually authenticate the session when connecting with SSL check box is selected. When this check box is selected, you must add the prefix msstd: to the RPC proxy server's name in the Principal name for proxy server field.

Note that the first point of contact for the Outlook client might be a local HTTP proxy server, such as Microsoft Internet Security and Acceleration (ISA) Server 2000. If so, make sure that this server (and not the RPC proxy server) is accurately identified in the Exchange Proxy Settings dialog box. In addition, you need to verify that the local HTTP proxy server is correctly directing connections to the RPC proxy server. In the case of ISA Server 2000, you can check the connections by inspecting its log files.

Below the Principal name for proxy server field, note the On fast networks, connect using HTTP first, then connect using TCP/IP check box and the On slow networks, connect using HTTP first, then connect using TCP/IP check box. By default, these check boxes are selected to provide the failover discussed earlier. Slow connections are those connections with a link speed of 115Kbps or slower.

Besides checking the configurations in the client MAPI profile, you need to verify that the Outlook 2003 client can indeed connect to the RPC proxy server using SSL communication. You can make this verification by opening Microsoft Internet Explorer (IE) and connecting to https://RpcProxyServer
/RPC, where RpcProxyServer is your RPC proxy server's name. Receiving the error message HTTP Error 403.2 Forbidden: read access is denied means you made a successful connection to the RPC Virtual Directory in Microsoft IIS on the RPC proxy server. (The error message displays because, although the client has the permission necessary to connect to the RPC Virtual Directory, it doesn't have the permission necessary to read that directory.)

Check the Servers' Configuration
In "Exchange 2003 RPC over HTTP Access," I describe the fundamentals of server configuration for RPC over HTTP access, so I won't repeat that information here. Instead, let's look at the biggest server-side culprits that prevent RPC over HTTP connectivity from working correctly:

Incorrectly configured ValidPorts entry. On the RPC proxy server, the ValidPorts entry in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\RpcProxy subkey must identify all the Exchange servers, DCs, and GC servers with which the RPC proxy server communicates. If you used the RPCHTTP_Setup.vbs script, which is available from Microsoft Product Support Services (PSS), to set up your RPC over HTTP connections, all the host names and ports should be correct. However, if you didn't use RPCHTTP_Setup.vbs, you need to check the host names and ports. In addition, make sure that the RPC proxy server can resolve all the host names specified in the ValidPorts entry and make sure the certificate presented by the RPC proxy server is valid.
Incorrectly configured back-end Exchange server. The back-end Exchange server's required registry settings are set automatically during the Exchange 2003 installation, so these settings shouldn't be the cause of any configuration problems. However, you need to make sure that the back-end Exchange server's port settings are in agreement with the port settings in the RPC proxy server's ValidPorts entry. In addition, be aware that Exchange 2003 SP1 introduces a new server property that lets you configure an Exchange server as an RPC proxy server. This multipurpose server is called an RPC-HTTP front-end server. Thus, if you have a separate RPC proxy server, you need to make sure that the RPC-HTTP back-end server option (and not the RPC-HTTP front-end server option) is selected. You can access this new property through Exchange System Manager (ESM). Simply right-click the Exchange server, select Properties, and click the RPC-HTTP tab.
Incorrectly configured DCs or GC servers. If the RPC proxy server communicates with any DCs or GC servers, the DCs' or GC servers' registries must be manually updated to add the NSPI Interface Protocol Sequences entry under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters subkey. This addition is a likely candidate for configuration errors, so you need to make sure it's accurate on each DC or GC server. In addition, you must make sure that the GC servers' port settings are valid and in agreement with the port settings in the RPC proxy server's ValidPorts entry.

If you're experiencing connection problems and you found that all the servers' configurations are correct, try rebooting the servers. Although rebooting might sound trite, I have observed situations in which it solved the problem.

Check for Connectivity
You need to make sure that communication is possible between the RPC proxy server, the back-end Exchange server, and the GC servers. The most powerful tool available for troubleshooting RPC over HTTP connections is the RPC Ping utility, which is available from the Microsoft Windows Server 2003 Resource Kit. Outlook 2003 also provides RPC tracing functionality to help identify connectivity problems.

RPC Ping. RPC Ping is a sophisticated tool that has a myriad of switches and qualifiers that you can use to validate whether RPCs can reach the specified destination (e.g., a server) over the specified HTTP connection. You can find a lot of useful information about RPC Ping in the Microsoft article "How to Use the RPC Ping Utility to Troubleshoot Connectivity Issues with the Exchange Over the Internet Feature in Outlook 2003" (http://support.microsoft.com/?kbid=831051).

You can use RPC Ping to conduct several tests. First, you want to use it to confirm that the Outlook client can make an RPC over HTTP connection to the RPC proxy server. To do so, use the command

rpcping -t ncacn_http
  -s ExchangeServer
  -o RpcProxy=ProxyServer
  -P "username,domain,*"
  -I "username,domain,*"
  -H 1 -u 10 -a connect -F 3
  -v 3 -E -R HttpProxy

(Although this command appears on several lines here, enter it on one line in the command-shell window. The same holds true for the other multiline commands in this article.) Admittedly, this command is complex, so let's take a look at each argument:

The -t argument specifies one of three protocol sequences: ncacn_http, ncacn_ip_tcp, or ncacn_np. For RPC over HTTP connections, you need to specify ncacn_http.
The -s argument specifies the back-end Exchange server, where ExchangeServer is that server's name.
The -o RpcProxy= argument specifies the RPC proxy server, where ProxyServer is that server's name.
The -P argument specifies the account to be used to authenticate with the RPC proxy server. To provide the necessary information, you can use one of two formats: "username,domain,*" (if no password is required) or "username,domain,password" (if a password is required), where username is the user's name, domain is the user's domain, and password is the user's password.
The -I argument specifies the account to be used to authenticate with the Exchange server. Again, you use either the "username,domain,*" or "username,domain,password" format to provide the necessary information.
The -H argument specifies the RPC proxy server's authentication scheme. You can specify 1 for NT LAN Manager (NTLM) authentication or 2 for Basic authentication. For this example, I specified 1 because I was testing a network link that supported NTLM.
The -u argument specifies the Security Support Provider (SSP). You can specify 9 (Negotiate), 10 (NTLM), 14 (Secure Channel), or 16 (Kerberos). Because I used NTLM for the authentication scheme, I specified 10 for this argument.
The -a argument specifies the authentication level that is to be used to connect to the RPC proxy server. The possible values are connect, call, pkt, integrity, or privacy. For this example, I specified connect because I was testing a connection to the service.
The -F argument specifies whether to use RPC over HTTP front-end authentication flags. You can specify 2 (no SSL flag) or 3 (use SSL flag). For this example, I specified 3 because I was using an SSL connection.
The -v argument specifies the logging mode. You can choose between minimal logging (for which you specify 1), normal logging (2), or complete logging (3). I recommend complete logging because you'll get the most information.
The -E argument restricts the connectivity test to the RPC proxy server only. (Note that there's also an -e argument, so letter case is important.)
The -R argument specifies the local HTTP proxy server to be used, where HttpProxy is that server's name. If no local HTTP proxy server is to be used, you need to specify none instead of the server's name.

Figure 2 shows sample output from when I ran this command. Note the error message Error 12029 returned in the WinHttpSendRequest. Ping failed. This error message lets you know that a problem lies in the connection between the Outlook 2003 client and the RPC proxy server. "How to Use the RPC Ping Utility to Troubleshoot Connectivity Issues with the Exchange Over the Internet Feature in Outlook 2003" provides a helpful table that lists the error messages that RPC Ping might return and possible reasons for those error messages.

If you run this command and see Pinging successfully completed in the output, the RPC proxy server responded to the ping, which means the problem doesn't lie in the connection between the client and the RPC proxy server. In that case, the next step is to check the connection between the RPC proxy server and the back-end Exchange server. For this test, use the command


rpcping -t ncacn_http
  -s ExchangeServer
  -o RpcProxy=ProxyServer
  -P "username,domain,*"
  -I "username,domain,*"
  -H 1 -u 10 -a connect -F 3
  -v 3 -R HttpProxy

Note the absence of the -E switch. Because you don't want to restrict the connectivity test to the RPC proxy server, you don't include that switch.

To perform an RPC Ping against the Exchange Store endpoint, use the command

rpcping -t ncacn_http
  -s ExchangeServer
  -o RpcProxy=ProxyServer
  -P "username,domain,*"
  -I "username,domain,*"
  -H 1 -u 10 -a connect -F 3
  -v 3 -R HttpProxy
  -f Uuid,Ver

Note the addition of the -f switch. The -f switch specifies the interface to ping, where Uuid is that interface's universally unique identifier (UUID) and Ver is that interface's version number. UUIDs facilitate connections from other sources. In this case, the UUID is a4f1db00-ca47-1067-b31f-00dd010662d. If you don't specify a version number, RPC Ping uses version 1.0.

RPC Ping is an advanced troubleshooting tool that has intimidating command syntax. However, after you know which switches to use and how to use them, using RPC Ping is relatively straightforward and provides good insight into connectivity problems.

Outlook 2003 RPC tracing. To enable Outlook 2003's basic RPC tracing functionality, open Outlook 2003 and select Options from the Tools menu. Go to the Other tab, then click Advanced Options. Select the Enable mail logging (troubleshooting) check box. Close Outlook 2003, wait for a minute or so, then reopen it. From this point on, Outlook 2003 will generate events and log them in the client PC's Application event log.

More advanced RPC tracing functionality is possible if you create several registry entries and replace several DLLs on the client PC. In the registry, you must create the RpcTraceEnable, RpcStackTrace, RpcDumpToFile, and RpcOutputDir entries under the HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\RPC subkey. In the \Program Files\Common Files\System\MSMAPI\1033 directory, you need to replace the emsmdb32.dll, emsabp32.dll, emsui32.dll, and msmapi32.dll files with special versions of those DLLs. For more information about these modifications, contact PSS.

Look for the Usual Suspects
Using RPC over HTTP as a transport between Outlook 2003 and Exchange 2003 is a complicated process. For the most part, the complexity is transparent to the user, but occasionally the process doesn't run that smoothly. During the early stages of implementation, problems are generally the result of relatively simple configuration mistakes. Incorrect server information, incorrect registry entries, and unresolvable names are the usual culprits. Problems that occur after the system has been operating a while can be more difficult to diagnose. Under such circumstances, the problems are likely the result of a component in the infrastructure going out of service. If you look for the common causes that I've discussed, you'll probably have an easier time and more success with finding and implementing the solution.

Comments

Plain text