Using IPsec with Exchange

The front-end and back-end topology that Exchange Server 2003 and Exchange 2000 Server offer lets you scale your Exchange infrastructure by separating the machines that clients communicate with from the (usually larger) machines that store the mail data. However, in a typical front-end/back-end setup, the front-end server is outside the corporate network boundary, often in a demilitarized zone (DMZ), which means that the front-end server and back-end server will likely be communicating across a trusted network boundary. Unfortunately, Exchange doesn't support the use of the Secure Sockets Layer (SSL) protocol to secure this traffic. You can work around this limitation by using a firewall that can do SSL bridging (such as Microsoft Internet Security and Acceleration—ISA—Server), but that solution isn't always practical. As an alternative to SSL bridging, you can use IP Security (IPSec) to secure your Exchange network for free.

Understanding IPSec
IPSec is a set of extensions to the basic IP technology that we use for Internet communications. IPSec operates at the transport layer, so applications don't need to be aware of whether IPSec security is in effect. That approach is a major advantage over SSL, which—as an application-level protocol—requires that the application on each end know about the protocol. Another IPSec advantage is that it's made up of two separate but complementary protocols:

• The Authentication Header (AH) protocol adds a cryptographic authentication header to each IP datagram on a secured connection. The AH protocol calculates and inserts a digital signature into the packet between the original IP datagram header and the packet's payload. This approach lets the packet be routed without losing the AH data; non­IPSec-capable devices think that the AH data is part of the payload. AH provides tamper-proofing, but no confidentiality—an attacker can still read AH-protected traffic in transit.
• The Encapsulating Security Payload (ESP) protocol provides confidentiality and integrity checking. ESP uses one of two modes to encrypt the datagram's contents: In tunnel mode, packets are protected to enable connections to two separate networks; transport mode provides end-to-end security between a client and a remote network. We'll use the transport mode with Exchange; you can use tunnel mode to establish IPSec-protected VPNs.

You can use the AH and ESP protocols in conjunction with each other or independently; each protocol also supports several cryptographic algorithms. Two IPSec-capable computers begin communication by using the Internet Key Exchange (IKE) protocol to exchange cryptographic keys. The computers then negotiate to find an algorithm and key length that they both support. This process establishes a secure channel—called a security association (SA)—which protects traffic between the two machines.

Using IPSec Between Front-End and Back-End Servers
Before you begin using IPSec, you need to open several ports to provide end-to-end connectivity between your front-end and back-end servers. You must open UDP port 500, which IPSec key exchange uses for the IKE protocol. If you want to use AH for front-end and back-end communication, open IP protocol 51; for ESP, open IP protocol 50. If you're using multiple firewalls, be sure to open the ports on both sides and in both directions. Note that when you open the ports on the firewall, you need to specify the source and destination addresses of your front-end and back-end servers.

When you use IPSec for front-end and back-end communications, you use Group Policy to control which types of communications a server attempts to encrypt or authenticate. You typically specify filters based on the IP address and port that traffic is going to or coming from; the Microsoft Management Console (MMC) IP Security Policy Management snap-in lets you create flexible policy rules, if you need them. Fortunately, each Windows 2000 machine has a local IPSec policy engine (simply a small version of the IPSec Group Policy mechanism) that lets you apply IPSec policies to individual machines without requiring a full-blown Group Policy deployment. It's important to keep your policies simple so that you don't make mistakes that block network traffic. If you're not already familiar with IPSec, you might want to hire an expert to help you design an appropriate policy set.

When you deploy IPSec, you can create rules that specify which protocols and ports to use for communications (e.g., protocol 50 with port 80) and whether they require or merely permit IPSec. This process can be daunting, but you can simplify it by keeping in mind the following two caveats:

• You can tell the front-end servers to initiate IPSec connections only to the back-end servers, not to any other servers. Kerberos and IPSec don't mix particularly well in Win2K, and protecting communications between the front-end servers and the Global Catalog (GC) isn't strictly necessary, although it's possible.
• The back-end servers can accept IPSec requests they receive from the front-end servers, but the back-end servers don't need to initiate outbound IPSec traffic because the back-end server will never initiate communications to a front-end server on its own.

Given this advice, you can easily build an IPSec policy that does what you need. How you target these policies will vary according to how your front-end servers and back-end servers are separated. If they're in separate organizational units (OUs), bear in mind that you can't apply IP security policies to an OU, only to a domain. For most applications, applying IPSec policies to the individual front-end servers and back-end servers makes the most sense because you'll have relatively few servers.

Choosing the Right Policy
Win2K includes three IPSec policies that you can use out of the box:

• The Client (Respond Only) policy specifies that the target machine can respond to IPSec SA negotiation requests but not initiate them on its own.
• The Server (Request Security) policy instructs the server to request an IPSec negotiation when connecting to another machine. However, if the negotiation fails for some reason, the connection can proceed without IPSec, leaving traffic on that connection unprotected.
• The Secure Server (Require Security) policy forces the server to require IPSec. A machine with the Secure Server policy assigned requests an IPSec SA; if negotiation fails, the Secure Server machine drops the connection. This policy enforces your security settings, but it can severely limit which machines the policy target can talk to.

Although you can create your own policies, doing so is outside the scope of this article. Fortunately, the predefined policies work quite well for most needs. In our case, we'll apply the predefined Client (Respond Only) policy to the back-end server so that the server uses IPSec when a connection requests it. For the other half of the equation, we must create a new policy to tell the front-end server to use IPSec when communicating with the back-end server.

Applying Back-End Policies
You use the IP Security Policy Management snap-in to apply the policy to the back-end server. Begin by opening the snap-in:

1. Launch MMC.
2. Open Console, Add/Remove Snap-in, and click Add.
3. In the Add Standalone Snap-in dialog box, scroll down until you find the entry for IP Security Policy Management. Select it, then click Add.
4. In the Select Computer dialog box, click Another computer and enter the name of the back-end server to which you want to apply the policy. Click OK.
5. Click Close, then click OK.

In the left pane of the IP Security Policy Management snap-in, navigate to and select the back-end server to which you want to assign a policy. The right pane displays the three default policies. Right-click Client (Respond Only) and select Assign. The value in the Policy Assigned column should be Yes; if it isn't, the snap-in hasn't applied the policy. The most common cause of this problem is that the IPSec Policy Agent service isn't running. The service must be running at all times to assign policies or communicate with IPSec. You can restart the IPSec Policy Agent from the Control Panel Services applet.

Applying Front-End Policies
You have to create a new IPSec policy to force the front-end server to initiate IPSec traffic with the back-end server. This policy is easy to create because the front-end server always uses port 80 to pass traffic to the back-end server, and we know the back-end server's IP address. To create the policy, launch the IP Security Policy Management snap-in as I outlined earlier, but this time, target it toward the front-end server. Right-click the IP Security Policies On Local Machine node (the name will obviously be different if you've targeted a remote machine). Select Create IP Security Policy to launch the IP Security Policy Wizard. Proceed through the wizard, giving your new policy a meaningful name and accepting the default response rule. The real magic comes when you create a rule to secure the front-end and back-end traffic. In the wizard's last screen, make sure that you clear the Edit Properties check box before you click Finish. Under the Rules tab of the new policy's Properties dialog box, click Add to start the Security Rule Wizard. Create the rule by following these steps:

1. The Tunnel Endpoint screen is useful only if you're using IPSec in tunnel mode. This policy doesn't use tunneling, so select This rule does not specify a tunnel and click Next.
2. The Network Type screen lets you specify whether this IPSec connection applies to remote access requests, LAN traffic, or all connections. In this case, we're encrypting traffic across the LAN (or at least the portion of it in the DMZ or perimeter network), so select the Local Area Network (LAN) option before you click Next.
3. The Authentication Method screen lets you choose how the two endpoints will authenticate each other. Whenever possible, you should use Kerberos, which is the default method; it's the most secure and the easiest authentication method to administer. Select Windows 2000 Default (Kerberos V5 Protocol), and click Next.
4. The IP Filter List screen contains two default choices: All ICMP Traffic and All IP Traffic. Unfortunately, neither choice works for our situation: We want to encrypt traffic only on port 80, not all IP traffic (remember, no Kerberos over IP, and our front-end server must be able to send authentication requests to the domain controller—DC). Accordingly, you must create a new filter by clicking Add, which opens the IP Filter List dialog box.
5. Give the new filter a description and a name, then click Add to start the IP Filter Wizard. Skip the wizard's first page by clicking Next.
6. The IP Traffic Source screen of the IP Filter Wizard lets you specify the source address for the filter rule. In our case, the source for the front-end server's rule is its IP address, so make sure the Source Address drop-down list has My IP Address selected, and click Next.
7. The IP Traffic Destination screen specifies the filter rule's endpoint. The default setting matches any IP address as the destination. We want our rule to match only the IP address of the back-end server, so select A Specific IP Address from the drop-down list, then fill in the back-end server's IP address and click Next.
8. In the IP Protocol Type screen, select TCP as the protocol. If you accept the default value of Any, you can't specify the port you want to use because the rule then attempts to secure all IP traffic. You probably don't want to secure all IP traffic, particularly if your front-end server is running SMTP.
9. If you select TCP as the protocol type, the wizard opens the IP Protocol Port screen. Select the From Any Port and To This Port options, then enter "80" in the destination port field. Click Next, then click Finish.
10. When you return to the IP Filter List dialog box, click Close to return to the Security Rule Wizard. Select the IP filter list you just defined, and click Next.

In the Filter Action screen, select Require Security, then click Next. In the wizard's final screen, click Finish. Click Close on the New IP Security Policy Properties page to return to the IP Security Policy Management snap-in.

Finishing Touches
Next, you'll want to test the IPSec connection. First, check that the front-end server and back-end server can communicate; second, verify that the communication is secure. The easiest way to perform these checks is to open an IMAP or POP session to the front-end server, although opening an OWA session will also work. If you can log on to your mailbox, you know that communication is working properly. If not, the most likely culprit is either that you've specified the wrong policy on the back-end server or bobbled one of the steps for setting up the front-end-to- back-end rule. Double-check the rule settings against the steps I described earlier to ensure that all is well.

To verify that IPSec is running, you can use the Ipsecmon tool from the Microsoft Windows 2000 Server Resource Kit. Ipsecmon returns a list of active SAs; you can use this list for the front-end server and back-end server to verify that the SAs exist and that they're using the AH or ESP combination that you've specified.

Applying IPSec to your front-end/back-end communication path is simple (and, dare I say it, fun). IPSec will add a degree of security by protecting your front-end/back-end traffic against eavesdropping and tampering. However, securing the front-end-server-to-back-end-server link is only a part of your overall security strategy; you need to pay proper attention to other areas (including using SSL with OWA, proper password policies, and good patch management) to get the best possible security out of your Exchange environment. IPsec adds a small performance overhead, but the extra security is well worth it.