IPSec Tunneling with ISA Server

Use ISA Server to build a better VPN

Some research companies, notably IDC, predict that nearly 100 percent of Internet traffic will be encrypted by 2005. Although a portion of this traffic will consist of credit card transactions, pretty good privacy (PGP)–encrypted email, and encrypted file transfers, most will be protected because of the increasing use of VPNs—between company sites, business partners, or the office and employees' homes. Many companies that use Microsoft Internet Security and Acceleration (ISA) Server 2000 as a firewall and proxy server ask me whether they can also use the product to establish an Internet VPN. The product's robust and useful wizards can indeed help you quickly establish a client-to-gateway or gateway-to-gateway VPN. Still, setting up an ISA Server VPN involves many steps as well as Certificate Services, DHCP, DNS, and RRAS, so the process is more complex than just setting up a firewall.

VPN Tunneling
If you've never set up a VPN, the way in which they work can seem a bit awkward. First, you must use hardware or software to establish endpoints—one or more VPN clients and a VPN server. You typically can establish a VPN's physical and data-link layers (i.e., Layers 1 and 2 of the Open System Interconnection—OSI—model) over a dial-up line or a high-speed dedicated digital line. The endpoints don't need to use the same vendor's solution, but they must use the same tunneling protocol. Most VPN implementations use Layer Two Tunneling Protocol over IP Security (L2TP/IPSec), as the sidebar "ISA Server VPN Protocols," page 2, explains, and can partner with one another, but interoperability conflicts still abound, especially for ISA Server VPNs. (Most vendors' L2TP implementations vary at least slightly, like different dialects within a language.) The key to interoperability is that both endpoint solutions must support the same IPSec protocol and configuration options. (See the Web-exclusive sidebar "IPSec Protocols and Modes," http://
www.winnetmag.com/windowssecurity, InstantDoc ID 40596, for a discussion of IPSec protocols.)

You can configure a client-to-gateway VPN between a client and a server, or you can configure a gateway-to-gateway (aka site-to-site) VPN between two or more VPN network endpoints (servers or clients). ISA Server's two most common Internet VPN scenarios are between a Windows PC client and an ISA Server (client-to-gateway) and between two ISA Servers (gateway-to-gateway).

Clients at either end of a gateway-to-gateway VPN maintain their own identities and traffic domains, but remote clients on a client-to-gateway VPN become virtual hosts on the VPN server's network. The process of establishing a VPN connection often assigns these clients new IP addresses, subjecting the clients to the same traffic as physical hosts on the network. Therefore, remote clients often lose their local networking services and might not be able to connect to their local servers, printers, or Internet services. This type of configuration is known as a tunnel-mode VPN; remote clients in a tunnel-mode VPN have access only to network servers and printers and often access the Internet through the VPN. You can configure your DHCP server or RRAS to assign valid network IP addresses to remote clients. With the first method, RRAS gets a range of addresses from the DHCP to pass to the VPN clients; with the second method, RRAS assigns the addresses from its own DHCP server pool. If you need to pass advanced DHCP scope options to the remote clients, you must set up a DHCP relay agent on the RRAS computer. I suggest that you use a dedicated subnet for your VPN clients so that you can easily distinguish between VPN and LAN clients.

Alternatively, split-mode VPNs give remote clients simultaneous local and network access. Split-mode clients, however, can become unsecured gateways and can introduce rogue traffic into the VPN. Some VPN clients, including the Windows Network Connection VPN client, support both tunnel and split modes. The mode you choose depends on your objectives and your VPN solution.

The clients on either side of a VPN transmit unencrypted information; the VPN endpoints—typically routers, firewalls, or ISA Server systems, as Figure 1 shows—perform all encryption. The application in which the user created the data is usually unaware of the VPN. The VPN hardware or software decrypts the encapsulated data at the other end of the VPN, so if someone were to capture the encapsulated traffic, the most that person could read would be the IP header, and even that header usually isn't the original. IPSec protects TCP and UDP headers, source and destination addresses, and packet payload content.

To support L2TP/IPSec, you might need to open certain ports on any involved firewalls. You'll probably need to open UDP port 1701 for L2TP traffic, UDP port 500 for Internet Key Exchange (IKE) traffic, and UDP port 4500 for Network Address Translation Transversal (NAT-T) traffic if you use NAT-T. (See the Web-exclusive sidebar "NAT Transversal," http://www.winnetmag.com/windowssecurity, InstantDoc ID 40597, for more information about NAT-T.) Configuring IPSec connections on ISA Server creates the appropriate inbound and outbound packet filters for these ports, although you might need to open additional ports for additional client access (e.g., DHCP, DNS, NetBIOS). You'll probably also need to permit IP Type 50 Encapsulating Security Payload (ESP) packets and IP Type 51 Authentication Header (AH) packets. Also be prepared to use preinstalled machine certificates or to install certificates. (See the Web-exclusive sidebar "Certificate Authentication," http://www.winnetmag.com/windowssecurity, InstantDoc ID 40598, for information about certificates.) Now that you have a bit of background about how client-to-gateway and gateway-to-gateway ISA Server VPNs work, let's look at the steps involved in setting up these popular VPNs.

Client-to-Gateway VPNs
Employees working from home, laptop users on the road, and Internet-based Terminal Services users are common users of client-to-gateway VPNs between a remote Windows client system and an ISA Server system. If the ISA server is a member server, remote clients can use established domain user accounts; if the ISA server is a standalone server, you must create user accounts on the ISA server for each VPN user.

After you've created any necessary user accounts, you must configure the ISA server to accept VPN client connections, then configure the clients to connect to the ISA server. On the ISA server, perform the following steps:

  1. Use the Certificate Import Wizard or Microsoft Management Console (MMC) Certificate Store snap-in to install the ISA Server X.509 machine certificate (see "Certificate Authentication" for more details). To use the Certificate Import Wizard, open the MMC Certificates snap-in and right-click the Personal Certificate object. Select All Tasks, Import to start the wizard. Click Next, browse to or type the machine certificate filename, then click Next. Choose Automatically select the certificate store based on the type of certificate, click Next, then click Finish. A message indicating that certificate import was successful will appear.
  2. Open the MMC ISA Management snap-in.
  3. Expand the ISA Server object and right-click the Network Configuration node. Select Allow VPN client connections to start the Local ISA VPN Wizard. Simply click Next, then click Finish, and the wizard automatically adds 128 inbound connections to RRAS.
  4. If you haven't already started RRAS, you'll receive a prompt to do so; click Yes.

A VPN connection to an ISA server from a remote client is actually two connections. The first connection establishes the physical link (usually to the client's ISP); the second connection establishes a link to the ISA server. To configure a Windows XP or Windows 2000 VPN client, follow these steps on the remote client system:

  1. Install the client's X.509 certificate.
  2. Select Settings, Network and Dial-up Connections from the Start menu.
  3. Click the Make a New Connection icon to start the Network Connection Wizard.
  4. Choose Connect to a private network through the Internet on the Network Connection Type screen, then click Next.
  5. Enter the remote VPN endpoint's IP address or domain name on the Destination Address screen, then click Next.
  6. Choose the appropriate selection on the Connection Availability screen, then click Next.
  7. Enter the name for the VPN connection, select the Add a shortcut to my desktop check box if desired, then click Finish.
  8. The Virtual Private Connection dialog box appears; click Properties.
  9. On the Networking tab, which Figure 2 shows, choose Layer-2 Tunneling Protocol (L2TP) from the Type of VPN server I am calling drop-down list. Click OK to return to the Connection dialog box.
  10. To test the new VPN connection, type in the user's assigned logon name and password (you might be prompted to complete the user's ISP logon procedures first). Perform a ping test to validate the VPN connection, then run Ipconfig /all to verify that DHCP, DNS, WINS, and the default gateway IP addresses are set correctly.

For clients running earlier Windows versions, you must install and configure the Microsoft L2TP/IPSec VPN Client. Installing this client also installs a new remote access device, called RASL2TPM, on Windows NT systems and installs the Microsoft L2TP/IPSec VPN Adapter on Windows Me and Windows 98 systems. NT systems must run Service Pack 6a (SP6a), Microsoft Internet Explorer (IE) 5.01 or later, PPTP, and RAS. Win98 systems must run IE 5.01 or later and the DUN 1.4 upgrade. Windows Me systems must run IE 5.5 or later and the VPN networking component, which is listed as VPN network adapter under Network properties. (IE must be installed on these systems but doesn't need to be active or configured as the default browser.)

Gateway-to-Gateway VPNs
To establish a gateway-to-gateway VPN between two ISA Server systems, you must configure both systems, although a configuration file that the first (local) system creates during its setup simplifies the setup of the second (remote) system. Perform the following steps:

  1. Install the machine certificate, as I described for client-to-gateway VPNs.
  2. Open the ISA Management snap-in.
  3. Expand the ISA Server object and right-click the Network Configuration node. Select Set Up Local ISA VPN Server to start the Local ISA VPN Wizard. (Be aware that small configuration errors often cause this wizard to close immediately without saving any entered information.) Click Next.
  4. The wizard prompts you to start RRAS if you haven't already. Click Yes if prompted.
  5. The ISA Virtual Private Network (VPN) Identification screen prompts you to enter short names for the local and remote ISA Server systems. You can use the machines' NetBIOS names or type in unique identifiers. The combination of both names must be no more than 20 characters, so many people use short unique identifiers. The wizard joins the two names to identify the new VPN connection, as Figure 3 shows. Click Next.
  6. On the ISA Virtual Private Network (VPN) Protocol screen, choose the Use L2TP over IPSec option, then click Next.
  7. On the Two-way Communication screen, select the Both the local and remote ISA VPN computers can initiate communication option. Enter the IP address or Fully Qualified Domain Name (FQDN) of the remote ISA server in the first field, and enter the remote ISA Server computer's NetBIOS (i.e., flat-style) name in the second field. Click Next.
  8. On the Remote Virtual Private Network (VPN) Network screen, enter the remote network's IP address range. Click OK, then click Next.
  9. On the Local Virtual Private Network (VPN) Network screen, verify the local ISA server's IP address and the local network's IP address scheme; modify as necessary. Click Next.
  10. On the ISA VPN Computer Configuration File screen, which Figure 4 shows, enter a name and use the Browse button to specify a location for the VPN configuration file, which contains information that you'll need to configure the remote ISA server. The file's default file extension is .vpc. Be sure to password-protect the file, and store it in or copy it to a location from which the remote ISA server can easily retrieve it. Click Next, then click Finish.

Make the configuration file and the local ISA server's X.509 certificate available to the remote ISA Server computer, or copy the file and certificate to the remote system. Then, on the remote ISA server, complete the following steps:

  1. Use the Certificate Import Wizard or Certificate Store snap-in to install the machine certificate.
  2. Open the ISA Management snap-in.
  3. Expand the ISA Server object and right-click the Network Configuration node. Choose Set Up Remote ISA VPN Server to start the Remote ISA VPN Wizard. Click Next.
  4. The wizard prompts you to start RRAS if you haven't already done so. Click Yes if prompted.
  5. On the ISA VPN Computer Configuration File screen, type the filename and path to the configuration file you created on the first ISA server. Enter the configuration file's password and click Next, then click Finish.
  6. Ping each network from the other. The first ping will be a little slow as the initial security association (SA) process takes place. If the installation adds new connections (called ports) to your existing RRAS configuration, you'll need to reboot ISA Server before you see an increase in the number of connections.

Steps to Security
Opening your company's private network to Internet VPNs increases the risk of a security exploit. ISA Server versions earlier than SP1 always trust VPN clients and therefore filter nothing between the remote client and the network. Therefore, any security threat on the VPN client has a clear channel around ISA Server's firewall filtering and onto your LAN. ISA Server SP1 applies packet filtering to most VPN connections, but RRAS dial-up VPN interfaces remain unfiltered. Before you implement an ISA Server VPN, I recommend that you study the related security concerns and read the Microsoft article "How to Configure IPSec Tunneling in Windows 2000" (http://support.microsoft.com/?kbid=252735) for more details about packet filtering between VPN endpoints.

ISA Server is a popular choice for Internet VPNs; the product uses L2TP/IPSec, which is becoming the world's VPN authentication and encryption standard, to connect two networks or let remote PCs connect to a corporate LAN. ISA Server's wizards walk you through a lot of the work of setting up a VPN, but you still need to know quite a bit about ISA Server, DHCP, DNS, RRAS, and VPNs, and to maintain network security, you must take precautions to filter VPN traffic beyond the defaults. For more information about these topics, see "Related Articles."

Related Articles
You can obtain the following articles from Windows & .NET Magazine's Web site at http://www.winnetmag.com.

"What's New in Routing and Remote Access,"
June 2001, InstantDoc ID 20710

"ISA Server: Your Network's Lifeguard,"
October 2001, InstantDoc ID 22251

"Extending ISA Server," May 15, 2003,
InstantDoc ID 38738

"DHCP in Windows 2000," July 2001
Web Exclusive, InstantDoc ID 21659

"DNS and Active Directory," July 2001,
InstantDoc ID 21128

"IPSec Enhancements for Windows XP and Win2K,"
June 2003 Web Exclusive,
InstantDoc ID 39166

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.