I recently had to perform a clean install of Microsoft ISA Server 2004. After the installation, I soon realized I had problems that only Microsoft could help me fix. After several hours of continuous Microsoft technical support and various configuration adjustments, I was able to work out the kinks. Here's a recap of the problems I encountered and the changes that were necessary to fix them:
- LAN connectivity would randomly stop throughout the day for no apparent reason. The alert logs in ISA Server 2004's GUI revealed that the default connection limits were too low, causing any computer accessing ISA Server 2004 (which is our default gateway) to lose connection after a minimal amount of time. (You can find the log alerts by highlighting Monitoring in the console tree, then selecting the Logging tab in the center display.)
To fix the problem, I had to make a small change in a connection value in the Define Connection Limit option, which you access by navigating to Configuration, General in the console tree. After adding our DNS servers to the list of custom connection values, I changed the default value of 1000 connections per second to 100,000 per second. Now, the network traffic doesn't randomly stop any more.
- After receiving complaints from users who were expecting but not receiving mail messages, I began monitoring incoming live traffic. I monitored the traffic with ISA Server 2004's new real-time monitoring feature, which you access by highlighting Monitoring in the console tree, then selecting the Logging tab in the center display. The monitoring revealed that ISA Server 2004 was rejecting legitimate email at the outside interface. (In case you're wondering, using the real-time monitoring feature didn't decrease the performance of ISA Server 2004 or the email server because the servers had adequate amounts of space for log file storage.)
To fix the rejection problem, I increased the NOOP value from 1024 to 65,535 in the SMTP Commands interface, which you access by navigating to Configuration, Add-Ins in the console tree, then selecting SMTP Filter on the Application Filters tab. After stopping and restarting the services, email began to flow normally.
- Client computers were being denied data exchange with some of our business partners that use AS2 software. This software secures EDI over the Internet by using MIME and HTTP instead of SMTP as the transport protocol. In our case, the business partners' AS2 software initiates a connection on a predetermined port with our client computers, which are located behind ISA Server 2004 and are made available through a Web publishing rule. To finalize the connection, the client computers are supposed to send back a Message Disposition Notification (MDN) packet to the initiating computer. However, in our case, the MDN packet wasn't being sent back.
This problem was the most difficult to solve. To fix it, I ended up having to add a registry entry named MTU (short for Maximum Transmission Unit) in the registry subkey for the default gateway (in this case, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces). I set the DWORD value to 1280. The addition of this registry entry allowed our client computers to send back an MDN packet. After I made the registry adjustment and rebooted the client computers, all MDN transmissions were allowed.
Now that I've worked out the kinks and it's fully functional, ISA Server 2004 has proven to be manageable and stable. It's definitely more secure than its predecessor. If you decide to upgrade to or install a fresh copy of ISA Server 2004 and you run into problems similar to those I've described, save yourself some frustration and time and try to make a few adjustments to the out-of-the-box settings.
