Last month, I explained that bigger is not always better, and that you can buy too much hardware for some Web site environments. I've seen several Web performance tests done on dual Pentium Pro 200 machines, and I haven't been impressed with the performance of my quad Pentium Pro 200 machine. So I decided to run some performance tests of my own. This month, I'll tell you how I tested the performance of my Web environment and show you what the results mean.
I wanted to make the tests as easy as possible, yet informative enough for me to draw some conclusions. So I grabbed the Microsoft Windows NT Server Resource Kit Version 4.0, Supplement 1 and loaded the Microsoft Web Capacity Analysis Tool (WCAT). WCAT lets you run a simulated workload on a Web server and comes with several prepared scripts that let you simulate different types of environments.
You can run any of four types of WCAT tests:
- Basic tests: These tests consist of static pages of different file sizes.
- Common Gateway Interface (CGI) and Internet Server API (ISAPI) tests: These tests consist of pages that you dynamically create.
- HTTP keepalive tests: These tests are variations of the basic tests and CGI and ISAPI tests that use the HTTP keepalive feature in HTTP 1.1.
- Secure Sockets Layer (SSL) tests: These tests are variations of the first two tests, but they use the SSL encryption protocol, which is typically slower because of the overhead involved.
You can customize the WCAT test environment to mimic your Web environment. For example, I could have created the scripts required to simulate the Windows NT Magazine Web environment (i.e., most of the pages dynamically created with Cold Fusion). But I wanted to try WCAT right out of the box to check its ease-of-use before I started tweaking it. Besides, because I was just glad to have access to the 22 machines I needed to run the tests when I needed them, I wasn't about to mess around with the configuration.
To run WCAT, you need a Web server, a controller server, and some clients. The Web server contains all the files that the WCAT tests request. The controller server is the test administrator. It tells the clients what to do and for how long. It then monitors the Web server and collects data from the clients at the end of the test. The client machines produce a load on the Web server. Each client machine can support up to 200 virtual clients.
Using WCAT, I wanted to evaluate the performance differences between a dual Pentium Pro 200 Web server and a quad Pentium Pro 200 Web server. To minimize the number of variables involved, I used one machine to serve as both Web servers and set it up to be two different machines during different stages of the testing. The Windows NT Magazine Lab lent me a quad Pentium Pro 200 with 512MB of RAM. To level the playing field for all the tests and to simulate practical configurations, I configured the machine to use just 256MB of RAM during the tests. Figure 1, page 211, shows my test configuration.
To set the number of processors and the amount of memory without cracking the case on the machine and yanking out RAM and processors, I used a setting in the Boot.ini file to specify the configuration I wanted to use. In this file, you can specify the number of processors and the maximum amount of memory available to NT. To do so, you need to add the line
multi(0)disk(0)rdisk(0)partition(1)\WINNTSER="Windows NT Server Version 4.00" /NUMPROCS=2 MAXMEM=256
This approach is not suitable for true benchmark testing, but for my purposes, this configuration worked perfectly.
For the controller server, I used a second quad Pentium Pro 200. You don't need to use a machine this big for this task, but the Windows NT Magazine Lab already had this system set up as a controller. For the clients, I used 20 Pentium 133 machines, each with 32MB of RAM, and added virtual clients to each of these machines. Finally, I performed the tests on a network made up of 100Mb connections, because I knew that this many clients would easily saturate a 10Mb network.
Following the WCAT installation directions to the letter, I first installed WCAT on the Web server. Then I installed and configured WCAT on the controller server and the 20 client machines. Setting up the tests was easy. However, installing and configuring 20 client machines can get tedious. Fortunately, you have to set them up only once.
At this point, I tested the WCAT client's ability. I didn't want to overload the clients and have them become a potential problem or bottleneck, so I performed a few tests to determine how many virtual clients each physical client could sustain. For this part of the exercise, I used the ISAPI WCAT test that I planned to use for the primary testing and kept adding virtual clients to each physical client until I started to see errors. I discovered that each client could sustain 60 virtual clients. So with 20 client machines, I was able to simulate a maximum of 1200 total clients.
The Test
After you finalize your test configuration, all you must do is decide which
test to run. I wanted to make the testing realistic (i.e., I wanted a test that
combined static and dynamic pages), so I ran one of the ISAPI tests. The test I
selected consisted of half dynamic pages and half static pages. I ran the test
using 5 client machines (300 virtual clients), 10 client machines (600 virtual
clients), 15 client machines (900 virtual clients), and then 20 client machines
(1200 virtual clients). I tested these settings for both the dual Pentium Pro
200 and the quad Pentium Pro 200. I considered conducting this test with only
one Pentium Pro 200 running, but the test choked after I started the 10 client
machines. After I discovered that this configuration couldn't make the grade, I
decided to stay with the original test configuration--dual versus quad.
With all the clients running and waiting, you only need to edit the test configuration file to tell it how many client machines to use, how many virtual clients on each, and how long to run the test (for my tests, the duration was 10 minutes). Then, from the command prompt, you simply type
run TESTNAME
where TESTNAME is the name of the WCAT test you want to use (in my case, the test name was isapi50). The controller server takes care of the rest and tells the clients everything they need to know.
Although WCAT is handy for this type of testing, Microsoft never intended WCAT to be an official benchmark or an extremely scientific test. This limitation doesn't mean you can't get down to some specific numbers about the performance of your Web site with WCAT; I think you can. However, for my purposes, I just wanted to run a test with everything equal except the number of processors until the machines choked. That way, I could see what the results told me about the benefits of beefing up my Web server hardware.
The Results
So what did the test tell me? I learned that in this environment, a quad
Pentium Pro 200 does not have a significant advantage over a dual Pentium Pro
200. Throughout the tests, the results were almost identical for both
configurations. As you see in Graphs 1 and 2, the dual and quad Pentium Pro 200
machines have almost identical performance numbers for 300, 600, and 900
clients. Both configurations have unacceptable numbers for 1200 clients because
of several connection errors. This falloff is probably the result of saturating
the network somewhere between 900 and 1200 clients. But as you can see, both
configurations handled the saturation in the same way. The test shows that the
dual Pentium Pro 200 machine can do everything the quad Pentium Pro 200 machine
can do up to the point of saturation of the 100Mb network (translation--save
your money or spend it on several Web servers rather than one big Web server).
WCAT is an excellent tool that makes performing consistent tests a breeze. With the proper environment and this type of tool, you can really fine-tune your Web server. I plan to use WCAT a lot in the near future. I've found several interesting Registry hacks I want to try out and some tuning suggestions for Cold Fusion. With WCAT, you run the test against your Web server, make whatever tweaks you want, and simply run the test again. The results can easily tell you if you're headed in the right direction.
| Microsoft Windows NT Server Resource Kit Version 4.0, Supplement 1 |
|
Author: Microsoft Publisher: Microsoft Press, Redmond, 1997 ISBN 1-57231-559-8 Price: $39.99; 336 pages |
