When civil engineers design a bridge, they try to determine the point at which demand and capacity intersect. Demand is composed of factors outside the physical structure of the bridge, such as the maximum number of vehicles that can be on it at one time and the number of vehicles that will cross it in a year. Capacity has to do with the known strength of bridge construction materials.
Server engineers face the same demands as bridge builders, although in a somewhat more virtual fashion. If a server supporting a mission-critical Windows NT application becomes processor bound and unresponsive during the busiest time of day, cars won’t fall into rivers, but the engineer who attested that the server would be able to handle the demand might feel like jumping.
Administrators face questions about capacity nearly every time they need to purchase or configure a server or an application, but they’re at a disadvantage compared with their counterparts in civil engineering. Determining a server’s capacity is an inexact science. You can use one of three methods to attempt it.
Rely on vendors’ recommendations. Hardware manufacturers and application software makers usually provide recommendations for the type of server hardware you should use and how you should configure the hardware and software. However, those estimates tend to be general and conservative.
Simulate the server’s actual use. If time and a suitable test environment are available, an application server’s engineering and construction phase can include a period during which that server is tested under "live fire" (i.e., actual use by users). Companies often use this method, however, for "proof of concept" rather than for a true capacity test (i.e., they don’t test the application server to the point of failure).
Use software to simulate a heavy workload. Simulation solves the other two options’ major shortcomings. On NT servers, you can perform workload simulation with a utility called Response Probe, which Microsoft provides in the Microsoft Windows NT Server 4.0 Resource Kit. (Response Probe is incompatible with Windows 2000.)
What Is Response Probe?
Server workloads can be difficult—even impossible—to duplicate because factors such as background network traffic and number of users logged on are hard to control. Response Probe’s strength is its controllability. Response Probe lets you design a reproducible, application-independent workload and use it to test the performance of a server’s hardware and software configuration without anyone having to use the server. The tool doesn’t manipulate the application running on the system; rather, it evaluates performance by generating unique threads that create a specific load on the server. In other words, Response Probe replaces the application in question. You run Response Probe instead of the system’s primary application but along with the system’s secondary applications.
Response Probe works by simulating real computer use. By real, I mean that the tool simulates a typical workload cycle—a variable think time followed by file access followed by computation. Response Probe assumes that many characteristics of real workloads follow a standard bell curve, or normal distribution, and designs the test workload accordingly. For example, the time that users spend thinking about what action to take varies from user to user and task to task, and the time spent finding a record in a file varies according to the location of the record on the hard disk relative to the read arm’s current position. Response Probe assumes that in a typical workload, these and other workload variables are distributed normally and so can be described by specifying a mean and a standard deviation. (For details about how to use a mean and standard deviation to specify a normal distribution, see Chapter 11 of the Microsoft Windows NT Workstation 4.0 Resource Guide at http://www.microsoft.com/technet/prodtechnol/ntwrkstn/reskit/03tools.asp?frame=true) Response Probe lets you specify means and standard deviations for several such workload characteristics, as you’ll see later.
To test your server’s performance, Response Probe relies on three closely linked text files that describe the simulated workload. These three files are
- a process script file (.scr), which creates each process in the test workload
- a thread definition file (.scp), which creates the threads that will run in the process
- a thread description file (.sct), which describes a thread process (You need a thread description file for each thread you define.)
Response Probe requires at least one of each of these files for every test it performs.
Using these files, Response Probe generates a precise activity level that’s identical each time you run the same script configuration. By varying the script parameters from test to test, you can first establish a baseline for performance, then use Response Probe to find the upper limits of a server’s capacity.
Creating the Script Files
You use the process script file to create each process in the test workload. Each line in the file creates a process and uses the syntax
REPEAT <n> PROCESS <ThreadDefFile.scp> <DataPages> <ProcessName> <PriorityClass>
REPEAT n is an optional parameter that runs the process n times. ThreadDefFile.scp specifies the name of the file that defines the process’s threads. DataPages specifies the number of pages designated to simulate data pages. (Data pages refers to NT’s internal buffer storage of data.) ProcessName is an optional parameter that specifies a name for the process in which the test will run, and PriorityClass is an optional parameter that sets a relative priority for the process. Possible values for PriorityClass are I (Idle), N (Normal), H (High), and R (Realtime). For example, the line
PROCESS servbase.scp 10
creates one process, associates it with thread definition file servbase.scp, and specifies a 10-page datapage file.
In the thread definition file, you create the threads that you name in the process script file by using the syntax
REPEAT <n> THREAD <ThreadDesFile.sct> <ThreadPriorityAdjustment>
REPEAT n is an optional parameter that creates n instances of the thread. ThreadDesFile.sct specifies the name of the thread description file, and ThreadPriorityAdjustment lets you set the thread to a priority other than the one you specified for the process. Possible values for ThreadPriorityAdjustment are T (TimeCritical), H (Highest), A (AboveNormal), N (Normal, the default), B (BelowNormal), L (Lowest), and I (Idle). For example, the line
THREAD servbase.sct
creates the thread that you’ve described in the thread description file named servbase.sct. Because this thread definition line doesn’t adjust the priority, the thread runs at the same priority as the process.
Finally, you describe the thread in each process in the thread description file. You set most of Response Probe’s configuration values in this file, including the means and standard deviations for several parameters that Response Probe uses to generate each thread. Table 1 shows the parameters that you can configure.
You build the thread description file in table format with one line per parameter. The parameters can be in any order. For example, the lines
THINKTIME 0 0 CPUTIME 0 0 CYCLEREADS 100 30 FILESEEK 0 0 DATAPAGE 0 0 FUNCTION 500 0 FILEACCESS workfile.dat FILEATTRIBUTE SEQUENTIAL FILEACCESSMODE UNBUFFER RECORDSIZE 2048 FILEACTION R
set the parameters for a thread that performs reads from a file. The parameters indicate that this thread
- requires no think time or additional CPU time
- reads data an average of 100 times between think times, with a standard deviation of 30 times
- uses sequential access and thus ignores file-seek time
- triggers no datapage activity
- repeatedly reads the function in the center of the codepage to simulate minimal codepage access
- reads records from the workfile.dat workload file
- reads records sequentially
- doesn’t use system cache
- reads data in 2KB chunks
- reads only (doesn’t write)
Unlike earlier versions of Response Probe, Response Probe 2.3 automatically generates and reads from an internal codepage file. The codepage file that Response Probe 2.3 uses contains a set of 1000 built-in function calls or actions that the tool can perform. This file simulates application code from which instructions are read during a process. You can tweak specific parameters after determining which parameter values best represent your application load.
You don’t need to create Response Probe’s script files wholly from scratch; you can modify the sample files in \ntreskit\perftool\probe\examples. Among these sample files are several sets of excellent baseline scripts that might suit many system designers’ testing needs:
- Diskmax.scr determines a disk drive’s maximum throughput by performing sequential, unbuffered reads of 64KB records from a 20MB file.
- Minread.scr measures how disk performance varies when reading one sector of the disk at a time; this sample file performs sequential, unbuffered 512-byte reads from a 20MB file.
- Sizeread.bat tests a disk configuration’s performance in reading records of increasing size. This file is a batch file rather than a process script file. Sizeread.bat runs a series of tests of unbuffered, sequential reads from a 20MB file in which the size of the record read increases from 2KB to 8KB, 64KB, 256KB, 1MB, and 4MB.
Configuring the Server
Before you launch Response Probe, configure the server hardware as you’d configure it for production work, then configure any secondary applications and utilities (e.g., antivirus utilities, system-monitoring software) that will run concurrently with the primary application. Run only those programs and services whose activity level won’t vary significantly over time. For example, a monitoring service such as NT’s SNMP Agent adds to the demand on a server, but that demand is constant—it doesn’t vary with the number of user connections or any other factor—so you can incorporate this service into the configuration. Conversely, you shouldn’t include Microsoft SQL Server in a test because that application’s impact varies.
Using Response Probe
Response Probe is installed on a server when you install the resource kit. Its installation has no parameters. The tool consists of the files in the \ntreskit\perftool\probe folder (including the examples subdirectory).
Running Response Probe is easy after you’ve defined the three script files. You run Response Probe from a command prompt. The syntax is
probe <ProcessFileName.scr> <TrialTime> <OutputFileName>
where ProcessFileName.scr is the name of the process script file that describes the threads you’re testing, TrialTime is the length of time (in seconds) that the test will run, and OutputFileName is an optional parameter that creates an output (.out) file to which Response Probe will save test results (the default output file name is ProcessFileName.out). You can’t direct the utility at another computer on the network; you must launch it from the system you want to test.
No Longer in the Dark
One of Response Probe’s most important uses is determining just how much more powerful a new server will be than the one it’s replacing. Manufacturers market server hardware with plenty of "performance figures," but how do increases in processor speed, disk-seek time, RAM capacity, processor bus size, and network-interface bandwidth really translate into overall performance? Many of these factors change with each new server model, so comparing servers based on the rated performance of their components becomes nearly impossible. Yet the most common question customers ask about a new server is, "How much faster is it going to run?"
I find Response Probe particularly valuable in answering this question because you can use it to create a true performance-based baseline. Run Response Probe on a 2-year-old application server that you’re using now, and you’ll have a relevant baseline for comparing candidates to replace it. Then, run Response Probe against the same software configuration on the newer servers, and you’ll have a result that correlates directly with the results you saw on the old server, clearly showing the performance difference between the two systems. Other tools, such as Microsoft Performance Monitor, provide information only about a server’s separate components. Response Probe gives a complete picture of the server’s overall performance.
