If you're getting ready to implement Microsoft Exchange 2000 Server,you've probably been thinking about load testing—an important step in evaluating Exchange 2000's effectiveness in your environment. You can use load testing to appraise basic Exchange system functionality (i.e., whether email goes where it's supposed to), but load testing's usefulness doesn't end there. Exchange 2000 setup involves populating Windows 2000's Active Directory (AD), but the process is fairly quick and populating the environment and driving a load in Exchange are easier than building other database-driven solutions. Therefore, load testing Exchange is a good way to determine the abilities of a new piece of hardware or to test a network design. (You can use Exchange load testing to demonstrate a proof of concept—for example, to show the performance of a distributed Web architecture that uses front-end servers for client connection points and back-end servers for mailbox stores.) Using this method, you can have new hardware running in a matter of days.
As Web Table 1 shows, a variety of tools are available for testing Exchange, depending on the type of clients you want to represent and the protocols you want to test. (For more information about how to view this table, see "More on the Web," page 61.) One of the most well-known tools is the Microsoft Exchange Load Simulator (LoadSim), which works well with Messaging API (MAPI) clients such as Microsoft Outlook. LoadSim 2000 is the most recent version of the tool and can provide excellent data about your Exchange 2000 servers. (For information about using earlier versions of LoadSim with Exchange Server 5.5, see Greg Todd's four-part series about LoadSim 5.0, beginning with "Understanding and Using LoadSim 5.0: Part 1," January 1998, InstantDoc ID 3429.) You can use Win2K's Performance Monitor to capture and analyze performance data, then adjust your Exchange Server configuration depending on what you discover.
LoadSim 2000
LoadSim lets you select the number of Outlook clients that you want to simulate against specific Exchange Servers. You can choose from preselected load profiles, or you can create custom profiles. The tool includes a light, medium, and heavy profile. Earlier versions of the tool used the MAPI Messaging Benchmark (MMB) for the medium profile, but to simulate more realistic email usage, LoadSim 2000 uses the updated MMB2 for the default light-usage profile. Don't be fooled into thinking that MMB2 is lightweight, though. MMB2 adds significantly more client-generated load than MMB did—MMB2 is about seven times heavier than MMB.
The Microsoft Exchange 2000 Server Resource Kit contains LoadSim, but I recommend you use the most recent version: LoadSim 2000 build 4612, which you can download at http://www.microsoft.com/exchange/downloads/2000/loadsim.asp. The version that ships with the Exchange 2000 resource kit is substantially different from the downloadable version. Instead of creating mailboxes and distribution lists (DLs) in AD in one step, the resource kit version generates two files (i.e., a .csv and a .pkl file). You need to use the Exchange Migration Wizard to import the .pkl file, which is a packing list; this step populates AD. This extra step is unnecessary when you use the most recent LoadSim version. Also, the earlier version is based on the Exchange 5.5 hierarchy (i.e., site.org.com) and doesn't use the predefined MMB2 profile.
To run a custom profile, you can enable or disable specific actions (e.g., browse public folders), as Figure 1, page 60, shows. You can even throw huge attachments into the mix. This capability makes LoadSim quite easy to use and gives you a great deal of control. One word of caution, though: When you customize client tasks, you must specifically enable each task that you want the customized test to simulate. Any tasks that you don't enable manually won't show up in your test. This seemingly simple point can be easy to forget because enabling and customizing tasks involves several steps and dialog boxes. For ease of use, stick with MMB2 unless you have a specific reason to alter the client profile.
Installing LoadSim
The LoadSim installation folder contains a setup program so that you can install LoadSim on your client systems from one location. However, I simply copy the entire folder to the local hard disk of each client machine I want to use. The folder contains the core files loadsim.exe, loadout.dll, and lslog.exe, as well as 33 .msg files, which are the messages that LoadSim sends out during testing. Your client systems need to run Outlook 2000—LoadSim 2000 doesn't support Outlook 2002. If you plan to use the MMB2 load, you can use a maximum of 500 simulated users for each client machine of recent vintage (i.e., a Pentium 500MHz processor or better with at least 128MB of RAM). If you use a slower Pentium processor with 128MB of RAM, consider limiting the number of simulated users to 200. To guard against client bottlenecks, which will cause the Exchange servers to sit idle, set up one client machine with 100 or fewer simulated clients. When you analyze your test results, this client should show the same results as the others. If not, the client machines are overloaded and can't generate a complete load against the server.
Log on at each client as the Domain Administrator. Using this account is a LoadSim requirement, so be sure to run your tests in a laboratory AD forest only. You don't need to configure AD; LoadSim will create the necessary containers. On each client, create an Outlook profile (e.g., for the Administrator), then launch Outlook to ensure that the program is working and to register Outlook as the default client for email and contacts. This step is vital if LoadSim is to function properly. I usually enable autologon—something I'd never do with a production Domain Administrator account—to enable fully automated testing. For instructions about setting up autologon, see the Microsoft article "How to Enable Automatic Logon in Windows" (http://support.microsoft.com/default.aspx?scid=kb;en-us;q310584).
If you want to gather LoadSim-specific performance information, you need to install the LoadSim counters on the clients. To install the counters, open a command prompt in the LoadSim folder and type
unlodctr loadsim
This command removes any previous versions of the counters. Next, to enter the necessary registry information, type
lsperf.reg
Accept the prompt to enter the information in the registry. To load the new counters, type
lodctr lsperf.ini
Copy or move the lsperf.dll file to the \%systemroot% directory (usually \winnt). Finally, launch Performance Monitor (you can do so from the command prompt) and verify the presence of the LoadSim Global and LoadSim Action Performance Monitor objects.
Defining the Test
After you configure the clients, the next step is to define the Exchange environment. Define the storage groups (SGs) and mailbox stores on each Exchange server you want to test. Make sure that the SGs and stores match your production Exchange configuration, especially if you place log files and databases (i.e., .edb and .stm files) on separate physical disk arrays. Open each SG's Properties dialog box, go to the General tab, and select the Enable circular logging check box. This action might run against everything you've ever been taught, but if you don't enable circular logging, the transaction log volumes will fill up during the mailbox initialization process. Don't worry—you'll turn off circular logging before you run the test load. When you see the prompt that Figure 2 shows, click Yes to continue.
Launch LoadSim on one of the clients. From the LoadSim menu bar, select Configuration, Topology Properties to open the Topology Properties screen, which Figure 3 shows. (Earlier LoadSim versions' screens are quite different: You need to add the servers to your test configuration and specify the server location properties—i.e., Admin Group, Organization, and Internet Address—manually.) On the Servers tab, expand each server object to see the SGs and mailbox stores that you configured earlier. In the right-hand pane, enter the desired number of users for each mailbox store. This step is tricky at first because you must expand the SGs, then click and edit in the right-hand pane the number that corresponds to each mailbox store. You can enter numbers only for the mailbox stores; LoadSim then calculates and displays the total number of users for each SG and server. You can distribute users however you want; you define the actual running load for each client later. For example, you can specify 500 users per mailbox store, then initialize and run only 400 of those users.
On the Topology Properties screen's Security tab, you can choose the Use a separate account for each user or Use one account for all users option. I prefer to use a separate account for each user because that option more accurately reflects logon activity on your domain controllers (DCs). On the Topology Properties screen's Distribution Lists tab, you can define the number and depth of DLs; I find the defaults acceptable.
The next step is to specify the number of users per client test machine. From LoadSim's menu bar, select Configuration, Test Properties to open the Test Properties dialog box, which Figure 4 shows. (When you first open this dialog box, the User Count column in the User groups section will contain only zeros.) In the Duration of simulation section, you can specify the length of your test run; an official run needs to be 8 hours. You can run a shorter test, but keep in mind that the test includes a warm-up period during which LoadSim logs on users; Exchange responds to the increasing load by allocating memory to the Store process through Dynamic Buffer Allocation (DBA). Don't analyze any information until Exchange reaches a steady-state period—usually after 2 hours. You can also specify daytime and nighttime periods to differentiate full-load from partial-load periods when you leave LoadSim running for more than 24 hours—which is useful for testing a WAN before placing users on it.
To add a user group, click Add in the User groups section. This action opens the Edit User Group dialog box, which Figure 5 shows. First, select the server that you want to run the load against. When you select a server from the Server drop-down list, LoadSim automatically limits subsequent input to the total number of users that you specified for that server. Although you entered those users according to mailbox store, the Edit User Group dialog box doesn't let you specify which store the users are homed in. Instead, you enter the server, the number of users, and the client that will generate the load. For the most part, accept Outlook in the Protocol box and MMB2 in the User type box. Enter a number in the First user box, then enter the Number of users you want in this group. Keep in mind that LoadSim considers 0 to be the first possible number (e.g., if you enter 0 as your first user and specify 250 users, the Users covered will be 0-249). In the Client Machine box, specify the client machine that will run the load, keeping in mind the constraints of 200 to 500 simulated users per client. Click OK to return to the Test Properties dialog box. Repeat the process to add more groups, spreading users over all servers and available client machines. Click OK to close the Test Properties dialog box.
After you finish adding user groups, save the .sim file—preferably to a file server that you've set up with your installation source files or to a monitoring station that all clients can access. To save the file, select File, Save from the LoadSim menu bar. During the test runs, all clients will access the same .sim file. An important trick is to use the menu bar's File, Save As option to save the file again with a different name. Open the second file, then open the Test Properties dialog box. For each server, remove all the client machines except one and increase the number of users on that client to cover all users on that server. Why? LoadSim's initialization phase creates the mailboxes and populates them with messages, appointments, and contacts. Running the initialization from one client per server causes it to run smoothly. Otherwise, the Exchange server gets hammered processing multiple requests to create and stuff mailboxes and will eventually fail to create all the required items. After you configure all clients for this initialization, save the second LoadSim file.
Open your initial LoadSim file and select Tools, Options from the menu bar. Both sections on the Option dialog box's Logging tab include an Archive previous file check box, which is cleared by default. Select both check boxes, close the dialog box, then save the LoadSim file. The default setting instructs LoadSim to reuse one file to generate output. Assuming that you have a few megabytes of free disk space on your clients, archiving the file will cause LoadSim to rename old output files (appending a number) and generate a new file. Archiving is helpful when you want to make several test runs before you analyze your data—otherwise, you have results only from your final test run.
Populating the Directory
After you define the topology and test properties, you're ready to populate AD with all the necessary DLs and user accounts, then mailbox-enable those objects. From the LoadSim menu bar, select Run, Create Topology. After the progress bar shows that the process is finished, open the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in and look for the Loadsim Users container, which holds additional containers for each server, to verify that the process was successful. If you find errors, delete all users and DLs and repeat the Create Topology process. Be aware that the user accounts won't display Exchange properties (e.g., SMTP addresses) until Exchange 2000's Recipient Update Service (RUS) has a chance to process all the new users. (This process usually takes a few minutes.)
Initializing the Mailboxes
The next step is to stuff your test mailboxes full of mail. To initialize your mailboxes and public folders, open LoadSim on each client that you configured in your second LoadSim file. On each client, select Run, Initialize Test from the LoadSim menu bar. On each client, LoadSim displays a prompt, asking whether you want to initialize public folders. If you initialize from multiple clients, click Yes to accept this prompt on one client and click No to refuse the prompt on the other clients. (Mailbox creation occurs during initialization—mailboxes are created when the first piece of mail arrives rather than when the accounts are created in AD—and is extremely load-intensive on your Exchange servers. If you attempt to run too many clients during the initialization process, you might overload the servers or your network and receive MAPI errors such as Unable to log on.) The initialization process takes several hours at best, so I suggest you run it overnight. If you forgot to enable circular logging earlier (or simply oppose it), be sure to monitor your log-disk space or you'll experience first-hand what happens to Exchange 2000 when the log volumes fill up.
When the initialization is complete, verify the number of items per mailbox. To do so, open the MMC Exchange System Manager snap-in and expand the Servers\Storage Groups\Mailbox Stores\Mailboxes container to see whether all mailboxes contain the same number of items for MMB2. Note that the mailboxes' sizes vary because LoadSim randomizes the attachment mix as it populates the mailboxes.
The initialization process takes a long time. Unless you want to risk having to repeat this step, back up the stores so that you can simply restore from backup before repeating a test run. Turn off circular logging before you create your backup. I typically use Win2K's built-in NT Backup utility to back up the Exchange stores directly to disk. (A minor problem with this backup tool is that you can't remotely back up a server. I log on to the Exchange server locally or through Win2K Server Terminal Services, then schedule and run the backup job from there.)
Capturing Performance Data
While LoadSim is initializing, you can set up your performance-logging environment—the final task before you run the performance test. This procedure contains three steps: Create the capturing log file, open the log-file settings and connect to a server, and add the performance counters to capture data for that server. On your collection workstation, which should run Win2K, launch Performance Monitor, expand the Performance Logs and Alerts node, then right-click the Counter Logs node. Select New Log Settings from the context menu (unless you've already saved some setting during previous monitoring, in which case you need to select New Log Settings From and enter the name of your existing file). Enter a descriptive name (usually the monitored server's name), then click OK to open the settings dialog box.
Now you need to add the servers and counters you want to monitor. (See Web Table 2 for suggested performance objects, counters, and instances for specific Exchange performance-monitoring categories.) The process is a bit different in Windows XP than it is in Win2K. In XP, click Add Objects, then enter the name of the server that you want to monitor. Wait for Performance Monitor to verify the connection and access privileges. If you don't have NetBIOS name resolution or administrator permissions on a particular server, the connection process might fail. After you establish a connection to the server, select a performance object, then click Add to add all counters and instances of that object. In Win2K, click Add to open the Select Counters dialog box, which Figure 6 shows. Select the server you want to monitor from the Select counters from computer drop-down list, then wait for Performance Monitor's verification. Next, select the All counters and All instances options. Click Add, then click Close, then click OK on the main settings dialog box to save the settings.
Alternatively, you can select a limited set of counters to capture data at a smaller interval and keep the log files fairly small (i.e., a few megabytes). With modern disk sizes, though, I select the All counters and All Instances options for the Performance Objects that Web Table 2 lists, then clean up later. Another common option in performance monitoring is to define two capture log files, one at 1-minute intervals over the 8-hour duration and one at 15-second intervals for an hour or two in the middle of the load test.
You're now ready to set the logging options. Double-click the log-settings file you just created to reopen the settings dialog box. On the Log Files tab, which Figure 7 shows, specify your log-file information. (Note that in Figure 7, I'm logging to the default location, which is my C drive. Therefore, I need to monitor disk space carefully.) The default log-file names are based on your log-settings name, which is why I suggest you use the server name so that you can easily separate and identify logs from each server. You can later move the full set of log files to a folder for each test run.
The Schedule tab, which Figure 8 shows, lets you choose more options than the original Windows NT 4.0 Performance Monitor did. You can schedule the start time for performance logging, and you can specify the end time as either an explicit time or as a number of hours after capture has started (I prefer the latter method). You can enter a command for LoadSim to run when a log file closes; I set up a netsend.cmd file to send me an alert. Click OK to close the dialog box.
Because you're capturing performance information remotely from computers across the network, you need to change the security parameters for Win2K's Performance Logs and Alerts service. By default, the service logs on as the Local System Account. You need to change the service's settings (which you can access through the MMC Services snap-in) to log on as a Domain Administrator account (not the local Administrator account).
You also need to enable logical disk counters to capture the performance data. To enable the counters for logical drives, open a command prompt and type
diskperf yv
To enable counters for both logical and physical drives, type
diskperf y
You can repeat this configuration process for each server you want to monitor, but a far easier method is available. Open the log-settings file, which Performance Monitor saves as an .htm file, in Notepad or another text editor. Save the file under a new name (by default, Notepad uses the .txt extension unless you put quotes around the new filename—e.g., "server1.htm"), then search for and replace the original server name with the new server name. Save your changes, then repeat the process for each server.
Performing Load Tests
Now you're ready to run the load test. You have two ways to go about this process: You can visit each load-generating client and select Run, Run Simulation from the LoadSim menu, or you can use fully automated testing (see the Web-exclusive sidebar "Fully Automated Testing with LoadSim 2000" at http://www.winnetmag.com, InstantDoc ID 24127, for details). Synchronizing the time on all client machines before you kick off the test is a good idea but isn't crucial because you can reset the LoadSim output results to account for clock inconsistencies. As the tests are running, visually inspect the environment to make sure that the clients aren't reporting any errors and that the servers keep chugging along. LoadSim logs messages to a loadsim.out file as well as to the display, which shows information about the simulations.
After the simulation starts, launch the Counter Logs that you set up earlier, and make sure that they keep running. Be sure to let the test load run for at least 2 hours to achieve steady state and another 4 hours to provide a reliable run. LoadSim might take a few additional hours to shut down and clean up all pending requests. After you run your first test, look for errors to see whether the data is worth keeping. For a few other testing options and information about setting up additional test runs, see the Web-exclusive sidebar "Testing Options and Additional Tests" (http://www.winnetmag.com, InstantDoc ID 24128).
Internet-protocol load testing, which simulates clients such as Outlook Web Access (OWA), is a bit more complex than MAPI-client testing. To test non-MAPI clients and protocols, you need to use other tools—in particular, the Microsoft Exchange Stress and Performance (ESP) tool. In my next article, I'll explain how to use ESP to simulate Internet-protocol clients and how to analyze your LoadSim and ESP test results.
