A concern in the Windows NT Magazine Lab is test results' reproducibility. Each time we run a test, we need to reproduce the original results within a reasonable margin of error. When we test software, we can make sure that the hardware platform we use for testing is consistent from product to product. For example, for our November 1999 SMTP server software testing, we used a dual-processor server in a fixed configuration from a major vendor. Now, when we perform additional testing of SMTP server software, we can use the same hardware configuration to produce test results that will let us make meaningful comparisons between test groups.
Using application software to test hardware is a more complicated proposition. Although we can use standardized synthetic benchmarks to get a sense of system performance, these benchmarks don't provide the "feel" that using real applications gives us. By testing with applications that our readers use, we can make a more realistic and useful judgment about how various systems will work for you.
Part of the trick to using real applications to perform comparative testing is ensuring that we install the application identically on every system we test. Microsoft's Intellimenu system, which changes the behavior of application menus, has made the task of identical application installation more difficult, although we can turn this feature off. The install-on-demand technology that Microsoft Office 2000 employs is more problematical. Because Office 2000 will stop a task and prompt the user for the Office installation CD-ROM, the program can turn a smooth system test into an agonizing process.
Fortunately, the Office 2000 Custom Installation Wizard, a tool in the Microsoft Office 2000 Resource Kit, not only makes testing with Office 2000 simpler for us but also is handy for administrators charged with deploying Office 2000. You can use the Custom Installation Wizard to create a Microsoft installer transform (.mst) file and effect a custom installation. When an .mst file exists, the Office 2000 Setup application reads the information in the file to do a customized installation of Office 2000.
However, Office 2000 has an annoying tendency to prompt you for the installation CD-ROM at random during operation, even if you've used an .mst file for the original installation. This behavior isn't a problem when the affected system is on a corporate network, and a network share stores the system files. But a random prompt is an unwelcome intrusion during our testing process and a major annoyance to users who must search for the installation CD-ROM. We've found that even when we do a complete installation and select all the available options during the setup process, we can't stop Office 2000 from prompting for files from the installation CD-ROM.
Another tool in the resource kit, the Office Profile Wizard, can address the problem. We have run the Office Profile Wizard on an Office 2000 installation that has been in use long enough to stop prompting for the installation CD-ROM. The Office Profile Wizard creates an .ops file that we can integrate with the Custom Installation Wizard's .mst file to create a stable Office 2000 installation that behaves consistently on installed systems. You can download the resource kit from http://www.microsoft.com/office/ork/2000.
