Microsoft recently responded to several new security holes in Windows NT. Some of these security holes required only a configuration change to protect the system, but many required Microsoft service packs or hotfixes.
Security experts estimate that patches (i.e., configuration changes, service packs, or hotfixes) are available for over 90 percent of system breaches that occur. Security suffers if you don’t apply patches in a timely manner. You can reduce your risk by keeping up-to-date with new exploits and their patches. However, managing service packs and hotfixes is daunting when you have hundreds or thousands of systems to maintain. For example, Microsoft often releases, updates, and removes hotfixes at its FTP sites without informing users. Documentation regarding hotfix installation and cumulative compatibility is sometimes contradictory and confusing. In addition, native NT functionality doesn’t permit easy distribution of updates to multiple computers. To further complicate matters, some hotfixes require you to make risky Registry changes to activate the patch’s functionality. Even simple hotfixes can destabilize systems and introduce new bugs. Finally, obtaining enterprisewide reports on current update levels is difficult, and total cost of ownership (TCO) soars when you maintain systems at wildly disparate update levels.
Without an easy method for administering patches, your costs increase and your security and stability suffer. In this article, I discuss how you can securely manage updates by using simple batch files, NT’s native tools, and inexpensive or free tools and services that are available on the Internet. The process involves discovery, evaluation, testing, deployment, and tracking. (For more information about service packs and hotfixes, see "Related Articles in Windows NT Magazine.")
Discovery
A proper discovery process depends on your level of specialization and responsibility in security matters. In the past, systems administrators often waited until they ran into a problem before they looked for a patch. But current security demands necessitate a proactive approach: You must search for fixes before you need them.
For years, UNIX users have had security bulletin services such as the Computer Emergency Response Team (CERT) and Computer Incident Advisory Capability (CIAC), which announce new exploits and vendor patches. Microsoft only recently introduced its Security Notification Service. At a minimum, you’ll want to subscribe to this service. (Go to http://www.microsoft.com/security/ services/subscribe.asp.) However, this vendor-based information source provides only information that serves Microsoft’s best interests.
Mailing lists exist on which the industry’s best hackers and security experts discuss exploits and fixes. My favorite resource is the NTBugtraq mailing list. You can subscribe to this list at http://www.ntbugtraq.com. Discussion on this list revolves around NT exploits and fixes. Russ Cooper effectively moderates the list, which more than 15,000 users subscribe to. NTBugtraq is one of the best resources for untangling hotfixes’ idiosyncrasies and contradictory documentation. For additional NT security mailing lists, go to http://www.ntsecurity.net and http://www.iss.net/vd/maillist.html.
The volume of email from security mailing lists can be overwhelming. I direct all the mail to an NT Security folder in Microsoft Outlook. Once a day, I scan the subject lines for topics of immediate relevance. I also scan the authors, because I’ve learned to recognize the regular posters whose messages are consistently valuable. This method lets me spend a minimal amount of time keeping up-to-date. When I have some downtime, such as on a flight, I scan the rest of the messages for new problems, tricks, and insights.
If you’re trying to solve a particular problem, you might have difficulty finding the appropriate hotfix. You can use NTBugtraq’s free service at http://ntbugtraq.ntadvice.com/ntfixes.asp to search for hotfixes by language, NT version, processor type, and service pack. The tool even highlights hotfixes that the vendor has updated or removed.
| Related Articles in Windows NT Magazine |
|
Mark Joseph Edwards and Paula Sharick "Service Packs and Hotfixes," November 1998 Paula Sharick "Lsa2-fix and Priv-fix," December 1998 |
Before you install a patch, determine whether you need it. Microsoft recommends not installing a hotfix unless you experience the problem it fixes or you need the added functionality the hotfix provides. You don't need every patch available. Moreover, some of your systems might need a patch that other systems don’t need. Even in the case of security fixes, each exploit has certain prerequisites or includes target components that don’t apply to all systems. For example, you don’t need to apply an Active Server Pages (ASP) fix to workstations or servers that don’t run Internet Information Server (IIS).
To evaluate which systems need a patch, many administrators divide their NT systems into classes such as workstations, file servers, and Web servers. You might also want to rate your systems’ security importance. You need to keep up-to-date those exposed servers that act as email gateways or Web servers, as well as those systems that contain crucial information such as R&D.
Testing
You need to test patches before you deploy them. Microsoft doesn’t perform regression testing on hotfixes, and the company admits you’ll probably have problems. Microsoft’s technical notes for deployment recommend that you conduct an extensive suite of lab tests before you implement a hotfix. Most administrators don’t have the time or resources to conduct such tests. The following guidelines will help you conduct manageable tests on service packs and hotfixes.
NT doesn’t run on a proprietary hardware platform: Each NT site is a unique combination of BIOS, video cards, storage systems, and drivers. Thus, you need to test updates on your hardware and configuration in a lab environment. This lab test doesn’t need to be extensive. You need only look for immediate and obvious incompatibility problems that are specific to your environment.
You need to test the update and installation process in a limited-production rollout. This method lets you introduce the update gradually and limit the impact of any problems you didn’t catch during the lab test. To determine who receives the update first, you can go by volunteer (i.e., apply the update to those IS groups that are most interested in the patch and most willing to test it), in order of importance (i.e., update less-critical systems first), or by task (i.e., apply the update to a few machines in each group of file servers, database servers, and domain controllers). Deployment by task is the most scientific route for discovering problems while limiting risk.
You need to run your limited-production test long enough to evaluate the update over an entire usage cycle for the server. A usage cycle includes peak load times and regular batch operations such as backups, downloads, reindexing, and accounting-period-related runs. Usage cycles vary by operation and server type. Application servers typically have daily, weekly, and monthly cycles. File servers typically cycle on a weekly basis. Most administrators test updates for at least a week, and many opt for a month.
The final step might seem obvious, but many administrators overlook it. You need to develop a recovery plan and test it. Be sure to incorporate backups and the uninstall option in an update’s installation process. Ideally, you’ll test an update before deployment. However, you might need to install a security patch on an exposed Web server or email gateway before you have time to test the patch. In this case, a good recovery process is essential.
Deployment
Deployment overlaps testing, because you develop your deployment procedure during the testing phase. You need to ensure that the deployment procedure is the same procedure you tested. In addition, you need to ensure consistent deployment across systems. Your deployment procedure needs to be easy to implement, especially in terms of required man-hours. An automated deployment procedure is ideal.
You don’t need Microsoft Systems Management Server (SMS) or third-party tools to automate patch deployment. All you need is the Microsoft Windows NT 4.0 Resource Kit and a few batch files. In the following sections, I discuss how to write an automated deployment batch file for one hotfix and how to use this batch file to install patches on new and existing workstations and servers.
Automated patch deployment. To configure automated patch deployment, you need to create a network directory that can serve as a hotfix and service pack repository. You also need to create a subdirectory for each hotfix.
After you create the directory and subdirectories, unpack the hotfix by running the fix with the /x option. Then, create a batch file (e.g., update.bat), with hotfix/m/q/z as the first command. These options let the hotfix install without waiting for user input and without requiring a reboot.
Determine whether you need to make any Registry changes to run the hotfix. For instance, when you install lm-fix, you need to add a new value called LMCompatibilityLevel to the Lsa subkey. You don’t need to make this change manually on every system that gets the hotfix: You can use System Policy Editor (SPE). Alternatively, you can use the resource kit's Reg command. This command is the easiest tool I’ve seen for using a batch file to make Registry changes. In my example, you’d add the command REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Control\Lsa\LMCompatibilityLevel=2 REG_DWORD to the update.bat file.
If you need to update the permissions on a Registry key (which is sometimes necessary for security-related patches), use the resource kit tool Regini. This tool lets you use arbitrary numbers to update key permissions for common access-control entries such as Everyone—Read or Administrators—Full Control.
The Reg tool sometimes creates problems related to deleting keys and adding values that already exist. If you experience these problems, consider using Regini, which is a more robust utility. (However, Regini is less convenient because it requires an additional file for input.)
After you include the necessary operations to complete the hotfix’s application in update.bat, you’ll have a simple batch file that you can use to install the associated patch in a variety of contexts. Use this batch file when you install the hotfix during testing. The testing process will help you determine whether you need to make additional changes (e.g., Registry changes) to accommodate the hotfix. If so, add these changes to the batch file. Then, you can use the final script to deploy patches to new or existing workstations and servers.
Deploying patches to new workstations and servers. When you adopt a patch as a standard in your organization, you need to deploy it not only to existing systems but also to new workstations and servers. An easy deployment procedure for new systems is to add the patch to the automated installation process.
If you’re using a script file with the standard installation programs winnt and winnt32, you can use the /e parameter to specify the name of a command to run after setup is complete. This command typically calls a batch file that carries out final installation procedures (e.g., standard applications). In a well-organized environment, this batch file also calls a baseline batch file that applies the current service pack and all adopted patches. Each class of systems needs a separate baseline batch file to apply the correct patches.
Listing 1shows an example baseline batch file. This batch file connects to your patch repository to install updates, using the DEPLOYER user account. You need to create this account and give it read and execute permissions on the repository directory tree.
Deploying patches to existing workstations. The main obstacle in deploying patches to an existing base of hundreds or thousands of NT workstations is configuring automation. You need a way to install the patch and make the Registry updates and other configuration changes without having to visit each computer. Scripting is only part of the solution. In the previous section, I explained how to write a batch file that applies the fix and makes other necessary changes. However, you need an automated way to run the batch file on each workstation.
You can use NT’s native Scheduler service or one of several resource kit tools (e.g., the Remote command-line utility) to easily install patches on multiple workstations. (For a list of Microsoft articles about deploying patches, see the sidebar "TechNet Articles About Service Pack and Hotfix Deployment".) However, these tools introduce security risks because they accept arbitrary commands over the network.
You need a secure way to submit commands to remote workstations. A safe tool to use is the resource kit's Autoexnt utility. This tool looks for and runs a batch file called autoexnt.bat whenever the system boots. (Thus, you’ll want to configure your workstations to reboot once a day.)
To use Autoexnt, you need to create and share a directory named DEPLOY$. Then, grant the DEPLOYER account (which I explained how to create in the previous section) list permissions to the DEPLOY$ directory and change permissions to the DEPLOY$ share. Finally, create a batch file named deploy_stub.bat with the following commands:
For /f %%i In (workstations.txt) Do Copy stub.bat %%i.bat
For /f %%i In (workstations.txt) Do Xcacls %%i.bat /e /p DEPLOYER:rxd /y.
(This code assumes that DEPLOY$ is on the same server as the patch repository.) These commands create a copy of stub.bat for each workstation and give the DEPLOYER account read, execute, and delete permissions on these files. The For command runs a given command for every line in a given text file, using the data from that line to replace the %%i variable. Note that substitution variables in the For command are different from environment variables and use two percent signs. (For more information about the For command, enter
For /? | more
at a command prompt.)
Next, you need to create a batch file named autoexnt.bat in the DEPLOY$ directory with the following commands:
net use \\server\DEPLOY$ <password> /user:DEPLOYER
call \\server\DEPLOY$\%computername%.bat
If errorlevel 1 GoTo error
del \%computername%.bat
shutdown /t:0 /l /r /y /c
:error
net use \\server\DEPLOY$ /d
Install the Autoexnt service and the autoexnt.bat file on each workstation. To reduce this workload, you can create another batch file—setup_autoexnt.bat—to automatically install the Autoexnt service and batch file. Listing 2 shows the setup_autoexnt.bat file, which you need to run as an administrator on each workstation. To help protect this batch file from changes and to hide the DEPLOYER password, the Xcacls command in this file modifies the permissions.
To install a patch on each workstation, you need to maintain a file called workstations.txt in the DEPLOY$ directory. This file must contain each workstation’s NetBIOS computer name on a separate line.
To submit the lm-fix patch (which I described previously) to each machine, create a simple batch file called stub.bat with the following commands:
Pushd \\server\ nt4sp3$\lm-fix
call update.bat
Popd
The Pushd command maps the next available drive to the specified directory and changes the current directory. The Popd command terminates the drive mapping and changes the directory to what it was originally—before the Pushd command. (For more information about these commands, enter
Pushd /?
or
Popd /?
at a command prompt.) Next, from the DEPLOY$ directory, run deploy_stub.bat.
The deploy_stub command makes a copy of stub.bat for each computer listed in the workstations.txt file. Then, when a workstation reboots, the Autoexnt service runs the autoexnt.bat file. This batch file maps a drive to the deployment directory and runs the workstation’s copy of stub.bat. The stub.bat file runs update.bat in the lm-fix directory, thus applying the hotfix and making the Registry changes. If all the commands run successfully, autoexnt.bat deletes the workstation’s copy of stub.bat and reboots the workstation so that the patch takes effect.
Deleting the workstation’s copy of stub.bat serves two purposes. First, the patch is no longer applied every time the workstation reboots. Second, you can easily track which workstations haven’t applied the patch. After a couple of days, you’ll notice a copy of stub.bat, with the associated computer name, in the DEPLOY$ directory of those workstations that haven’t rebooted or that have failed to run autoexnt.bat successfully for some other reason. You can manually reboot those workstations and force the batch file to run. If your workstations don’t reboot regularly, you might want to use the resource kit’s Sleep or Waitfor tools to make autoexnt.bat an infinite loop with an efficient pause in each iteration.
The Shutdown command, which is another resource kit tool, runs an immediate unconditional reboot by virtue of the /c parameter. This command doesn’t cause user data loss, because autoexnt.bat runs before the user logs on.
Automated patch deployment might make you nervous about security. However, handling the process properly prevents most problems. If you use file permissions to prevent users from changing the autoexnt.bat file, the Autoexnt service doesn’t create a security hole. If Autoexnt is compromised, the damage is limited to the local workstation and possible deletion of the stub.bat copies on the deployment server. Be sure you remove the DEPLOYER account from the Domain Users group, and give the account only read, execute, and delete permissions. With limited permissions, the DEPLOYER user account can’t change the stub.bat files and thus can't run arbitrary commands on other workstations.
Some users might object to coding the DEPLOYER password into autoexnt.bat. Alternatively, you can run the Autoexnt service as the DEPLOYER domain account. However, you’d have to make DEPLOYER an administrator on every workstation, which creates additional security risks: If the DEPLOYER account were compromised, users would have administrator access to every workstation. To prevent this type of problem, you’d need to make significant changes to users’ default right assignments.
Make sure you have tight control over write and add permissions on the DEPLOY$ directory. If this directory is compromised, an intruder can propagate malicious code to every computer listed in workstations.txt.
Deploying patches to existing servers. You can use the same method to deploy patches to servers as you used for workstations. However, because you have far fewer servers than workstations, and because of the servers’ importance, you might want to install updates manually at each server. You’ll still want to use the update.bat file coded for the patch to ensure consistency and to save labor.
If you use automated deployment for some of your servers, keep in mind that Autoexnt relies on a reboot to trigger the patch. Servers typically don’t reboot as often as workstations do. Thus, you’ll need to use the infinite loop and efficient pause method that I mentioned in the previous section.
A security concern is that users who have write or add permissions on the deployment directory also have indirect Administrator access on the computers that look in that directory (because Autoexnt runs batch files located in the directory, under the LocalSystem account). To apply hotfixes, you must have this level of access to the systems you’re updating. Users having Administrator access on workstations might not cause problems, but you typically don’t want someone to have Administrator access on all the servers. You might want to eliminate the central DEPLOY$ directory for servers. Instead, create a DEPLOY$ directory on each server, and grant write and add permissions only to that system’s local administrators group. Install autoexnt.bat on this directory so that systems administrators can deploy patches to it.
Tracking Patch Levels
Even with a well-conducted patch-management procedure, you’ll sometimes wonder what patch level a system on your network is running. Without SMS, determining patch levels can be tricky. You can use the hotfix command with the /l parameter to display the Microsoft Knowledge Base article numbers for installed hotfixes; however, this method doesn’t accommodate remote workstations, and the information it provides isn’t particularly useful.
MTE Software (http://www.mtesoft.com) has a useful tool called SPQuery. This tool lists current service packs and hotfixes on a system. It provides the Knowledge Base number, the hotfix date, and a text description. The latest version of SPQuery can even download, install, and remove hotfixes from a system.
Get A Good Night’s Rest
You don’t have to work nights and weekends to keep your network current and secure. You just need to implement a well-organized strategy for managing service packs and hotfixes. To maintain a secure network in today’s dynamic environment, you must stay up-to-date with security developments, evaluate patches for applicability to your systems, test the patches you need, and deploy them to your systems
