NT Gatekeeper: High-Encryption

After a recent security review of our systems, our audit team recommended upgrading the systems to the high-encryption version of Windows NT 4.0. What does high-encryption refer to, and why is the high-encryption version more secure than the standard version?

NT 4.0 offers standard- and high-encryption strengths. Microsoft released the OS with a standard-encryption strength of 40 bits but upgraded the standard strength to 56 bits in Service Pack 6a (SP6a). High-encryption (i.e., 128-bit) updates have been available as separate downloads since SP3.

Encryption strength refers to the length of the keys that NT 4.0 employs for RC2 and RC4 secret-key algorithms. (For more information about encryption and the impact of key lengths, see the sidebar "Encryption Basics," page 12.) The encryption provider in NT 4.0's standard version uses either 40-bit or 56-bit keys, depending up the service-pack level. The high-encryption version extends the key length of RC2 and RC4 secret-key ciphers to 128 bits and doubles the key length for RSA public-key ciphers. Key length is the biggest factor in determining how easily someone can crack an intercepted message, so NT's high-encryption version delivers stronger protection than the standard version.

The high-encryption version provides the best protection for passwords and the best data encryption for remote users who access the LAN through dial-up or VPN connections. High encryption also provides the most secure communication between domain controllers (DCs), servers, and workstations and the most secure connections between Microsoft Outlook and Microsoft Exchange Server.

Which NT 4.0 component is responsible for encryption, and which encryption services does it provide?

CryptoAPI provides access to Cryptographic Service Providers (CSPs) for the OS and for developers who want to create secure applications. The CryptoAPI module contains links to housekeeping functions, key-generation functions, certificate encoding and decoding functions, certificate store functions, message encryption and decryption services, message signatures, and low-level support computations. To prevent tampering with encryption-algorithm calculations, CryptoAPI doesn't let programs interact directly with the CSPs.

Microsoft supports two CSPs—rsabase.dll (i.e., the base provider) and rsaenh.dll (i.e., the enhanced provider). NT 4.0's standard version installs rsabase.dll, and the high-encryption version installs rsaenh.dll. Table 1 shows the differences between key lengths in the base and enhanced CSPs. The enhanced CSP doubles the key lengths for public-key and secret-key algorithms and adds Data Encryption Standard (DES) and Triple DES (3DES) encryption.

The CSPs have companion signature files—rsasig.dll for the base CSP and enhsig.dll for the enhanced CSP. CryptoAPI periodically validates the companion file's signature to ensure that no one has surreptitiously modified the CSP.

I need to upgrade several servers to high encryption, and they're running different service-pack levels. Where can I find the 128-bit updates?

You can easily upgrade a system to 128-bit encryption; simply download and install the 128-bit version of the current service pack. Microsoft posts high-encryption updates for NT 4.0 SP6a, SP5, and SP4 at http://www.microsoft.com/technet/security/crypload.asp. This page also contains links to high-encryption updates for Windows 2000, Microsoft Internet Explorer (IE), Outlook 2000, Win2K Terminal Server Access Client, and service-pack-specific versions of NT Server 4.0, Terminal Server Edition (WTS).

When I upgrade NT 4.0 to the high-encryption version, should I also update IE?

The answer depends on the IE version that's on your system. IE 5.5 supports 128-bit encryption, but earlier versions don't. Because of changes in US export law, Microsoft no longer distributes IE's high-encryption version on CD-ROM. To ensure compliance with government regulations for exporting 128-bit OS updates, Microsoft maintains a controlled Web page that you can reach from http://www.microsoft.com/technet/security/crypload.asp. The Web page verifies that you're downloading from a US location (including territories, possessions, and dependencies) and requires that you agree to the online export notice before you download the files you need.

To download updates to IE 5.1, 5.0, and 4.0, scroll down the page at the URL above until you see Internet Explorer and High Encryption Packs. IE 5.1, 5.0, and 4.0 updates are specific to NT 4.0 service packs, so be sure you download the file for the service-pack level you need to update. The updates contain the same files—rsaenh.dll, sch128c.dll, enhsig.dll, ie5dom.inf, advpack.dll, w95inf32.dll, and w95inf16.dll—but some DLLs vary in size according to the IE versions they're updating, ostensibly to accommodate service-pack implementation differences.

I need to verify that all systems at my location are running the high-encryption versions of NT 4.0 and IE. What's the best way to verify the encryption level?

NT 4.0. According to the update.inf file in the high-encryption update for SP6a, NT 4.0's high-encryption installation replaces four files in the system32 directory—schannel.dll, security.dll, ntlmssps.dll, and ndiswan.sys—with high-encryption files of the same name. The high-encryption version numbers for these files are

If these four files exist in the system root with these version numbers, the system you're checking is running high encryption. NT 4.0's 128-bit versions also install rsaenh.dll with version number 5.0.1877.8 and the companion signature file enhsig.dll with version number 5.0.1877.7. However, because IE high-encryption updates install the same files, the presence of these files doesn't necessarily indicate that the OS is the 128-bit version.

IE. You can verify the IE version by selecting About Internet Explorer on IE's Help menu. IE 5.5 uses high encryption by default. To verify that you're running the high-encryption version of IE 5.1 or IE 5.0, look for the file %systemroot%system32sch128c.dll. The Original Filename description on the Version tab of sch128c.dll indicates that the file replaces schannel.dll. Based on a comparison of the 128-bit downloads for NT and IE 5.0, this file isn't present in the high-encryption version of NT, so the presence of sch128c.dll uniquely identifies the high-encryption version of IE.

Another way to tell whether your system is running the high-encryption version of IE 5.1 or IE 5.0 is to look in the registry's Add/Remove Programs list for an entry that lets you uninstall the 128-bit version. Start a registry editor, and look for the entry HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionUninstall128PATCH. This entry's existence denotes that the system is running the high-encryption version.

Does the high-encryption version of NT 4.0 decrease my LAN's vulnerability to network sniffer attacks?

Yes. NT 4.0's high-encryption version enhances security for a variety of operations, including crucial communication among DCs. An important benefit of high encryption is that the OS uses a 128-bit key instead of a 40- or 56-bit key to encrypt all secure Netlogon channels. DCs use a secure Netlogon channel to create trust relationships with one another so that they can securely replicate SAM and Local Security Authority (LSA) data and coordinate account password changes.

DCs also use secure Netlogon channels to implement cross-domain trusts and pass-through authentication. NT workstations and servers create secure Netlogon channels to a DC for computer account authentication and password synchronization. If a sniffer can capture Netlogon traffic between NT systems, NT 4.0's high-encryption version requires the intruder to crack a 128-bit key to successfully decrypt all these network messages. However, the high-encryption version doesn't protect user data or email traffic.

I support a law firm's NT 4.0 system, and the firm's attorneys remotely access and update legal briefs and other key material for pending lawsuits. Does the high-encryption version improve security for remote users?

Yes. You can configure RAS and RRAS servers and RAS clients to require encryption. When you define a remote connection, one option is require encryption; when you enable encription, the server and client refuse connections that aren't encrypted. When servers and clients establish a connection, whether the connection is a dial-up Point-to-Point Protocol (PPP) or a PPTP VPN connection, both endpoints use a 128-bit key to encrypt the data they transmit. When you use a PPTP connection, the encrypted data is encapsulated inside a second network packet, which provides an additional level of security. Also, PPTP connections will fail if one endpoint is standard encryption (i.e., a 56-bit key) and the other endpoint is high encryption (i.e., a 128-bit key).

Does NT 4.0's high-encryption version provide more secure storage for user accounts and passwords?

NT 4.0 never stores a password in clear text. For security purposes, the OS hashes the password, uses another function to modify the hash value, and stores the result (a password derivative) in the SAM. The OS performs a similar operation before caching logon credentials on the local system. Because password protection depends on hashing rather than encryption, the 128-bit version doesn't provide additional protection for passwords.

However, you can introduce a second level of protection for passwords by activating the Syskey feature. Syskey, which ships with every post-SP3 system, strongly encrypts all password derivatives that the OS stores in the SAM database. This feature prevents a user with valid or illegally obtained Administrator permission from using registry programming interfaces to access password derivatives.

When you enable Syskey encryption, the utility prompts you to select one of three options for defining and storing the system key that encrypts password derivatives. You can use a machine-generated random key and a complex obfuscation algorithm to store the key on the local system; this option is the only one that permits an unattended system restart. You can select a machine-generated random key and store the system encryption key on a floppy disk. When you shut down the system, NT 4.0 won't restart until you insert the floppy disk containing the system key. For the third option, you can enter a password to generate the System Key, and NT 4.0 won't restart until you enter the password during system startup.

If you want to experiment with System Key encryption, you first need to read the Microsoft article "Windows NT System Key Permits Strong Encryption of the SAM" (http://support.microsoft.com/support/kb/articles/q143/4/75.asp). Practice on a system that you can trash at will, perform a full backup, and use the /s option to update the Emergency Repair Disk (ERD) and the security databases before you activate Syskey. If you lose the encryption key, the only way you can return the system to its previous state is to use the ERD that contains the unencrypted password database.