My consulting company received a call from a client whose email system was down. Although users could access their Inboxes, they couldn't send or receive any mail. The client was running Microsoft Exchange 2000 Server on Windows 2000. I hoped that a simple reboot would fix the problem, but the email system didn't come back up after I restarted the server.
A quick look at the Event Viewer showed multiple 7031 errors stating that services had terminated unexpectedly. The log showed 7031 errors on the following services: • IIS Admin Service (inetinfo.exe)
• Microsoft Exchange IMAP4 (inetinfo.exe)
• FTP Publishing Service (inetinfo.exe)
• Network News Transport Protocol (NNTP--inetinfo.exe)
• Microsoft Exchange POP3 (inetinfo.exe)
• Microsoft Exchange Routing Engine (inetinfo.exe)
• Simple Mail Transport Protocol (SMTP--inetinfo.exe)
• World Wide Web Publishing Service (inetinfo.exe)
While monitoring the System Event Viewer, I noticed the services would start, then suddenly crash with a 7031 error. These services would continually loop in a start, crash, start, crash cycle. Sometimes the services would run long enough to deliver a small email message, but the system wouldn't deliver any messages containing attachments. A search of the Microsoft Knowledge Base found the article "IIS and Exchange Services Quit Unexpectedly" (http://support.microsoft.com/?kbid=316612). The article describes the behavior of a server infected with the Code Red worm. The company's network had virus protection—Symantec's Norton AntiVirus Corporate Edition (NAVCE) and Norton AntiVirus for Microsoft Exchange (NAVMSE) on the server with the latest virus patterns—but to verify that it was working correctly, I ran a virus scan and the Code Red cleaner on the server. The server wasn't infected with the Code Red worm.
Because the Private and Public mail stores were mounted and users could still access their Inboxes, I suspected a Microsoft IIS problem. You probably noticed that all of the services use the same executable name – inetinfo.exe. The Microsoft article "How to Manually Restore the Metabase When No Proper Backup Exists or When the MMC Does Not Start" (http://support.microsoft.com/?kbid=234429) discusses how to manually restore a corrupt metabase. The file in %systemroot%\system32\inetsrv\metabase.bin contains the IIS configuration. If the file becomes corrupted, it can produce the problems the company was experiencing. You can view any metadata backups by starting the Internet Services Manager (ISM). Right-click the server and select Backup/Restore configuration. A window will display the backups created from the metabase.bin file. By default, metabase.bin backup files are stored in %systemroot%\system32\inetsrv\metabase.bin\metaback. I first backed up the current IIS configuration and tried to restore a metabase backup from February 4, 2004. The problem still existed after I restored the IIS configuration. You should create a backup of metabase.bin whenever you make changes to the IIS configuration. The metabase.bin file is in use when the IIS services are running, so a tape backup typically skips over this file unless it can back up the System State in Win2K. Third-party backup software such as VERITAS Software's VERITAS Backup Exec tries to restore the entire System State, which I didn't want to do in this situation. I found another article, "Exchange Databases Are Not Mounted and Event Logs Contain Error Messages" (http://support.microsoft.com/?kbid=304166), which describes how to reinstall IIS on the Exchange computer. Because the metabase.bin restoration didn't work, reinstalling IIS was the next logical step. However, doing so will destroy your IIS configuration, so make sure to document your IIS settings first, including all the non-Exchange Web site settings, FTP, NNTP, and SMTP settings. To reinstall IIS and Exchange, complete the following steps:
1. Disable all Exchange-related services.
2. Remove IIS by using the Control Panel Add/Remove Programs applet. 3. Restart the server.
4. Reinstall IIS through Add/Remove Programs. Make sure to install the SMTP and NNTP services.
5. If necessary, reinstall the latest Win2K service pack if your i386 files weren't slipstreamed with the latest service pack.
6. Insert the Exchange 2000 CD-ROM and select the reinstall option. 7. Install any Exchange 2000 service packs.
8. Recreate any IIS settings not restored by the IIS and Exchange reinstallation.
9. Reinstall NAVMSE.
After I reinstalled IIS and Exchange, the services stayed up and mail started working again. After reviewing the logs and comparing them with when the server went down, I determined that the metabase corruption might have been caused by a new virus pattern file, although I'm not sure. And I'm not sure that a more recent metabase restore would have fixed the problem, but backing up the metabase is still a good idea whenever you make a change to the IIS configuration.
Tip
Good news! Symantec has fixed the Microsoft Exchange Server 2003 freezing problem in Symantec Mail Security 4.0 for Microsoft Exchange (SMS for Exchange). In last month's column (http://www.winnetmag.com/article/articleid/41738/41738.html), I wrote about a client company that was experiencing freezing problems on a heavily loaded Exchange 2003 server running SMS for Exchange. I have clients who run the 4.0.10.461 version of Symantec Mail Security without any problems on heavily loaded servers. Symantec Platinum customers can download the latest version from Symantec's secure Web site. Symantec Gold customers can call technical support to receive access to a secure site to download the latest version.
