People make mistakes. Programmers are people (well, most of us are). Ergo, programmers make mistakes. Therefore, software has flaws. (I don't like to call these flaws "bugs" because that makes them sound like cute, harmless little critters that just squirmed in—which they aren't.) Although many companies—including Microsoft—undertake rigorous testing procedures, no software is perfect.
For commercial developers, several software practices can reduce the incidence of flaws in delivered software. Microsoft rates well on the practices front, but even Microsoft developers occasionally slip up, and no organization builds flawless software from the start.
Microsoft has an extensive product-testing program. First, Exchange Server developers test code before they submit it for a build. Next, testers test the Exchange Server build. Then, the entire company runs the release candidate (RC) version well before Microsoft releases that version to the public—a practice known as "eating your own dog food" or just "dogfooding." Dogfooding gives Microsoft (or any company that follows the procedure) firsthand experience with how the product behaves under real-world conditions. Several partnership programs also exist to put products such as Exchange 2000 Server and Whistler in the hands of enterprise customers so that they can see how the products work for them. Build tests, integration tests, and dogfooding catch a large percentage of flaws. Beta tests by users catch an additional (but surprisingly small) percentage. But when the product ships, guess what? It still contains some flaws.
Vendors often offer postrelease product fixes that deal with the worst problems, but these patches can sometimes cause as many problems as they solve. How can you find out about these flaws and fixes, and where can you locate the fixes? How can you make the best use of available patches? How do you decide when to use a fix and when to hold off? Let's examine how Microsoft corrects flaws in Exchange Server (as well as in Windows 2000 and Windows NT) and how you can take advantage of Microsoft fixes.
In a Fix
Understanding how to apply post-release fixes depends on knowing what those fixes are and how they work. You'll see two kinds of fixes for Exchange Server and Windows-family products. The first type
is called Quick Fix Engineering (QFE)—a term Microsoft now uses instead of "hotfix." The idea behind QFEs is that they fix a problem now, not later—in other words, they let you fix the problem before the next service pack release. QFEs aren't always regression tested (a fancy way of saying that although they fix one problem, you have no guarantee they won't exacerbate—or cause—another problem). Therefore, Microsoft recommends that you install a QFE only when you're experiencing the problem that it addresses.
In practice, most administrators grab as many QFEs as possible and slap the fixes on their servers. This habit isn't always the best course, but sometimes preventing a problem is better than waiting for the problem to occur. For example, suppose a QFE corrects a flaw that corrupts the Information Store (IS). You'd hardly want to sit and wait for the flaw to corrupt your store (even though the fix might cause another, lesser problem). But Microsoft doesn't test QFEs with third-party products and can't test every possible hardware and software configuration, so you should always first apply QFEs to a test system.
If a QFE is the software equivalent of an emergency patch kit for a flat tire, a service pack is the equivalent of an automaker-produced package that fixes or replaces several parts to make your car safer, more reliable—perhaps even faster. Service packs collect fixes for several (or many) problems into one unit and contain all the QFEs that Microsoft has released since the preceding service pack. For example, Exchange Server 5.5 Service Pack 4 (SP4) contains all the post-SP3 QFE releases in one easy-to-apply package. In addition, service packs are cumulative. In other words, instead of installing SP3 and SP4, you need to install only SP4, which includes the SP3 fixes and post-SP3 fixes in one bundle.
Of course, a service pack isn't truly one monolithic chunk; it's made up of files that replace or add to installed components of the OS or application to which the pack applies. The old-school approach to service packs bundled fixes and new features together in the same release, as when Microsoft introduced the Exchange Server 5.5 Move Server Wizard (MSW) in Exchange Server 5.5 SP1. But Microsoft seems to be rethinking that approach with Exchange 2000 Server and Win2K. The number of new features that the company will include in future service packs is unclear; Exchange 2000 SP1 introduced a few new features, but that trend might not continue.
Who Done It?
You understand the difference between QFEs and service packs, and which works best for a particular type of flaw. But the first step in fixing a flaw is identifying that it exists. Sometimes, Microsoft hands you this knowledge: When you download a new service pack, the release notes list all the flaws that the service pack fixes. Other times, you might suspect the presence of a flaw because your server suddenly starts acting strangely (such as throwing away incoming mail), or you might learn about the flaw from another administrator. (In "7 Steps to Using Tech Support," June 2000, I discuss how to work with Microsoft Product Support Services—PSS—and Internet peer-support resources to find out whether a particular Exchange Server behavior is a flaw, a feature, or a pilot error.)
Presuming that you've applied the current service packs for your Exchange Server and OS configuration, what do you do when you spot a suspicious behavior or hear rumors of a problem? You can call PSS, but finding out whether someone has reported the problem on the Microsoft Knowledge Base or your favorite mailing list or newsgroup is a cheaper alternative. Most of the time, a Microsoft article that documents a flaw will either include a workaround or resolution or will state that Microsoft is investigating the problem. Depending on the severity of the flaw, the article might also announce that a QFE is available. Generally, minor problems don't get QFEs because the cure might be worse than the disease. Instead, expect to see fixes for minor operational or cosmetic issues in the next service pack.
Decisions, Decisions
So, how do you decide when to apply QFEs or service packs on your servers? The situation gets tricky when you hear about a flaw and its fix rather than actually experience the problem on your systems. As I mentioned earlier, one theory states that you should apply a QFE only when you're experiencing the problem it corrects. The other theory says applying a QFE is a good preventive approach, like having regular checkups or brushing your teeth after each meal. Which plan is right? The answer depends on your problem.
I certainly don't recommend applying every QFE that you hear about; that approach is no different from swallowing a random assortment of pills from your local pharmacy, just in case you start having the symptoms the medicines treat. But consider the occasional instance in which a QFE cures a serious problem, such as throwing away all inbound SMTP mail. In a case such as this one, installing the relevant QFE as soon as possible would seem prudent.
Of course, most QFEs correct problems that aren't supercritical or widespread. For this reason, most QFEs are available only from PSS or through a support agreement. If you see Microsoft giving out a QFE directly on its FTP site, you should install the fix sooner rather than later.
What about service packs? Because Microsoft has more time to test them before release, you might think you're safe slapping a brand-new service pack onto your machine. In most cases, you are—but every once in a while, something unpleasant happens. (For example, NT 4.0 SP6 contained a fairly nasty flaw, which quickly resulted in the issuance of NT 4.0 SP6a.) Ideally, you already have a test server that closely mimics—if not exactly duplicates—your production server. If so, your test server is the best place to put a new service pack so that you can find out how the service pack works without disrupting your production environment. You'll always be tempted to install a shiny new service pack as soon as it's available, but a little caution can save you a lot of trouble. Keep an eye on Microsoft's Web site and your peer-support resources. You'll usually see a lot of discussion on newsgroups and mailing lists within the first few days after a new OS or Exchange Server service pack release. You can often determine whether a particular service pack is ready for your environment without even installing the pack.
When the time does come to apply a service pack, what should you do? Regular readers will probably guess that my first recommendation is to perform a complete backup of Exchange Server and the underlying OS. Better to be safe than sorry, whether you're installing a service pack for Exchange Server or the OS. Next, you can expand and install the service pack. The installer will offer you the option to create a backup directory in which it will store any files it replaces; this option lets you remove the service pack if necessary. Always accept this option. You can remove the backup files later if you're tight for space.
Probably the most frequently asked service pack question I hear is deceptively simple: In what order do you apply service packs? Suppose you're building a new Exchange Server 5.5 machine to replace an old system. Do you install NT 4.0, then NT 4.0 SP6a, then Exchange Server 5.5, then Exchange Server 5.5 SP4? Do you first install SP4, then SP6a? Microsoft's recommendation is simple: Always apply service packs in chronological order. If you're installing Exchange Server 5.5 on NT 4.0, first install NT 4.0 (which Microsoft released first), then Exchange Server 5.5, then NT 4.0 SP6a, then Exchange Server 5.5 SP4. This rule applies to Win2K and Exchange 2000, too, although the exact mechanics of how you install components might vary. (For example, you use the Control Panel Add/Remove Programs applet to install the SMTP service but need a separate setup application to install Exchange Server 5.5.)
You might be tempted to always have the latest and greatest code running on your production servers. But use a little discretion and forethought when you decide which QFEs or service packs to apply. Doing so can go a long way toward making your Exchange Server systems as stable as possible, which should be your ultimate goal.
