In November 2005, Microsoft released a great free tool, and most Exchange administrators probably didn't notice it. Originally called the Exchange Disaster Recovery Analyzer (ExDRA), this tool was aimed toward simplifying Exchange disaster recovery for administrators who aren't intimately familiar with database internals or command-line database tools. With the December 2006 release of Exchange Server 2007, ExDRA is no longer available as a separate download; rather, the new Database Troubleshooter is bundled into the Exchange Troubleshooting Assistant (ExTrA), which is embedded in Exchange 2007's Exchange Management Console (EMC). You can use ExTrA to troubleshoot Exchange 2000 Service Pack 3 (SP3) or later databases. (Note that if you still have a copy of ExDRA lying around, you can continue to use it on Exchange 2000 SP3 or later databases.)
With a look and feel based on the familiar Exchange Best Practices Analyzer (ExBPA) framework, the Database Troubleshooter analyzes an Exchange mailbox or public folder database and any available associated transaction log files to provide guidance on recovery feasibility and recommendations. Exchange administrators faced with database-mount problems frequently make incorrect assessments about the problem at hand, often running Eseutil or Isinteg when they don't need to, resulting in lengthy and avoidable messaging-system outages. The Database Troubleshooter seeks to demystify recoverability at the Exchange database level, letting you make sound recovery decisions without stumbling through trial and error.
Database Troubleshooter Requirements
Because the Database Troubleshooter is embedded in Exchange System Manager, you’ll need to ensure that you have all the prerequisites for Exchange 2007 in place. At the time of this writing, Microsoft requires that you're running on a machine with the Microsoft .NET Framework 2.0 and Microsoft Management Console (MMC) 3.0 installed, running either Windows Server 2003 Service Pack 1 (SP1) Standard or Enterprise Edition, Windows 2003 Release 2 (R2) Standard or Enterprise Edition, Windows 2003 X64 Standard or Enterprise Edition, or Windows 2003 x64 R2 Standard or Enterprise Edition. The Database Troubleshooter that ships with Exchange 2007 is supported when run against databases and transaction log files created in Exchange 2000 SP3 or later.
To run the Database Troubleshooter, you’ll need to have sufficient permissions to read Exchange configuration data from the Active Directory (AD) configuration naming context, and you’ll need to have read permissions to the Exchange folders housing your databases and transaction logs. The Database Troubleshooter doesn't make any changes to your Exchange servers or AD in any way, so there's no requirement for write access to either of these locations. At a minimum, then, you’ll need read permissions at the NTFS file level to the directories containing your database and transaction log files, and to the AD configuration naming context.
Currently, the Database Troubleshooter is focused solely on database and transaction-log file analysis; the tool doesn't make any changes to the files it analyzes and requires that you dismount databases prior to analysis.
Updating Configuration Files
Because the Database Troubleshooter is based on the same framework as ExBPA, the business logic is driven by an XML configuration file and the tool is auto-updatable from Microsoft’s Web site. You can launch the Database Troubleshooter by launching Exchange System Manager and navigating to Toolbox (new to Exchange 2007). You’ll see the Database Troubleshooter in the middle pane under Disaster Recovery Tools, as Figure 1 shows.
Double-clicking Database Troubleshooter launches ExTrA. When it starts, ExTrA gives you the option to always automatically check for updates or to skip the update check. I recommend that you let ExTrA automatically check for updates, unless you’re running on a machine without Internet access; in that case, you’ll want to disable automatic update checking. Either way, it’s good to understand the process that ExTrA goes through during an update check, in case you ever have to manually check for and install updates.
When ExTrA connects to the Microsoft Web site, it first checks for a new version of the tool itself. ExTrA compares Microsoft’s most current supported version with the local isntallation, then gives you the option to download a newer version, if applicable. You can check which version you have installed by clicking on the About the Exchange Troubleshooting Assistant link in ExTrA's left pane and looking at the entries for Application Version and Configuration File Version. If your version is out of date, you can download and install the new version manually.
After ExTrA confirms that the current binaries are in place, it proceeds to download a new ExTrA Master Input File (i.e., ExTRA.config.xml) and a new ExDRA Master Input File (i.e., ExDRA.config.xml) from Microsoft’s Web site. To perform a manual update, simply copy these two files to the \Program Files\Microsoft\Exchange Server\bin\en directory.
Database Analysis
When you start the Database Troubleshooter, you'll see several options listed in the screen's upper left pane, namely to Start troubleshooting and Select a result file to review. You're immediately presented with the option to label your current troubleshooting project and identify the Exchange server that you want to analyze, along with an optional domain controller (DC) server name to authenticate against, as you see in Figure 2. If you choose this route, the Database Troubleshooter looks up the Exchange server you identified in the AD configuration naming context to determine which storage groups (SGs) and databases reside on that server, along with the paths to each of these files. By default, the Database Troubleshooter uses the credentials with which you're currently logged on; however, you can click the Show advanced logon options button if you want to select alternative credentials.
Assuming your logon is valid, ExTrA prompts you to choose from a list of all SGs on the server that you have permissions to, then to choose one of the dismounted stores in that SG. The performance impact on AD throughout this operation is minimal. It’s important to note that the Database Troubleshooter will detect whether a database is mounted and will permit you to troubleshoot only dismounted stores. Much of what the tool does under the hood is equivalent to command-line Eseutil /MH and Eseutil /ML analysis, which requires that a store be dismounted. If auto-detection of Exchange databases isn’t something you’re interested in, the Database Troubleshooter lets you perform manual analysis of raw data files and simply input the paths to the Exchange database and transaction log file directories.
After you scope the database to be analyzed, the Database Troubleshooter analyzes the database and transaction log file headers and checks transaction log file integrity. The operation is entirely read-only; nothing changes in the accessed files.
Interpreting the Results
After the completion of database analysis, the Database Troubleshooter lets you choose from a number of analysis and management tasks. You can troubleshoot databases that aren’t mounting properly, analyze the space that transaction log files consume, move transaction log files to another volume, filter database-related event-log entries, and create a Recovery Storage Group. Transaction-log file analysis shows you how much free space is available for your logs and lets you move non-essential logs to a new location. However, if Local Continuous Replication (LCR) is enabled on a given database, you won’t be able to use this wizard to move log files.
Troubleshooting database-mount problems lets you see whether the database is in a Clean ShutDown state (meaning the Exchange services were stopped and all transaction log files committed, with no recovery required), as Figure 3 shows, a Dirty ShutDown state, restored from online backup, or in an unknown state (Microsoft Product Support Services—PSS—intervention required). The Database Troubleshooter will identify whether database files are missing (e.g., .stm files for Exchange 2003 and Exchange 2000, transaction log files) and provide suggestions for where to find them.
Most important, the Database Troubleshooter provides a clear step-by-step process for the recovery of an unmountable database. The tool really pays off with this functionality, particularly when you consider how many times well-meaning Exchange administrators have wasted countless hours trying to recover a database that's unrecoverable, have made incorrect assumptions about which tools and command-line switches to use to attempt recovery, or have simply given up on a perfectly recoverable database. When a production Exchange database is down, getting it back up and running is generally of paramount importance, and for that reason, the Database Troubleshooter's expert guidance is invaluable.
The Database Troubleshooter gives you a reports list that should look familiar if you use other Exchange Analyzer tools, such as ExBPA. As Figure 4 shows, the default List Reports format lets you choose whether to view Critical Issues or All Issues. For critical issues, you can click on the Tell me more about this issue and how to resolve it link to launch an informative Help article that explains the meaning of the error in depth. Common errors the Database Troubleshooter detects include signature discrepancies between EDB files and transaction log files (i.e., through an unsuccessful recovery operation), and missing required or optional transaction log files. You can arrange your list report by issue, class of issue, or severity of issue (i.e., error or warning). The Database Troubleshooter also provides a Tree Reports view that contains all the information you (or Microsoft) would ever need to troubleshoot your database. This detailed view is useful for advanced Exchange administrators, whereas the summary view is adequate for most troubleshooting purposes.
In writing this article, I used a database that was consistent (i.e., clean shutdown); however, had the database been inconsistent, I could have taken a look at the Database Troubleshooter's detailed tree report, which Figure 5 shows, to determine the minimum required log file (i.e., MinLogRequired) that I would need to start replaying to bring this database to consistency, as well as the maximum required log file (i.e., MaxLogRequired). Replaying just these log files would be sufficient to get my database to a consistent and mountable state. In the case of a clean shutdown, no transaction log files need to be replayed because all data has been committed to the database. So, if you have a clean database that can't mount, you’ll need to look for corruption in the database itself. If I had more log files, however, I could optionally replay any available sequential log file up to and including the most recent log file (i.e., HighestExistingSequenceNumber and HighestExistingSequenceNumberFileName) to get all possible data back and bring the database to the most recent state.
After you use the Database Troubleshooter for a while, you’ll notice that it maintains a history of all reports that have been run, accessible through the View a report option in the left panel. You can also export reports in comma-separated value (CSV), HTML, or XML format. Finally, the Database Troubleshooter maintains a log of all actions performed in the tool, accessible in the Other Reports section of the reporting page.
Looking Ahead
As I mentioned, Exchange 2007's Database Troubleshooter doesn’t make any changes to your Exchange databases. As you look more closely at the ESM's Toolbox, you’ll start to recognize some of the other former Exchange Analyzer tools—namely, the Exchange Performance Troubleshooting Analyzer (ExPTA), which has been rebranded the Performance Troubleshooter, and the relatively new Exchange Mailflow Analyzer (ExMFA), which has been rebranded the Mail Flow Troubleshooter. Similar to the Database Troubleshooter, both of these tools now launch ExTrA. ExBPA will continue to focus on proactive analysis of Exchange environments, whereas ExTrA will focus on more reactive troubleshooting scenarios.
Think of ExTrA as Exchange's underlying troubleshooting framework. Either way, the Database Troubleshooter promises to continue to play a vital role in assisting companies who need their Exchange databases back up when things have gone wrong.
