Diagnose Shutdown Problems with Xbootmgr

As an IT administrator, you've probably faced an issue where one particular system will not restart in a timely fashion. Perhaps you've just updated your servers with the latest security updates, and one particular server has not restarted yet. This is a common issue that can be diagnosed in a number of ways, such as using msconfig.exe to turn off all nonessential services in an effort to determine which service or driver is not responding in a timely fashion. But there should be some way to get an overall view of the shutdown statistics in a way that you can easily interpret, to help you get a bird's-eye view of how your system performs during the shutdown process.

Enter the Windows Performance Toolkit xbootmgr.exe tool, which is supported on Windows Vista, Windows 2008, Windows 7, and Windows 2008 R2. You can use xbootmgr.exe to generate a report that provides you with valuable shutdown information in an easy-to-interpret XML file—information that you can use to help in diagnosing your next shutdown problem. You can obtain the latest Windows Performance Toolkit by downloading the SDK for Windows 7 at www.microsoft.com/downloads/details.aspx?FamilyID=c17ba869-9671-4330-a63e-1fd44e0e2505&displaylang=en.

You do not have to download the entire SDK, however. You'll need only the actual WPT.exe files from this large download. To make the download faster, I recommend you perform the following steps:

1. On the Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 page, click the Download button, then click Run in the dialog box.
2. Click Next to the resulting dialog box, then click I agree to the license agreement.
3. Continue to click Next until you get to the dialog box that Figure 1 shows, which is the Installation Options page. Here you want to uncheck all the options except for Win32 Development Tools.
4. Click Next to begin installation. The installation which will take a few seconds or longer, depending on your Internet connection.

After setup is finished, you will find three files called wpt_ia64.msi, wpt_x64.msi, and wpt_x86.msi in the C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin directory.

Performing a Shutdown Trace
Once you've downloaded the WPT binaries, you can install the one that is the appropriate platform for your system. After Xperf is installed, you can now open a command prompt and navigate to the install directory, which by default is C:\Users\Admin\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Microsoft Windows Performance Toolkit. Now type the following command to produce a shutdown trace:

C:\xperf\>xbootmgr.exe -trace shutdown -traceflags
   latency+dispatcher -numruns 1 -stackwalk
   Profile+CSwitch

(Type the command on one line.) Here are the descriptions for each parameter:

• -trace shutdown: Perform a shutdown trace.
• -traceflags latency+dispatcher: Enable traceflags in the latency Kernel group, plus the dispatcher Kernel flag. You can also issue the command xperf -help providers to view definitions of the trace flags.
• -numruns 1: Set the number of Shutdown runs to 1.
• -stackwalk Profile+CSwitch: Enables stackwalking for Profile and CSwitch. This switch is available only on Windows Vista and later.

Xbootmgr will perform one initial boot after which you'll need to log back into your system. However, Xperf will perform one additional boot for each shutdown trace specified in the numruns switch. After the second boot when the tracing is finished, the Xbootmgr status window will disappear. In the previous command example, my machine rebooted twice.

After the second reboot, Xperf will automatically merge two files into one. You will need to give Xperf a few moments to merge the .etl files with the "premerge" filename into a single file. A few moments after the final reboot, you'll notice a single file called shutdown_latency+dispatcher_1.etl file in the xperf directory. Now you can export this .etl file into a shutdown XML report by issuing the following command (type on one line):

C:\xperf>xperf -i shutdown_latency+dispatcher_1.etl
   -o shutdown_demo1.xml -a shutdown

Viewing and Interpreting the Trace
Your next step is to open the shutdown_demo1.xml file in a browser. When you open the XML file, shown in Figure 2 and Figure 3, the first thing you should notice is that the output is formatted to have a node/leaf relationship, so that you can expand a single node to get more information.

In Figure 2, notice some very valuable information:

• The first node shows you that the time format is in milliseconds.
• You can also see the overall shutdown time; in my example, it's 20,761 milliseconds, or 20.76 seconds to completely shut down my system.
• The services shutdown time took 5100 milliseconds, or 5.1 seconds.
• Application Shutdown listings can be found under the perSessionInfo node. This area will list the shutdown time for each application running under each session. Again, the time is in milliseconds.

Additionally, the shutdown_demo1.xml file contains the following additional pieces of useful information, shown in Figure 3.

• You can determine which services were unresponsive—that is, any service that failed to handshake correctly with the Service Control Manager (SCM). The SCM will wait 20 seconds before terminating these services, if they fail to shut down properly. In my case, I had one service, WcesComm.exe, which failed to terminate in the allotted 20 seconds.
• You can find out how long the various phases of the shutdown took to finish.

Another piece of information you might be interested in investigating is each service's shutdown details (not shown in the figures), located under the services autoStartStartTime node. Here you can find the exact length of time that each service took to shut down.

Solve Slow Shutdowns
When using xbootmgr.exe, make sure that you enter the correct syntax to collect the trace and export it to the XML report. The reason this is key is because the error handling for Xperf isn't very helpful, so you might not have a clue as to where your typo is if you make a mistake. You might find it helpful to create a batch file for these commands, so that you don't have to type them in all the time.

Become familiar with the output of the XML file. This file contains a lot of information. You can use this article as a reference to familiarize yourself with the different sections of the XML output, to help you solve your next slow shutdown issue. As always, contact me if you have any questions about this or any of my other articles. Thanks for reading!