\[Author's Note: Last month, I explained how to download and install the Windows 32-bit debugging tools from http://www.microsoft.com/ddk/debugging. You need those tools to perform the tasks and examples I provide this month. Also, you must have symbols installed. (See "Starting the Troubleshooting Process," June 2001.)\]
Over the past 2 months, I've been setting up the theory necessary to start troubleshooting IIS problems. This month, I show you how to use the Windbg debugging tool to gather information about your servers.
The Microsoft 32-bit debugging tools that you installed give you several debugging tools, such as Command Debugger (Cdb), Kernel Debugger (KD), and Windows Debugger (Windbg), and support tools. As I go through different scenarios in future articles, I'll point out which tools are most appropriate and refer you to the online Help file that comes with the debugging tool.
Preparing to Use Windbg
To begin using Windbg, you need to create a dump (.dmp) file that you can load and look at. On the machine on which you've installed the debugging tools, open a command window, then switch to the directory in which you installed the debugger (e.g. C:\dbg). Type the command
cscript.exe adplus.vbs hang iis
This command causes the Autodump Plus tool, which is bundled with the debugging tools you downloaded, to produce full-memory dumps for the inetinfo.exe process and all Out-Of-Process (OOP) applications that are running. You'll use the dump file simply as a sample file, but this process is also an example of how to create dump files when your production site is hung. Dump files appear in a subfolder of the debugger folder with a name such as C:\dbg\Normal_Hang_Mode__Date_04-17-2001__Time_12-39-30PM.
To start Windbg, choose Start, Programs, Debugging Tools For Windows, then select Windbg. Because you haven't used this tool before, you need to set up the program so it knows where your symbol files reside. To tell Windbg about the symbol path you're using, choose File, Symbol File Path, then add the fully qualified path to your symbol file root (e.g., C:\winnt\symbols). Now, when you load a dump file or attach Windbg to a running process, Windbg will know where to find symbolic information.
To open the dump file, from Windbg, choose File, Open Crash Dump. Select the dump file you used Autodump Plus to create (e.g., PID-1188__Inetinfo.exe__Date_04-17-2001__Time_12-39-30PM__full.dmp). When a dialog box asking to Save Workspace Information appears, click Yes. You now have two windows within Windbg. The top window is labeled Disassembly and looks like a lot of assembler code. Because I'm not going to explain the process in that much depth, close the top window. Your screen will look like the one that Figure 1 shows.
The top pane shows the output from Windbg. The bottom pane, which is preceded by 0:000>, is where you input commands. This set of numbers is the prompt area. Whenever a command is running, the prompt window is blank. When the command finishes, the numbers reappear. The number to the left of the colon (:) identifies the processor on which the current thread in Windbg was running. The number to the right of the colon indicates what number thread in the dump file is the currently active thread. (For information about the terminology I'm using, see the chapter "Terminology" in the debugger online Help file Introduction to Debugging.) Because you made the dump file manually, Windbg sets the active thread to 0. If the dump file had been generated as the result of a crash, the tool would have set the active thread to the thread that caused the crash.
Examining a Dump File
Now that you've created a dump file and loaded it into Windbg, let's dig into it and see what IIS was doing. Type kb, which stands for Stack Backtrace, in the prompt window, then press Enter. Output similar to that in Figure 2 appears. This output tells you what the active thread was doing at the time the dump was created. The output in Figure 2 shows five columns to the left of the function names. These columns are
- ChildEBP (column 1)
- RetAddr (column 2)
- Args to Child (columns 3 through 5)
(For now, don't worry about what these five columns mean. I'll explain the columns and functions as I use them in specific scenarios in future articles.)
Look at the functions that have been called. The top function (i.e., ntdll!ZwReadFile) is the most recently executed function. This function should be the top item for thread 0 in any dump of the inetinfo.exe process. (Note that other threads in inetinfo.exe and all threads in other processes will most likely have different functions on top.) You should see a stack similar to the one in Figure 2 on any dump file you generate from inetinfo.exe. If you don't see such a stack, you might have received a symbol mismatch error, which means that the symbols weren't lined up properly (i.e., the loaded symbol file doesn't match the module that it's supposed to represent). You might see some variations in thread 0, but the stack should end up with ZwReadFile as the top function.
However, what if you wanted to see what all the threads were doing? You can type the command
~*kb
in the prompt window, where the tilde (~) means run the command on a thread different from the current one, the asterisk (*) means that you want all threads, and Kb is the command that should be run on all threads. Depending on where your symbols reside, this command can take a while to execute. Be patient, and wait for the prompt to return to the prompt window. When the command finishes running, browse through the results. Can you tell what the threads were doing based on the function names you get back? Figure 3 shows one thread that comes up repeatedly in dump files.
The stack in Figure 3 is the most important stack to remember because it shows that the Asynchronous Thread Queue (ATQ) has at least one thread (the current thread) listening for an incoming request, which means that IIS can handle a static request (e.g., .htm, .jpg). For information about the ATQ, see "Diagnosing Problems in IIS," May 2001. If your server appears to be hung completely (i.e., it serves no files) and you see this thread stack in a dump file taken from the server at the time of the hang, you can be sure that incoming requests aren't making it to IIS and that a network or connectivity problem is occurring.
Here's another basic command with which you should be familiar. Type the command
!sympath
in the prompt window, then press Enter. You'll receive a response showing the path to your symbol files. If you want to change the path or add another path, you can do so by typing the !sympath command followed by the paths you want. You just separate multiple paths by a semicolon (;). After you've entered a new symbol path, you need to make the current dump file start reading from the new path. To make the dump file read the new path, type
.reload
in the prompt window. Notice the period (.) at the beginning of the command. You must include the period for the command to work.
Another useful command is lm, which lists all loaded modules in memory. With this list, you can see exactly which modules are running. This list often shows modules that you thought had been removed (e.g., Internet Server API—ISAPI—filters). Figure 4, page 3, shows sample (incomplete) output from running the lm command on the inetinfo.exe dump file.
Notice that you get a start column, an end column, a module name column, and a column that displays information about the symbol file for the module. Note that this list doesn't tell you whether the module is an .exe file, a .dll file, and so on. The start column tells you where in memory the module is loaded. The end column tells you where the module's code stops. By looking at the list in Figure 4, the VBScript entry tells you that IIS has loaded an Active Server Pages (ASP) page that contains VBScript. The msjava entry is the runtime for Java applications. You also know that you're looking at the in-process IIS dump because inetinfo.exe is loaded. That is, if a module appears in the list, you know that it was loaded in this process. User mode dumps (such as the one in this example) show information only about the one process in which the dump was created. Thus, you know that you're looking at the inetinfo.exe process because this process is the only one that contains this module in the list of loaded modules.
You can also gather information about specific modules. If you want more information about a specific module, type the command
!lmi <module name>
in the prompt window. You'll receive header information about the module, including the date and time, which can often tell you whether you're running an older version of a program and need to upgrade. Figure 5 shows details about the VBScript module.
As you can see in Figure 5, the date for vbscript.dll is June 9, 2000. A security patch dated March 2001 exists for Windows Script Host (WSH) 5.5 and WSH 5.1. (Note that VBScript ships with WSH, and I recommend updating vbscript.dll by installing the latest version of WSH.) See the Microsoft article "IE can Divulge Location of Cached Content" at http://www.microsoft.com/technet/security/bulletin/ms01-015.asp.) I need to apply this fix because my DLL isn't up-to-date. The sidebar "Common Windbg Commands," page 3, lists other commands that you can use on your dump files.
Here's a command that's a bit tricky, but you'll get the hang of it after you've run it a few times. (The output from this command, which Figure 6 shows, is long; I've omitted quite a bit.) I'll point out the things you want to look for. To find out what scripts are currently running, type
0:000> dt asp!g_ScriptManager r
in the prompt window. Three linked lists contain the information that you need to determine which scripts are running:
- m_htFSQ—contains the list of scripts that are cached but not currently in use (callout A in Figure 6)
- m_htRSL—contains the list of running scripts (callout B)
- m_hTPLL—contains the list of script engines (callout C)
Next Month
Many other commands are available, and you can learn much more from dump files than I show you here. As I present scenarios in future articles, I'll point out what commands are necessary to retrieve the data you want. For now, experiment with the commands listed in the Help file and in the sidebar, and see what you can learn from them. When this column resumes in the September issue, I'll jump into the first scenario and work through it to resolution.
Note: To keep the Disassembly window from popping up as you work, choose Automatically Open Disassembly from the Window menu to prevent Windbg from reopening the window during the current Windbg session. Unfortunately, Windbg doesn't remember the Automatically Open Disassembly setting when you close the tool, so you need to disable the setting each time you open Windbg.
