Read part 2 of this article here.
A developer can never have too many tools in his or her tool chest. This is especially true when troubleshooting memory allocation issues in native applifcations on Windows. Typically, processes allocate dynamic memory from one or many heaps that are local to the process. Whenever you call C runtime functions such as malloc, or whenever you instantiate a new C++ object via the new operator, memory is allocated from a heap within the process.
We've all likely encountered issues where the heap becomes corrupted. This could happen for a variety of reasons, such as code performing a random memory write that just happens to be within a heap's memory. But typically, it is something much more common, such as code overflowing or underflowing a dynamically allocation or code that uses a pointer to already freed memory. Such pointers are called dangling pointers.
In this article I want to show you how you can use Windows' built-in heap implementation, including the debug heap, to help troubleshoot memory problems both during development and in released product. The tools I'll use are the WinDbg debugger included in the Debugging Tools for Windows, which you can easily download using the web installer for the Windows SDK 7.1. I will also be using the !heap extension command available within WinDbg, among others. If you're unfamiliar with WinDbg, I highly recommend you learn this debugger as it is the native debugger and certainly the most powerful debugger for Windows. The discussion to follow is based on the code shown in Figure 1.
Dreaded Dangling Pointers
The heap code within Windows has evolved over the years, resulting in such flavors as the debug heap, the low fragmentation heap, and pageheap. Knowing a few of the nuances between the various flavors goes a long way in putting together a troubleshooting plan when you encounter problems. For this discussion, I will focus on the MyClass type as well as the intentional dangling pointer I created in main() in Figure 1, where I delete the object pointed to by the anInstance variable, then subsequently dereference it.
First, notice that MyClass contains a field called tag near the beginning of the instance footprint in memory. It provides a handy way for me to uniquely tag my object instances, so that given a particular pointer to an object within the heap, I can determine what type it may be. Of course, the tag field comes with some overhead; if many instances of your type will be dynamically allocated on the heap, the tag may not be for you. However, for types where a few instances may exist at one time, the space overhead will be low compared to the benefit of the tag.
To demonstrate, go ahead and launch the application, which I have built as HeapMagic.exe within WinDbg, set a breakpoint on the line where I delete anInstance, and execute up to that point.
Note: Since debug builds can drastically modify behavior when dealing with the heap, be sure to work with a release build of this application. Release builds, especially on x64, are tricky to debug since many of the local variables and function parameters are optimized into registers. For more information about how to effectively debug release builds on x64, check out my blog entry "x64 Manual Stack Reconstruction and Stack Walking" on our team blog site (ntdebugging).
After you reach the breakpoint, you need to find out where the anInstance object is stored, because it is probably optimized into a register. Unassembling from the current instruction pointer shows me that it is stored in r12. I know this because the x64 calling convention puts the first parameter to a function in rcx, and in this case it fills rcx with the contents of r12 before calling the delete operator, as Figure 2 shows.
How do I know that this call is about to call the delete operator? Since the call is an indirect call via a pointer, I can use the dps command to show me the symbol of the value stored in that indirect memory location, as follows:
0:000> dps 00000001`3fdd3198 L1
00000001`3fdd3198 00000000`60ce89a0 MSVCR100!operator delete
Now, I can take a look at what is in r12 by using the r command:
0:000> r r12
r12=00000000003468e0
Armed with that pointer, I can inspect a few things about the object by using !heap with the -x argument and looking at the memory. Figure 3 shows the !heap -x output. The value in the User column is the pointer value given to the client of the heap (which matches the contents of r12), whereas the pointer in the Entry column points to the heap entry header, which the internal implementation uses to manage heap allocations. You can see that the overhead is 16 bytes. Also, note that the !heap output shows that the entry is currently busy, meaning it is allocated and in use.
Viewing the actual memory contents by using the dc command shows the footprint of the object, and the tag is clearly visible, as you can see in Figure 4. What you're seeing is a result of the debug heap, where it fills the memory with 0xfeefee upon free of the memory.
Now, set a breakpoint on the point where I call SetField() on the dangling pointer and execute up to that point, then step over the SetField() call. You should see that the memory was modified and the fill pattern is tainted, as Figure 5 shows.
Now that the heap is in this state, you can validate the heap, and it will discover that you've modified the heap block after it was freed. In the !heap -x output in Figure 3, the Heap column contains the pointer to the entire heap, and you can pass that pointer to !heap -v to validate that heap. Once you do so, you should see something like the code in Figure 6, which I have abbreviated.
There's More
The process I've just described is helpful during live debugging—but it's not really very helpful when you receive crash dumps from the field. In part 2 of this series next month, I'll discuss further techniques that will help you reproduce bugs and troubleshoot behaviors related to heap memory issues.
Trey Nash is a senior escalation engineer at Microsoft working on the Windows OSs and various other products and is the author of Accelerated C# 2010 (Apress). When not working feverishly within the bowels of the OS, he is delivering training on .NET platform and kernel-mode debugging or playing ice hockey.