Troubleshooting ASP.NET using WinDbg and the SOS extension

Hello again, and welcome to the January 2005 edition of the Support Voice Column. Once again, I would like to thank Jim Cheshire, a support engineer here at Microsoft supporting ASP.NET, for his contributions. Jim has great ideas for the Support Voice column and wanted to continue to share them. Look for Jim's contributions last month and over the next month or so, and as always, send us your suggestions for future columns. Thank you Jim!

Jim has been working with Microsoft for six years on the FrontPage, VB, and ASP.NET teams. During that time, he has written for the Office Developer Center on MSDN, and he is the author of a book on FrontPage, Special Edition Using Microsoft Office FrontPage 2003. Jim also has a Web site where he provides free add-ins for FrontPage to enable Web developers to make the most out of Microsoft products. Here is that Web site address: So please, pull up a chair, kick off your shoes, and read through our column all about troubleshooting ASP.NET ,and remember, you can submit your ideas to us using the "ASK FOR IT" link included in every column we publish.
Jeremy

Troubleshooting ASP.NET using WinDbg and the SOS extension

A good developer will plan for any contingency. Naturally, part of that planning is developing some robust exception handling routines. Many ASP.NET developers will handle exceptions by presenting the user with a nicely formatted Web page, and then logging the exception information in a file for analysis later.

Scenario:

You have developed and tested a new ASP.NET application that logs exceptions into a log file. You deploy the ASPX pages and the assemblies to your production Web server. When people use your application, they are seeing an error page, but no exception logging is written to your log files. You also notice that there are many ASP.NET events in the Application Event Viewer log saying that Aspnet_wp.exe stopped unexpectedly.

This is a common type of problem that we see in Developer Support, and the majority of our customers don't really know how to approach troubleshooting it. Even worse is the dreaded "Server Application Unavailable" error message. This month, I will show you how you can troubleshoot these types of problems with the Debugging Tools for Windows.

Disclaimer: If you call Microsoft Developer Support, we may often ask you to send us a crash dump file or a hang dump file. The analysis of the dump file in this article will be quite easy and fast. In many cases, the cause of a problem is not apparent and it can take many hours of analysis to get to the root cause. Don't let the ease and straight-forwardness of this dump file analysis fool you into thinking that they are all this easy. Most, in fact, are not!

Getting prepared

Before we get started, you'll need to install the Debugging Tools for Windows. Even if you already have a copy installed, make sure that you have the latest version because we will be using an extension for managed debugging that is only included with the most recent version of the Debugging Tools for Windows.

You can download the Debugging Tools for Windows from the following Microsoft Web site: When you install the Debugging Tools for Windows, you may want to perform a custom installation to the C:\Debuggers folder. We will be running tools from the command line, and if the path is a short one, it will make doing so easier for you. After installation is complete, you will want to download the sample ASP.NET page that will crash the worker process. Note that in order for you to be able to generate the dump file using this page, you need to be running ASP.NET 1.1 with Service Pack 1 (SP1) installed. Without SP1, this page will run without generating a crash.

You will also want to set up the symbol path for WinDbg. Symbols are not required for function name resolution on managed assemblies, but you will need symbols for native function resolution. To set the symbol path in WinDbg:

1. Open WinDbg. Click Start, point to All Programs, and then click Debugging Tools for Windows.
2. On the File menu, click Symbol File Path.
3. In the Symbol Path dialog box, type the following symbol path:
   srv*c:\symbols*http://msdl.microsoft.com/download/symbols
4. Close WinDbg, and then type Yes at the prompt to save base workspace information.

Now you're ready to create the crash dump file.

Creating the crash dump file

In this walkthrough, clicking the button on the sample page will cause the ASP.NET worker process to crash. In many cases, you won't know precise steps to generate a crash. In fact, your production application may crash at some random time. As you'll soon see, it doesn't matter whether a crash occurs predictably or randomly. Creating a dump file is quite easy using Adplus, the VBScript file that automates the entire process.

To generate the dump file, you need to first browse to the sample page. (Don't click the button!) After the page has been successfully served, you know that the Aspnet_wp.exe process (or the W3wp.exe process on Windows Server 2003) has been launched. Now you're ready to attach the debugger to the worker process so that you can capture a dump file when it crashes.

To create a dump file:

1. Open a new command prompt on the web server, and then change to the directory where the Debugging Tools for Windows are installed.
2. Run the following command:
   cscript adplus.vbs -crash -pn aspnet_wp.exe -o c:\crashdump
   If you are running Microsoft Windows Server 2003, run the following command:
   cscript adplus.vbs -crash -pn w3wp.exe -o c:\crashdump

I won't go into great detail on Adplus.vbs here, but if you'd like more information, you can review the following Microsoft Knowledge Base article:After you run this command, you will be presented with a couple of dialog boxes. Click OK in each dialog box. You should now see a button on your task bar for the debugger, CDB, the application name for debugger. However, it may be different in your window.

Note You must be at the console on the web server when you run Adplus in crash mode. You cannot run Adplus in crash mode remotely.

Now you're ready to generate a crash! Click the button on the page to crash the worker process. If you have JIT debugging enabled in Visual Studio .NET, a dialog will appear letting you know that an exception occurred but the attached debugger couldn't handle it. If you don't have JIT debugging enabled, the dump file will be created without any dialogs being displayed. In either case, after you see the CDB button disappear from the task bar, the dump file has finished, and you're ready to move on to the next step.

Checking out the dump file

1. Start WinDbg.
2. On the File menu, click Open Crash Dump to open the dump file. (Don't click the Open button on the toolbar.)
3. Browse to the C:\Crashdump folder. In that folder, you will see another folder with a long name that starts with Crash_Mode_Date. Adplus saved the dump files inside of this folder. (You should have more than one dump file in the folder.) The dump file that you want to open in WinDbg is the 2nd chance exception dump file. It should be obvious by the name of the file which dump file is the 2nd chance exception dump file.

After you've opened the dump file, WinDbg will start downloading symbols. Depending on the speed of your Internet connection, this process may take several minutes. When it's complete, you will see the command prompt appear in WinDbg, as shown in the following figure.Figure 1 : WinDbg - The "0:000>"at the command prompt indicates the current thread, thread 0.

You can see the faulting stack by typing the kb command at the command prompt. Here is the output from the dump file:


Note If you are running on a multi-processor system (or a hyper-threaded system), you will see function calls in mscorsvr instead of mscorwks.

You can see the function names in Mscorwks.dll because WinDbg downloaded symbols for mscorwks. However, following the calls into mscorwks, you see no module names and no function names. That's what managed code (code that runs under the Common Language Runtime) looks like in WinDbg by default, and it's obviously not helpful output. To see what happened in that managed code, you need to use the SOS extension for WinDbg.

The SOS extension

SOS is an extension for WinDbg that allows you to debug managed code. SOS adds many commands to WinDbg, all of which are focused on debugging managed applications and many are ASP.NET-specific. The extension is a DLL file called Sos.dll, and it's located in the Clr10 folder in the folder where the Debugging Tools for Windows are installed.

To load the SOS extension, type the following command at the command prompt in WinDbg:Don't skip the period at the beginning of the command! If all goes well, you'll be taken back to the command prompt. If an error occurs, make sure that you have the latest version of the Debugging Tools for Windows installed.

We'll cover a few of the commands from SOS to debug the crash dump file, but feel free to explore the other commands in SOS on your own time. You can use the !help command to get a list of the commands and a brief description of each.

Digging into root cause

Now that SOS has been loaded, let's have a look at some of the commands that are useful in troubleshooting a crash dump file.

Let's look at the managed stack for the current thread, thread 0. We'll do that using the !clrstack command. Type !clrstack at the command prompt. WinDbg starts dumping the stack. A normal stack should dump out completely in just a few seconds. In this case, the output seems to go on forever. Go ahead and press CTRL-BREAK (or click Break on the Debugmenu in WinDbg) and let's look at it. Here's a portion of the stack: If you look closely, you'll see a pattern here. We start off processing the request and raising a PostBack. The OnClick event is then fired for a button. That results in a Server.Transfer. The Server.Transfer then seems to start the entire process all over again. In this case, each time this loop executes, objects are pushed onto the stack. Eventually, the system runs out of stack space, and a StackOverflowException occurs.

To get a better idea of what happened here, you can get more detailed output from the stack by running the !clrstack -s command. This gives you the full stack. Here is some of that output. For the full output from this command, create a dump file using the sample page, and then run the command. I have edited it here because the output is quite verbose. You can tell from this output that the call to Server.Transfer is passing crash.aspx as a parameter. How can you tell that? One indication is that the page that executes after the call is crash.aspx. However, you can also tell by looking at the HttpRequest object.

You can see the HttpRequest object in this stack, so let's have a look at that to see what was passed in that request. We'll do that using the dump object command, !do. This command takes one parameter: the address of the object to dump. In the preceding output, the address is the second hex number in the output.

Here is the output of the HttpRequest object. I have edited this output for the sake of brevity. In this output, you can see the _filePath object. If you dump that using the !do command, you will see that it is a string that contains "/crash.aspx". This confirms that the recursion is caused by crash.aspx loading recursively. Why is that happening though? Let's dig deeper.

Look back at the output from !clrstack. If you look carefully, you'll see that the Click event for the button is firing each time the page loads. This tells you that you are in a PostBack state each time the page loads. You can also tell from the stack that Server.Transfer is called in that Click event. Therefore, to stop ASP.NET from crashing when this page is loaded, you will need to stop this from being a PostBack each time the page is loaded due to the Server.Transfer.

Note If you do a Server.Transfer in the ASP.NET Framework 1.1 with SP1 installed and you retain form data, it will always cause this problem. For more information, click the following article number to view the article in the Microsoft Knowledge Base: The Server.Transfer method has two parameters; the first is the URL to which execution should transfer, and the second is a Boolean that specifies whether or not the form data should be retained. What happens if we retain the form data? For one thing, ASP.NET will consider that a PostBack. You can, therefore, stop this crash by setting the second parameter to the Server.Transfer call to false.

Wrapping it up

This article presented a brief walkthrough of how you can use WinDbg and SOS to debug your ASP.NET applications. The purpose of this article has been to introduce you to some of the concepts and terminology involved in user-mode debugging. It is a different skill set than source code debugging, and getting good at it requires a significant time investment. It is a skill that is learned by experience.
For more experience, I recommend that you load up your ASP.NET application in a development or staging environment, and capture some hang dump files at different times. You can do that by running Adplus in hang mode by using the following command:
This command will dump the aspnet_wp.exe process immediately and create a dump file in the C:\Aspnet_dump folder. A hang dump file such as this can be run from a remote connection.

Microsoft's Patterns and Practices group (PAG) has also written some excellent and in-depth articles on debugging .NET applications using WinDbg. Visit the following Microsoft Web site to view these articles: Indeed, one of my goals in writing this article was to provide you with the foundation necessary to take advantage of the excellent articles that PAG has provided. I strongly encourage you to take advantage of them.

I’ll see you back here next month when I’ll teach you how to use WinDbg and the SOS extension to analyze crash dump files. Until next time,

Jim Cheshire
Support Engineer
Microsoft Developer Support

As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base using the Ask For It (http://support.microsoft.com/common/survey.aspx?scid=sw;en;1176&p0=&p1=&p2=&p3=&p4=) form.