Article ID: 892277 - View products that this article applies to.
ASP .NET Support Voice Column
Troubleshooting ASP.NET using WinDbg and the SOS extensionTo customize this column to your needs, we want to invite you to submit your ideas about topics that interest you and issues that you want to see addressed in future Knowledge Base articles and Support Voice columns. You can submit your ideas and feedback using the Ask For It
(http://support.microsoft.com/common/survey.aspx?scid=sw;en;1176&p0=&p1=&p2=&p3=&p4=)form. There's also a link to the form at the bottom of this column.
Hello again, and welcome to the January 2005 edition of the Support Voice Column. Once again, I would like to thank Jim Cheshire, a support engineer here at Microsoft supporting ASP.NET, for his contributions. Jim has great ideas for the Support Voice column and wanted to continue to share them. Look for Jim's contributions last month and over the next month or so, and as always, send us your suggestions for future columns. Thank you Jim!
Jim has been working with Microsoft for six years on the FrontPage, VB, and ASP.NET teams. During that time, he has written for the Office Developer Center on MSDN, and he is the author of a book on FrontPage, Special Edition Using Microsoft Office FrontPage 2003. Jim also has a Web site where he provides free add-ins for FrontPage to enable Web developers to make the most out of Microsoft products. Here is that Web site address:
http://www.jimcosoftware.comSo please, pull up a chair, kick off your shoes, and read through our column all about troubleshooting ASP.NET ,and remember, you can submit your ideas to us using the "ASK FOR IT" link included in every column we publish.
Troubleshooting ASP.NET using WinDbg and the SOS extension
A good developer will plan for any contingency. Naturally, part of that planning is developing some robust exception handling routines. Many ASP.NET developers will handle exceptions by presenting the user with a nicely formatted Web page, and then logging the exception information in a file for analysis later.
Scenario:You have developed and tested a new ASP.NET application that logs exceptions into a log file. You deploy the ASPX pages and the assemblies to your production Web server. When people use your application, they are seeing an error page, but no exception logging is written to your log files. You also notice that there are many ASP.NET events in the Application Event Viewer log saying that Aspnet_wp.exe stopped unexpectedly.
This is a common type of problem that we see in Developer Support, and the majority of our customers don't really know how to approach troubleshooting it. Even worse is the dreaded "Server Application Unavailable" error message. This month, I will show you how you can troubleshoot these types of problems with the Debugging Tools for Windows.
Disclaimer: If you call Microsoft Developer Support, we may often ask you to send us a crash dump file or a hang dump file. The analysis of the dump file in this article will be quite easy and fast. In many cases, the cause of a problem is not apparent and it can take many hours of analysis to get to the root cause. Don't let the ease and straight-forwardness of this dump file analysis fool you into thinking that they are all this easy. Most, in fact, are not!
Getting preparedBefore we get started, you'll need to install the Debugging Tools for Windows. Even if you already have a copy installed, make sure that you have the latest version because we will be using an extension for managed debugging that is only included with the most recent version of the Debugging Tools for Windows.
You can download the Debugging Tools for Windows from the following Microsoft Web site:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspxWhen you install the Debugging Tools for Windows, you may want to perform a custom installation to the C:\Debuggers folder. We will be running tools from the command line, and if the path is a short one, it will make doing so easier for you. After installation is complete, you will want to download the sample ASP.NET page that will crash the worker process.
http://download.microsoft.com/download/c/a/1/ca1c6329-3b3f-4d9e-a76d-54c78d3ddfbc/crash.exeNote that in order for you to be able to generate the dump file using this page, you need to be running ASP.NET 1.1 with Service Pack 1 (SP1) installed. Without SP1, this page will run without generating a crash.
You will also want to set up the symbol path for WinDbg. Symbols are not required for function name resolution on managed assemblies, but you will need symbols for native function resolution. To set the symbol path in WinDbg:
Creating the crash dump fileIn this walkthrough, clicking the button on the sample page will cause the ASP.NET worker process to crash. In many cases, you won't know precise steps to generate a crash. In fact, your production application may crash at some random time. As you'll soon see, it doesn't matter whether a crash occurs predictably or randomly. Creating a dump file is quite easy using Adplus, the VBScript file that automates the entire process.
To generate the dump file, you need to first browse to the sample page. (Don't click the button!) After the page has been successfully served, you know that the Aspnet_wp.exe process (or the W3wp.exe process on Windows Server 2003) has been launched. Now you're ready to attach the debugger to the worker process so that you can capture a dump file when it crashes.
To create a dump file:
286350After you run this command, you will be presented with a couple of dialog boxes. Click OK in each dialog box. You should now see a button on your task bar for the debugger, CDB, the application name for debugger. However, it may be different in your window.
(http://support.microsoft.com/kb/286350/ )How to use ADPlus to troubleshoot "hangs" and "crashes"
Note You must be at the console on the web server when you run Adplus in crash mode. You cannot run Adplus in crash mode remotely.
Now you're ready to generate a crash! Click the button on the page to crash the worker process. If you have JIT debugging enabled in Visual Studio .NET, a dialog will appear letting you know that an exception occurred but the attached debugger couldn't handle it. If you don't have JIT debugging enabled, the dump file will be created without any dialogs being displayed. In either case, after you see the CDB button disappear from the task bar, the dump file has finished, and you're ready to move on to the next step.
Checking out the dump file
Figure 1 : WinDbg - The "0:000>"at the command prompt indicates the current thread, thread 0.
Collapse this imageExpand this image
You can see the faulting stack by typing the kb command at the command prompt. Here is the output from the dump file:
0:000> kb ChildEBP RetAddr Args to Child 00771018 791b6173 00000643 00000000 00b59bc0 mscorwks!_EH_prolog+0x2 00771928 791b60c0 00771a04 00000000 00000000 mscorwks!EEClass::DoRunClassInit+0xbe 0077193c 791d75a7 00771a04 00771a50 00b59bc0 mscorwks!MethodTable::CheckRunClassInit+0x1d 00771a14 791d7746 00000000 0086fb88 00771a50 mscorwks!MethodDesc::DoPrestub+0x28e 00771a2c 00a92f76 00771a50 0180f154 00ee5cd4 mscorwks!PreStubWorker+0x42 WARNING: Frame IP not in any known module. Following frames may be wrong. 00773a38 0337f2bc 00f8463c 00eb593c 00ee5cc8 0xa92f76 00773a64 031cd7e4 00eb593c 0180ea7c 00f84560 0x337f2bc 00773aac 033a3299 00000000 00eb593c 0180ea7c 0x31cd7e4 00773b50 033a3031 00000001 00000000 0180e238 0x33a3299 00773bec 0332ee47 00edb274 0180df00 00f8463c 0x33a3031 00773c28 0332e71b 0180df00 033a35c5 0180d870 0x332ee47 00773cc4 033a3031 00000001 00000000 0180d2b4 0x332e71b 00773d60 0332ee47 00edb274 0180cf7c 00f8463c 0x33a3031 00773d9c 0332e71b 0180cf7c 033a35c5 0180c8ec 0x332ee47 00773e38 033a3031 00000001 00000000 0180c330 0x332e71b 00773ed4 0332ee47 00edb274 0180bff8 00f8463c 0x33a3031 00773f10 0332e71b 0180bff8 033a35c5 0180b968 0x332ee47 00773fac 033a3031 00000001 00000000 0180b3ac 0x332e71b 00774048 0332ee47 00edb274 0180b074 00f8463c 0x33a3031 00774084 0332e71b 0180b074 033a35c5 0180a9e4 0x332ee47
Note If you are running on a multi-processor system (or a hyper-threaded system), you will see function calls in mscorsvr instead of mscorwks.
You can see the function names in Mscorwks.dll because WinDbg downloaded symbols for mscorwks. However, following the calls into mscorwks, you see no module names and no function names. That's what managed code (code that runs under the Common Language Runtime) looks like in WinDbg by default, and it's obviously not helpful output. To see what happened in that managed code, you need to use the SOS extension for WinDbg.
The SOS extensionSOS is an extension for WinDbg that allows you to debug managed code. SOS adds many commands to WinDbg, all of which are focused on debugging managed applications and many are ASP.NET-specific. The extension is a DLL file called Sos.dll, and it's located in the Clr10 folder in the folder where the Debugging Tools for Windows are installed.
To load the SOS extension, type the following command at the command prompt in WinDbg:
.load clr10\sosDon't skip the period at the beginning of the command! If all goes well, you'll be taken back to the command prompt. If an error occurs, make sure that you have the latest version of the Debugging Tools for Windows installed.
We'll cover a few of the commands from SOS to debug the crash dump file, but feel free to explore the other commands in SOS on your own time. You can use the !help command to get a list of the commands and a brief description of each.
Digging into root causeNow that SOS has been loaded, let's have a look at some of the commands that are useful in troubleshooting a crash dump file.
Let's look at the managed stack for the current thread, thread 0. We'll do that using the !clrstack command. Type !clrstack at the command prompt. WinDbg starts dumping the stack. A normal stack should dump out completely in just a few seconds. In this case, the output seems to go on forever. Go ahead and press CTRL-BREAK (or click Break on the Debugmenu in WinDbg) and let's look at it. Here's a portion of the stack:
If you look closely, you'll see a pattern here. We start off processing the request and raising a PostBack. The OnClick event is then fired for a button. That results in a Server.Transfer. The Server.Transfer then seems to start the entire process all over again. In this case, each time this loop executes, objects are pushed onto the stack. Eventually, the system runs out of stack space, and a StackOverflowException occurs.
To get a better idea of what happened here, you can get more detailed output from the stack by running the !clrstack -s command. This gives you the full stack. Here is some of that output. For the full output from this command, create a dump file using the sample page, and then run the command. I have edited it here because the output is quite verbose.
You can tell from this output that the call to Server.Transfer is passing crash.aspx as a parameter. How can you tell that? One indication is that the page that executes after the call is crash.aspx. However, you can also tell by looking at the HttpRequest object.
You can see the HttpRequest object in this stack, so let's have a look at that to see what was passed in that request. We'll do that using the dump object command, !do. This command takes one parameter: the address of the object to dump. In the preceding output, the address is the second hex number in the output.
Here is the output of the HttpRequest object. I have edited this output for the sake of brevity.
In this output, you can see the _filePath object. If you dump that using the !do command, you will see that it is a string that contains "/crash.aspx". This confirms that the recursion is caused by crash.aspx loading recursively. Why is that happening though? Let's dig deeper.
Look back at the output from !clrstack. If you look carefully, you'll see that the Click event for the button is firing each time the page loads. This tells you that you are in a PostBack state each time the page loads. You can also tell from the stack that Server.Transfer is called in that Click event. Therefore, to stop ASP.NET from crashing when this page is loaded, you will need to stop this from being a PostBack each time the page is loaded due to the Server.Transfer.
Note If you do a Server.Transfer in the ASP.NET Framework 1.1 with SP1 installed and you retain form data, it will always cause this problem. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
839521The Server.Transfer method has two parameters; the first is the URL to which execution should transfer, and the second is a Boolean that specifies whether or not the form data should be retained. What happens if we retain the form data? For one thing, ASP.NET will consider that a PostBack. You can, therefore, stop this crash by setting the second parameter to the Server.Transfer call to false.
(http://support.microsoft.com/kb/839521/ )FIX: The Server.Transfer method causes a stack overflow and causes the ASP.NET worker process to stop responding
Wrapping it upThis article presented a brief walkthrough of how you can use WinDbg and SOS to debug your ASP.NET applications. The purpose of this article has been to introduce you to some of the concepts and terminology involved in user-mode debugging. It is a different skill set than source code debugging, and getting good at it requires a significant time investment. It is a skill that is learned by experience.
For more experience, I recommend that you load up your ASP.NET application in a development or staging environment, and capture some hang dump files at different times. You can do that by running Adplus in hang mode by using the following command:
cscript adplus.vbs -hang -pn aspnet_wp.exe -o c:\aspnet_dump
This command will dump the aspnet_wp.exe process immediately and create a dump file in the C:\Aspnet_dump folder. A hang dump file such as this can be run from a remote connection.
Microsoft's Patterns and Practices group (PAG) has also written some excellent and in-depth articles on debugging .NET applications using WinDbg. Visit the following Microsoft Web site to view these articles:
http://msdn2.microsoft.com/en-us/library/ms954594.aspxIndeed, one of my goals in writing this article was to provide you with the foundation necessary to take advantage of the excellent articles that PAG has provided. I strongly encourage you to take advantage of them.
I’ll see you back here next month when I’ll teach you how to use WinDbg and the SOS extension to analyze crash dump files. Until next time,
Microsoft Developer Support
As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base using the Ask For It