Troubleshooting ASP.NET using WinDbg and the SOS extension
A good developer will plan for any contingency. Naturally, part of that planning is developing some robust exception handling routines. Many ASP.NET developers will handle exceptions by presenting the user with a nicely formatted Web page, and then logging the exception information in a file for analysis later.
Scenario:You have developed and tested a new ASP.NET application that logs exceptions into a log file. You deploy the ASPX pages and the assemblies to your production Web server. When people use your application, they are seeing an error page, but no exception logging is written to your log files. You also notice that there are many ASP.NET events in the Application Event Viewer log saying that Aspnet_wp.exe stopped unexpectedly.
This is a common type of problem that we see in Developer Support, and the majority of our customers don't really know how to approach troubleshooting it. Even worse is the dreaded "Server Application Unavailable" error message. This month, I will show you how you can troubleshoot these types of problems with the Debugging Tools for Windows. Disclaimer:
If you call Microsoft Developer Support, we may often ask you to send us a crash dump file or a hang dump file. The analysis of the dump file in this article will be quite easy and fast. In many cases, the cause of a problem is not apparent and it can take many hours of analysis to get to the root cause. Don't let the ease and straight-forwardness of this dump file analysis fool you into thinking that they are all this easy. Most, in fact, are not!
Before we get started, you'll need to install the Debugging Tools for Windows. Even if you already have a copy installed, make sure that you have the latest version because we will be using an extension for managed debugging that is only included with the most recent version of the Debugging Tools for Windows.
You can download the Debugging Tools for Windows from the following Microsoft Web site:
When you install the Debugging Tools for Windows, you may want to perform a custom installation to the C:\Debuggers folder. We will be running tools from the command line, and if the path is a short one, it will make doing so easier for you. After installation is complete, you will want to download the sample ASP.NET page that will crash the worker process.
Note that in order for you to be able to generate the dump file using this page, you need to be running ASP.NET 1.1 with Service Pack 1 (SP1) installed. Without SP1, this page will run without generating a crash.
You will also want to set up the symbol path for WinDbg. Symbols are not required for function name resolution on managed assemblies, but you will need symbols for native function resolution. To set the symbol path in WinDbg:
- Open WinDbg. Click Start, point to All Programs, and then click Debugging Tools for Windows.
- On the File menu, click Symbol File Path.
- In the Symbol Path dialog box, type the following symbol path:
- Close WinDbg, and then type Yes at the prompt to save base workspace information.
Now you're ready to create the crash dump file.
Creating the crash dump file
In this walkthrough, clicking the button on the sample page will cause the ASP.NET worker process to crash. In many cases, you won't know precise steps to generate a crash. In fact, your production application may crash at some random time. As you'll soon see, it doesn't matter whether a crash occurs predictably or randomly. Creating a dump file is quite easy using Adplus, the VBScript file that automates the entire process.
To generate the dump file, you need to first browse to the sample page. (Don't click the button!) After the page has been successfully served, you know that the Aspnet_wp.exe process (or the W3wp.exe process on Windows Server 2003) has been launched. Now you're ready to attach the debugger to the worker process so that you can capture a dump file when it crashes.
To create a dump file:
- Open a new command prompt on the web server, and then change to the directory where the Debugging Tools for Windows are installed.
- Run the following command:
cscript adplus.vbs -crash -pn aspnet_wp.exe -o c:\crashdump If you are running Microsoft Windows Server 2003, run the following command:
cscript adplus.vbs -crash -pn w3wp.exe -o c:\crashdump
I won't go into great detail on Adplus.vbs here, but if you'd like more information, you can review the following Microsoft Knowledge Base article:
How to use ADPlus to troubleshoot "hangs" and "crashes"
After you run this command, you will be presented with a couple of dialog boxes. Click OK
in each dialog box. You should now see a button on your task bar for the debugger, CDB, the application name for debugger. However, it may be different in your window.Note
You must be at the console on the web server when you run Adplus in crash mode. You cannot run Adplus in crash mode remotely.
Now you're ready to generate a crash! Click the button on the page to crash the worker process. If you have JIT debugging enabled in Visual Studio .NET, a dialog will appear letting you know that an exception occurred but the attached debugger couldn't handle it. If you don't have JIT debugging enabled, the dump file will be created without any dialogs being displayed. In either case, after you see the CDB
button disappear from the task bar, the dump file has finished, and you're ready to move on to the next step.
Checking out the dump file
- Start WinDbg.
- On the File menu, click Open Crash Dump to open the dump file. (Don't click the Open button on the toolbar.)
- Browse to the C:\Crashdump folder. In that folder, you will see another folder with a long name that starts with Crash_Mode_Date. Adplus saved the dump files inside of this folder. (You should have more than one dump file in the folder.) The dump file that you want to open in WinDbg is the 2nd chance exception dump file. It should be obvious by the name of the file which dump file is the 2nd chance exception dump file.
After you've opened the dump file, WinDbg will start downloading symbols. Depending on the speed of your Internet connection, this process may take several minutes. When it's complete, you will see the command prompt appear in WinDbg, as shown in the following figure.
Figure 1 : WinDbg - The "0:000>"at the command prompt indicates the current thread, thread 0.
You can see the faulting stack by typing the kb
command at the command prompt. Here is the output from the dump file:
0:000> kbChildEBP RetAddr Args to Child 00771018 791b6173 00000643 00000000 00b59bc0 mscorwks!_EH_prolog+0x200771928 791b60c0 00771a04 00000000 00000000 mscorwks!EEClass::DoRunClassInit+0xbe0077193c 791d75a7 00771a04 00771a50 00b59bc0 mscorwks!MethodTable::CheckRunClassInit+0x1d00771a14 791d7746 00000000 0086fb88 00771a50 mscorwks!MethodDesc::DoPrestub+0x28e00771a2c 00a92f76 00771a50 0180f154 00ee5cd4 mscorwks!PreStubWorker+0x42WARNING: Frame IP not in any known module. Following frames may be wrong.00773a38 0337f2bc 00f8463c 00eb593c 00ee5cc8 0xa92f7600773a64 031cd7e4 00eb593c 0180ea7c 00f84560 0x337f2bc00773aac 033a3299 00000000 00eb593c 0180ea7c 0x31cd7e400773b50 033a3031 00000001 00000000 0180e238 0x33a329900773bec 0332ee47 00edb274 0180df00 00f8463c 0x33a303100773c28 0332e71b 0180df00 033a35c5 0180d870 0x332ee4700773cc4 033a3031 00000001 00000000 0180d2b4 0x332e71b00773d60 0332ee47 00edb274 0180cf7c 00f8463c 0x33a303100773d9c 0332e71b 0180cf7c 033a35c5 0180c8ec 0x332ee4700773e38 033a3031 00000001 00000000 0180c330 0x332e71b00773ed4 0332ee47 00edb274 0180bff8 00f8463c 0x33a303100773f10 0332e71b 0180bff8 033a35c5 0180b968 0x332ee4700773fac 033a3031 00000001 00000000 0180b3ac 0x332e71b00774048 0332ee47 00edb274 0180b074 00f8463c 0x33a303100774084 0332e71b 0180b074 033a35c5 0180a9e4 0x332ee47Note
If you are running on a multi-processor system (or a hyper-threaded system), you will see function calls in mscorsvr instead of mscorwks.
You can see the function names in Mscorwks.dll because WinDbg downloaded symbols for mscorwks. However, following the calls into mscorwks, you see no module names and no function names. That's what managed code (code that runs under the Common Language Runtime) looks like in WinDbg by default, and it's obviously not helpful output. To see what happened in that managed code, you need to use the SOS extension for WinDbg.
The SOS extension
SOS is an extension for WinDbg that allows you to debug managed code. SOS adds many commands to WinDbg, all of which are focused on debugging managed applications and many are ASP.NET-specific. The extension is a DLL file called Sos.dll, and it's located in the Clr10 folder in the folder where the Debugging Tools for Windows are installed.
To load the SOS extension, type the following command at the command prompt in WinDbg:
Don't skip the period at the beginning of the command! If all goes well, you'll be taken back to the command prompt. If an error occurs, make sure that you have the latest version of the Debugging Tools for Windows installed.
We'll cover a few of the commands from SOS to debug the crash dump file, but feel free to explore the other commands in SOS on your own time. You can use the !help
command to get a list of the commands and a brief description of each.
Digging into root cause
Now that SOS has been loaded, let's have a look at some of the commands that are useful in troubleshooting a crash dump file.
Let's look at the managed stack for the current thread, thread 0. We'll do that using the !clrstack
command. Type !clrstack
at the command prompt. WinDbg starts dumping the stack. A normal stack should dump out completely in just a few seconds. In this case, the output seems to go on forever. Go ahead and press CTRL-BREAK (or click Break
on the Debug
menu in WinDbg) and let's look at it. Here's a portion of the stack:
Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)Void System.Web.HttpServerUtility.Transfer(String,Boolean)Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)Void System.Web.UI.Page.ProcessRequestMain()Void System.Web.UI.Page.ProcessRequest()Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)Void System.Web.HttpServerUtility.Transfer(String,Boolean)Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)Void System.Web.UI.Page.ProcessRequestMain()Void System.Web.UI.Page.ProcessRequest()Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext
If you look closely, you'll see a pattern here. We start off processing the request and raising a PostBack. The OnClick event is then fired for a button. That results in a Server.Transfer. The Server.Transfer then seems to start the entire process all over again. In this case, each time this loop executes, objects are pushed onto the stack. Eventually, the system runs out of stack space, and a StackOverflowException occurs.
To get a better idea of what happened here, you can get more detailed output from the stack by running the !clrstack -s
command. This gives you the full stack. Here is some of that output. For the full output from this command, create a dump file using the sample page, and then run the command. I have edited it here because the output is quite verbose.
0:000> !clrstack -sThread 0System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean) ESP/REG Object Name 00773ac4 0180e7f4 System.EventArgs 00773ac8 00f86c14 System.Web.HttpServerUtility 00773acc 0180e238 System.Web.UI.WebControls.Button 00773ad0 0180eac0 System.Object 00773ad4 0180ea7c System.String /crash.aspx 00773ad8 00eb593c System.String c:\inetpub\wwwroot\crash.aspx 00773ae4 00f84754 System.Web.HttpResponse 00773ae8 00f846bc System.Web.HttpRequest 00773b00 00f86bec System.String crash.aspx 00773b04 00f86c14 System.Web.HttpServerUtility 00773b2c 00eef57c System.String btnCrash 00773b34 00eef57c System.String btnCrash 00773b38 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode 00773b44 00f797d8 System.String CausesValidation 00773b4c 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode00773b60 033a3031 [DEFAULT] [hasThis] Void System.Web.HttpServerUtility.Transfer(String,Boolean) ESP/REG Object Name 00773b60 0180e238 System.Web.UI.WebControls.Button00773b6c 03360398 [DEFAULT] [hasThis] Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs) ESP/REG Object Name 00773b70 0180e7f4 System.EventArgs00773b74 033a2fc5 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs) ESP/REG Object Name 00773b74 0180df00 ASP.crash_aspx 00773b78 0180e238 System.Web.UI.WebControls.Button 00773b7c 00f8601c System.Web.HttpValueCollection 00773b80 0180e314 System.ComponentModel.EventHandlerList00773b88 033a2da2 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
You can tell from this output that the call to Server.Transfer is passing crash.aspx as a parameter. How can you tell that? One indication is that the page that executes after the call is crash.aspx. However, you can also tell by looking at the HttpRequest object.
You can see the HttpRequest object in this stack, so let's have a look at that to see what was passed in that request. We'll do that using the dump object command, !do
. This command takes one parameter: the address of the object to dump. In the preceding output, the address is the second hex number in the output.
Here is the output of the HttpRequest object. I have edited this output for the sake of brevity.
0:000> !do 00f846bc Name: System.Web.HttpRequestMethodTable 0x03089170EEClass 0x02e9a1ccSize 152(0x98) bytesmdToken: 02000072 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)FieldDesc*: 030888b0 MT Field Offset Type Attr Value Name03089170 4000626 14 CLASS instance 00f8457c _path03089170 4000627 8c System.Boolean instance 0 _computePathInfo03089170 4000628 18 CLASS instance 00f8457c _filePath03089170 400062e 2c CLASS instance 00f845a4 _pathTranslated03089170 400062f 30 CLASS instance 00e6703c _baseVirtualDir03089170 4000630 34 CLASS instance 00edce64 _contentType03089170 4000631 88 System.Int32 instance 85 _contentLength03089170 400063c 60 CLASS instance 00f8601c _form
In this output, you can see the _filePath object. If you dump that using the !do
command, you will see that it is a string that contains "/crash.aspx". This confirms that the recursion is caused by crash.aspx loading recursively. Why is that happening though? Let's dig deeper.
Look back at the output from !clrstack
. If you look carefully, you'll see that the Click
event for the button is firing each time the page loads. This tells you that you are in a PostBack state each time the page loads. You can also tell from the stack that Server.Transfer is called in that Click
event. Therefore, to stop ASP.NET from crashing when this page is loaded, you will need to stop this from being a PostBack each time the page is loaded due to the Server.Transfer. Note
If you do a Server.Transfer in the ASP.NET Framework 1.1 with SP1 installed and you retain form data, it will always cause this problem. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
FIX: The Server.Transfer method causes a stack overflow and causes the ASP.NET worker process to stop responding
The Server.Transfer method has two parameters; the first is the URL to which execution should transfer, and the second is a Boolean that specifies whether or not the form data should be retained. What happens if we retain the form data? For one thing, ASP.NET will consider that a PostBack. You can, therefore, stop this crash by setting the second parameter to the Server.Transfer call to false.
Wrapping it up
This article presented a brief walkthrough of how you can use WinDbg and SOS to debug your ASP.NET applications. The purpose of this article has been to introduce you to some of the concepts and terminology involved in user-mode debugging. It is a different skill set than source code debugging, and getting good at it requires a significant time investment. It is a skill that is learned by experience.
For more experience, I recommend that you load up your ASP.NET application in a development or staging environment, and capture some hang dump files at different times. You can do that by running Adplus in hang mode by using the following command:
cscript adplus.vbs -hang -pn aspnet_wp.exe -o c:\aspnet_dump
This command will dump the aspnet_wp.exe process immediately and create a dump file in the C:\Aspnet_dump folder. A hang dump file such as this can be run from a remote connection.
Microsoft's Patterns and Practices group (PAG) has also written some excellent and in-depth articles on debugging .NET applications using WinDbg. Visit the following Microsoft Web site to view these articles:
Indeed, one of my goals in writing this article was to provide you with the foundation necessary to take advantage of the excellent articles that PAG has provided. I strongly encourage you to take advantage of them.