Article ID: 892277 - Last Review: May 18, 2007 - Revision: 2.4
Troubleshooting ASP.NET using WinDbg and the SOS extension
ASP .NET Support Voice Column
Troubleshooting ASP.NET using WinDbg and the SOS extension
To customize this column to your needs, we want to invite you to submit your ideas about topics that interest you and issues that you want to see addressed in future Knowledge Base articles and Support Voice columns. You can submit your ideas and feedback using the Ask For It (http://support.microsoft.com/common/survey.aspx?scid=sw;en;1176&p0=&p1=&p2=&p3=&p4=) form. There's also a link to the form at the bottom of this column.On This Page
Hello again, and welcome to the January 2005 edition of the
Support Voice Column. Once again, I would like to thank Jim Cheshire, a support
engineer here at Microsoft supporting ASP.NET, for his contributions. Jim has
great ideas for the Support Voice column and wanted to continue to share them.
Look for Jim's contributions last month and over the next month or so, and as
always, send us your suggestions for future columns. Thank you Jim!
Jim has been working with Microsoft for six years on the FrontPage, VB, and ASP.NET teams. During that time, he has written for the Office Developer Center on MSDN, and he is the author of a book on FrontPage, Special Edition Using Microsoft Office FrontPage 2003. Jim also has a Web site where he provides free add-ins for FrontPage to enable Web developers to make the most out of Microsoft products. Here is that Web site address:
Jeremy
Jim has been working with Microsoft for six years on the FrontPage, VB, and ASP.NET teams. During that time, he has written for the Office Developer Center on MSDN, and he is the author of a book on FrontPage, Special Edition Using Microsoft Office FrontPage 2003. Jim also has a Web site where he provides free add-ins for FrontPage to enable Web developers to make the most out of Microsoft products. Here is that Web site address:
http://www.jimcosoftware.com
(http://www.jimcosoftware.com)
So please, pull up a chair, kick off your shoes, and read through
our column all about troubleshooting ASP.NET ,and remember, you can submit your
ideas to us using the "ASK FOR IT" link included in every column we publish.
Jeremy
Back to the topTroubleshooting ASP.NET using WinDbg and the SOS extension
A good developer will plan for any contingency. Naturally, part of that planning is developing some robust exception handling routines. Many ASP.NET developers will handle exceptions by presenting the user with a nicely formatted Web page, and then logging the exception information in a file for analysis later.
Scenario:
You have developed and tested a new ASP.NET application that logs exceptions into a log file. You deploy the ASPX pages and the assemblies to your production Web server. When people use your application, they are seeing an error page, but no exception logging is written to your log files. You also notice that there are many ASP.NET events in the Application Event Viewer log saying that Aspnet_wp.exe stopped unexpectedly.This is a common type of problem that we see in Developer Support, and the majority of our customers don't really know how to approach troubleshooting it. Even worse is the dreaded "Server Application Unavailable" error message. This month, I will show you how you can troubleshoot these types of problems with the Debugging Tools for Windows.
Disclaimer: If you call Microsoft Developer Support, we may often ask you to send us a crash dump file or a hang dump file. The analysis of the dump file in this article will be quite easy and fast. In many cases, the cause of a problem is not apparent and it can take many hours of analysis to get to the root cause. Don't let the ease and straight-forwardness of this dump file analysis fool you into thinking that they are all this easy. Most, in fact, are not!
Getting prepared
Before we get started, you'll need to install the Debugging Tools for Windows. Even if you already have a copy installed, make sure that you have the latest version because we will be using an extension for managed debugging that is only included with the most recent version of the Debugging Tools for Windows.You can download the Debugging Tools for Windows from the following Microsoft Web site:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
(http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx)
When you install the Debugging Tools for Windows, you may want to
perform a custom installation to the C:\Debuggers folder. We will be running
tools from the command line, and if the path is a short one, it will make doing
so easier for you. After installation is complete, you will want to download
the sample ASP.NET page that will crash the worker process. http://download.microsoft.com/download/c/a/1/ca1c6329-3b3f-4d9e-a76d-54c78d3ddfbc/crash.exe
(http://download.microsoft.com/download/c/a/1/ca1c6329-3b3f-4d9e-a76d-54c78d3ddfbc/crash.exe)
Note that in order for you to be able to generate the dump file
using this page, you need to be running ASP.NET 1.1 with Service Pack 1 (SP1)
installed. Without SP1, this page will run without generating a
crash.You will also want to set up the symbol path for WinDbg. Symbols are not required for function name resolution on managed assemblies, but you will need symbols for native function resolution. To set the symbol path in WinDbg:
- Open WinDbg. Click Start, point to All Programs, and then click Debugging Tools for Windows.
- On the File menu, click Symbol File Path.
- In the Symbol Path dialog box, type the
following symbol path: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
- Close WinDbg, and then type Yes at the prompt to save base workspace information.
Creating the crash dump file
In this walkthrough, clicking the button on the sample page will cause the ASP.NET worker process to crash. In many cases, you won't know precise steps to generate a crash. In fact, your production application may crash at some random time. As you'll soon see, it doesn't matter whether a crash occurs predictably or randomly. Creating a dump file is quite easy using Adplus, the VBScript file that automates the entire process.To generate the dump file, you need to first browse to the sample page. (Don't click the button!) After the page has been successfully served, you know that the Aspnet_wp.exe process (or the W3wp.exe process on Windows Server 2003) has been launched. Now you're ready to attach the debugger to the worker process so that you can capture a dump file when it crashes.
To create a dump file:
- Open a new command prompt on the web server, and then change to the directory where the Debugging Tools for Windows are installed.
- Run the following command: cscript adplus.vbs -crash -pn aspnet_wp.exe -o c:\crashdumpIf you are running Microsoft Windows Server 2003, run the following command:cscript adplus.vbs -crash -pn w3wp.exe -o c:\crashdump
286350
(http://support.microsoft.com/kb/286350/
)
How to use ADPlus to troubleshoot "hangs" and "crashes"
After you run this command, you will be presented
with a couple of dialog boxes. Click OK in each dialog box.
You should now see a button on your task bar for the debugger, CDB, the
application name for debugger. However, it may be different in your
window.Note You must be at the console on the web server when you run Adplus in crash mode. You cannot run Adplus in crash mode remotely.
Now you're ready to generate a crash! Click the button on the page to crash the worker process. If you have JIT debugging enabled in Visual Studio .NET, a dialog will appear letting you know that an exception occurred but the attached debugger couldn't handle it. If you don't have JIT debugging enabled, the dump file will be created without any dialogs being displayed. In either case, after you see the CDB button disappear from the task bar, the dump file has finished, and you're ready to move on to the next step.
Checking out the dump file
- Start WinDbg.
- On the File menu, click Open Crash Dump to open the dump file. (Don't click the Open button on the toolbar.)
- Browse to the C:\Crashdump folder. In that folder, you will see another folder with a long name that starts with Crash_Mode_Date. Adplus saved the dump files inside of this folder. (You should have more than one dump file in the folder.) The dump file that you want to open in WinDbg is the 2nd chance exception dump file. It should be obvious by the name of the file which dump file is the 2nd chance exception dump file.
Collapse this image
Click to expand this image
You can see the faulting stack by typing the kb command at the command prompt. Here is the output from the dump file:
0:000> kb ChildEBP RetAddr Args to Child 00771018 791b6173 00000643 00000000 00b59bc0 mscorwks!_EH_prolog+0x2 00771928 791b60c0 00771a04 00000000 00000000 mscorwks!EEClass::DoRunClassInit+0xbe 0077193c 791d75a7 00771a04 00771a50 00b59bc0 mscorwks!MethodTable::CheckRunClassInit+0x1d 00771a14 791d7746 00000000 0086fb88 00771a50 mscorwks!MethodDesc::DoPrestub+0x28e 00771a2c 00a92f76 00771a50 0180f154 00ee5cd4 mscorwks!PreStubWorker+0x42 WARNING: Frame IP not in any known module. Following frames may be wrong. 00773a38 0337f2bc 00f8463c 00eb593c 00ee5cc8 0xa92f76 00773a64 031cd7e4 00eb593c 0180ea7c 00f84560 0x337f2bc 00773aac 033a3299 00000000 00eb593c 0180ea7c 0x31cd7e4 00773b50 033a3031 00000001 00000000 0180e238 0x33a3299 00773bec 0332ee47 00edb274 0180df00 00f8463c 0x33a3031 00773c28 0332e71b 0180df00 033a35c5 0180d870 0x332ee47 00773cc4 033a3031 00000001 00000000 0180d2b4 0x332e71b 00773d60 0332ee47 00edb274 0180cf7c 00f8463c 0x33a3031 00773d9c 0332e71b 0180cf7c 033a35c5 0180c8ec 0x332ee47 00773e38 033a3031 00000001 00000000 0180c330 0x332e71b 00773ed4 0332ee47 00edb274 0180bff8 00f8463c 0x33a3031 00773f10 0332e71b 0180bff8 033a35c5 0180b968 0x332ee47 00773fac 033a3031 00000001 00000000 0180b3ac 0x332e71b 00774048 0332ee47 00edb274 0180b074 00f8463c 0x33a3031 00774084 0332e71b 0180b074 033a35c5 0180a9e4 0x332ee47
Note If you are running on a multi-processor system (or a hyper-threaded system), you will see function calls in mscorsvr instead of mscorwks.
You can see the function names in Mscorwks.dll because WinDbg downloaded symbols for mscorwks. However, following the calls into mscorwks, you see no module names and no function names. That's what managed code (code that runs under the Common Language Runtime) looks like in WinDbg by default, and it's obviously not helpful output. To see what happened in that managed code, you need to use the SOS extension for WinDbg.
The SOS extension
SOS is an extension for WinDbg that allows you to debug managed code. SOS adds many commands to WinDbg, all of which are focused on debugging managed applications and many are ASP.NET-specific. The extension is a DLL file called Sos.dll, and it's located in the Clr10 folder in the folder where the Debugging Tools for Windows are installed.To load the SOS extension, type the following command at the command prompt in WinDbg:
.load clr10\sos
Don't skip the period at the beginning of the command! If all goes
well, you'll be taken back to the command prompt. If an error occurs, make sure
that you have the latest version of the Debugging Tools for Windows installed.
We'll cover a few of the commands from SOS to debug the crash dump file, but feel free to explore the other commands in SOS on your own time. You can use the !help command to get a list of the commands and a brief description of each.
Digging into root cause
Now that SOS has been loaded, let's have a look at some of the commands that are useful in troubleshooting a crash dump file.Let's look at the managed stack for the current thread, thread 0. We'll do that using the !clrstack command. Type !clrstack at the command prompt. WinDbg starts dumping the stack. A normal stack should dump out completely in just a few seconds. In this case, the output seems to go on forever. Go ahead and press CTRL-BREAK (or click Break on the Debugmenu in WinDbg) and let's look at it. Here's a portion of the stack:
Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
Void System.Web.HttpServerUtility.Transfer(String,Boolean)
Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)
Void System.Web.UI.Page.ProcessRequestMain()
Void System.Web.UI.Page.ProcessRequest()
Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
Void System.Web.HttpServerUtility.Transfer(String,Boolean)
Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)
Void System.Web.UI.Page.ProcessRequestMain()
Void System.Web.UI.Page.ProcessRequest()
Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext
To get a better idea of what happened here, you can get more detailed output from the stack by running the !clrstack -s command. This gives you the full stack. Here is some of that output. For the full output from this command, create a dump file using the sample page, and then run the command. I have edited it here because the output is quite verbose.
0:000> !clrstack -s
Thread 0
System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
ESP/REG Object Name
00773ac4 0180e7f4 System.EventArgs
00773ac8 00f86c14 System.Web.HttpServerUtility
00773acc 0180e238 System.Web.UI.WebControls.Button
00773ad0 0180eac0 System.Object[]
00773ad4 0180ea7c System.String /crash.aspx
00773ad8 00eb593c System.String c:\inetpub\wwwroot\crash.aspx
00773ae4 00f84754 System.Web.HttpResponse
00773ae8 00f846bc System.Web.HttpRequest
00773b00 00f86bec System.String crash.aspx
00773b04 00f86c14 System.Web.HttpServerUtility
00773b2c 00eef57c System.String btnCrash
00773b34 00eef57c System.String btnCrash
00773b38 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode
00773b44 00f797d8 System.String CausesValidation
00773b4c 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode
00773b60 033a3031 [DEFAULT] [hasThis] Void System.Web.HttpServerUtility.Transfer(String,Boolean)
ESP/REG Object Name
00773b60 0180e238 System.Web.UI.WebControls.Button
00773b6c 03360398 [DEFAULT] [hasThis] Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
ESP/REG Object Name
00773b70 0180e7f4 System.EventArgs
00773b74 033a2fc5 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
ESP/REG Object Name
00773b74 0180df00 ASP.crash_aspx
00773b78 0180e238 System.Web.UI.WebControls.Button
00773b7c 00f8601c System.Web.HttpValueCollection
00773b80 0180e314 System.ComponentModel.EventHandlerList
00773b88 033a2da2 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
You can see the HttpRequest object in this stack, so let's have a look at that to see what was passed in that request. We'll do that using the dump object command, !do. This command takes one parameter: the address of the object to dump. In the preceding output, the address is the second hex number in the output.
Here is the output of the HttpRequest object. I have edited this output for the sake of brevity.
0:000> !do 00f846bc
Name: System.Web.HttpRequest
MethodTable 0x03089170
EEClass 0x02e9a1cc
Size 152(0x98) bytes
mdToken: 02000072 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 030888b0
MT Field Offset Type Attr Value Name
03089170 4000626 14 CLASS instance 00f8457c _path
03089170 4000627 8c System.Boolean instance 0 _computePathInfo
03089170 4000628 18 CLASS instance 00f8457c _filePath
03089170 400062e 2c CLASS instance 00f845a4 _pathTranslated
03089170 400062f 30 CLASS instance 00e6703c _baseVirtualDir
03089170 4000630 34 CLASS instance 00edce64 _contentType
03089170 4000631 88 System.Int32 instance 85 _contentLength
03089170 400063c 60 CLASS instance 00f8601c _form
Look back at the output from !clrstack. If you look carefully, you'll see that the Click event for the button is firing each time the page loads. This tells you that you are in a PostBack state each time the page loads. You can also tell from the stack that Server.Transfer is called in that Click event. Therefore, to stop ASP.NET from crashing when this page is loaded, you will need to stop this from being a PostBack each time the page is loaded due to the Server.Transfer.
Note If you do a Server.Transfer in the ASP.NET Framework 1.1 with SP1 installed and you retain form data, it will always cause this problem. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
839521
(http://support.microsoft.com/kb/839521/
)
FIX: The Server.Transfer method causes a stack overflow and causes the ASP.NET worker process to stop responding
The Server.Transfer method has two parameters; the
first is the URL to which execution should transfer, and the second is a
Boolean that specifies whether or not the form data should be retained. What
happens if we retain the form data? For one thing, ASP.NET will consider that a
PostBack. You can, therefore, stop this crash by setting the second parameter
to the Server.Transfer call to false. Wrapping it up
This article presented a brief walkthrough of how you can use WinDbg and SOS to debug your ASP.NET applications. The purpose of this article has been to introduce you to some of the concepts and terminology involved in user-mode debugging. It is a different skill set than source code debugging, and getting good at it requires a significant time investment. It is a skill that is learned by experience.For more experience, I recommend that you load up your ASP.NET application in a development or staging environment, and capture some hang dump files at different times. You can do that by running Adplus in hang mode by using the following command:
cscript adplus.vbs -hang -pn aspnet_wp.exe -o c:\aspnet_dump
This command will dump the aspnet_wp.exe process immediately and create a dump file in the C:\Aspnet_dump folder. A hang dump file such as this can be run from a remote connection.
Microsoft's Patterns and Practices group (PAG) has also written some excellent and in-depth articles on debugging .NET applications using WinDbg. Visit the following Microsoft Web site to view these articles:
http://msdn2.microsoft.com/en-us/library/ms954594.aspx
(http://msdn2.microsoft.com/en-us/library/ms954594.aspx)
Indeed, one of my goals in writing this article was to provide you
with the foundation necessary to take advantage of the excellent articles that
PAG has provided. I strongly encourage you to take advantage of
them.I’ll see you back here next month when I’ll teach you how to
use WinDbg and the SOS extension to analyze crash dump files. Until next time,
Jim Cheshire
Support Engineer
Microsoft Developer Support
As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base using the Ask For It (http://support.microsoft.com/common/survey.aspx?scid=sw;en;1176&p0=&p1=&p2=&p3=&p4=) form.
Jim Cheshire
Support Engineer
Microsoft Developer Support
As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base using the Ask For It (http://support.microsoft.com/common/survey.aspx?scid=sw;en;1176&p0=&p1=&p2=&p3=&p4=) form.