Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications

When you make calls to XML Web services from an ASP.NET application, you may experience contention, poor performance, and deadlocks. Clients may report that requests stop responding (or "hang") or take a very long time to execute. If a deadlock is suspected, the worker process may be recycled. You may receive the following messages in the application event log.

• If you are using Microsoft Internet Information Services (IIS) 5.0, you receive the following messages in the application event log:

     Event Type:     Error
     Event Source:   ASP.NET 1.0.3705.0
     Event Category: None
     Event ID:       1003
     Date:           5/4/2003
     Time:           6:18:23 PM
     User:           N/A
     Computer:       <ComputerName>
     Description:
        aspnet_wp.exe  (PID: <xxx>) was recycled because it was suspected to be in a deadlocked state.
        It did not send any responses for pending requests in the last 180 seconds.

• If you are using IIS 6.0, you receive the following messages in the application event log:

     Event Type:     Warning
     Event Source:   W3SVC-WP
     Event Category: None
     Event ID:       2262
     Date:           5/4/2003
     Time:           1:02:33 PM
     User:           N/A
     Computer:       <ComputerName>
     Description:
        ISAPI 'C:\Windows\Microsoft.net\Framework\v.1.1.4322\aspnet_isapi.dll' reported itself as
        unhealthy for the following reason: 'Deadlock detected'.

• If you are using IIS 6.0, you receive the following messages in the System event log:

     Event Type:     Warning
     Event Source:   W3SVC
     Event Category: None
     Event ID:       1013
     Date:           5/4/2003
     Time:           1:03:47 PM
     User:           N/A
     Computer:       <ComputerName>
     Description:
        A process serving application pool 'DefaultAppPool' exceeded time limits during shut down.
        The process id was '<xxxx>'.

You may also receive the following exception error message when you make a call to the HttpWebRequest.GetResponse method:You may also receive the following exception error message in the browser:Note This article also applies to applications that make HttpWebRequest requests directly.

This problem might occur because ASP.NET limits the number of worker threads and completion port threads that a call can use to execute requests.

Typically, a call to a Web service uses one worker thread to execute the code that sends the request and one completion port thread to receive the callback from the Web service. However, if the request is redirected or requires authentication, the call may use as many as two worker and two completion port threads. Therefore, you can exhaust the managed ThreadPool when multiple Web service calls occur at the same time.

For example, suppose that the ThreadPool is limited to 10 worker threads, and all 10 worker threads are currently executing code that is waiting for a callback to execute. The callback can never execute because any work items that are queued to the ThreadPool are blocked until a thread becomes available.

Another potential source of contention is the maxconnection parameter that the System.Net namespace uses to limit the number of connections. Generally, this limit works as expected. However, if many applications try to make many requests to a single IP address at the same time, threads may have to wait for an available connection.

To resolve these problems, you can tune the following parameters in your Machine.config file to best fit your situation:

• maxWorkerThreads
• minWorkerThreads
• maxIoThreads
• minFreeThreads
• minLocalRequestFreeThreads
• maxconnection
• executionTimeout

To successfully resolve these problems, do the following:

• Limit the number of ASP.NET requests that can execute at the same time to approximately 12 per CPU.
• Permit Web service callbacks to freely use threads in the ThreadPool.
• Select an appropriate value for the maxconnections parameter. Base your selection on the number of IP addresses and AppDomains that are used.

Note The recommendation to limit the number of ASP.NET requests to 12 per CPU is a little arbitrary. However, this limit has proved to work well for most applications.

maxWorkerThreads and maxIoThreads

ASP.NET uses the following two configuration settings to limit the maximum number of worker threads and completion threads that are used: The maxWorkerThreads parameter and the maxIoThreads parameter are implicitly multiplied by the number of CPUs. For example, if you have two processors, the maximum number of worker threads is the following:

minFreeThreads and minLocalRequestFreeThreads

ASP.NET also contains the following configuration settings that determine how many worker threads and completion port threads must be available to start a remote request or a local request:If there are not sufficient threads available, the request is queued until sufficient threads are free to make the request. Therefore, ASP.NET will not execute more than the following number of requests at the same time:Note The minFreeThreads parameter and the minLocalRequestFreeThreads parameter are not implicitly multiplied by the number of CPUs.

minWorkerThreads

As of ASP.NET 1.0 Service Pack 3 and ASP.NET 1.1, ASP.NET also contains the following configuration setting that determines how many worker threads may be made available immediately to service a remote request. Threads that are controlled by this setting can be created at a much faster rate than worker threads that are created from the CLR's default "thread-tuning" capabilities. This setting enables ASP.NET to service requests that may be suddenly filling the ASP.NET request queue due to a slow-down on a back end server, a sudden burst of requests from the client end, or something similar that would cause a sudden rise in the number of requests in the queue. The default value for the minWorkerThreads parameter is 1. We recommend that you set the value for the minWorkerThreads parameter to the following value. By default, the minWorkerThreads parameter is not present in either the Web.config file or the Machine.config file. This setting is implicitly multiplied by the number of CPUs.

maxconnection

The maxconnection parameter determines how many connections can be made to a specific IP address. The parameter appears as follows:The settings for the parameters that are discussed earlier in this article are all at the process level. However, the maxconnection parameter setting applies to the AppDomain level. By default, because this setting applies to the AppDomain level, you can create a maximum of two connections to a specific IP address from each AppDomain in your process.

executionTimeout

ASP.NET uses the following configuration setting to limit the request execution time: You can also set this limit by using the Server.ScriptTimeout property.

Note If you increase the value of the executionTimeout parameter, you may also have to modify the processModel responseDeadlockInterval parameter setting.

Recommendations

The settings that are recommended in this section may not work for all applications. However, the following additional information may help you to make the appropriate adjustments.

If you are making one Web service call to a single IP address from each ASPX page, Microsoft recommends that you use the following configuration settings:

• Set the values of the maxWorkerThreads parameter and the maxIoThreads parameter to 100.
• Set the value of the maxconnection parameter to 12*N (where N is the number of CPUs that you have).
• Set the values of the minFreeThreads parameter to 88*N and the minLocalRequestFreeThreads parameter to76*N.
• Set the value of minWorkerThreads to 50. Remember, minWorkerThreads is not in the configuration file by default. You must add it.

Some of these recommendations involve a simple formula that involves the number of CPUs on a server. The variable that represents the number of CPUs in the formulas is N. For these settings, if you have hyperthreading enabled, you must use the number of logical CPUs instead of the number of physical CPUs. For example, if you have a four-processor server with hyperthreading enabled, then the value of N in the formulas will be 8 instead of 4.

Note When you use this configuration, you can execute a maximum of 12 ASP.NET requests per CPU at the same time because 100-88=12. Therefore, at least 88*N worker threads and 88*N completion port threads are available for other uses (such as for the Web service callbacks).

For example, you have a server with four processors and hyperthreading enabled. Based on these formulas, you would use the following values for the configuration settings that are mentioned in this article.
Also, when you use this configuration, 12 connections are available per CPU per IP address for each AppDomain. Therefore, in the following scenario, very little contention occurs when requests are waiting for connections, and the ThreadPool is not exhausted:

• The Web hosts only one application (AppDomain).
• Each request for an ASPX page makes one Web service request.
• All requests are to the same IP address.

However, when you use this configuration, scenarios that involve one of the following will probably use too many connections:

• Requests are to multiple IP addresses.
• Requests are redirected (302 status code).
• Requests require authentication.
• Requests are made from multiple AppDomains.

In these scenarios, it is a good idea to use a lower value for the maxconnection parameter and higher values for the minFreeThreads parameter and the minLocalRequestFreeThreads parameter.

This behavior is by design.

If you are experiencing poor performance and contention on IIS 7.0 with ASP.NET, see the following Microsoft blogs:

ASP.NET Thread Usage on IIS 7.0 and 6.0
ASP.net Hang in IIS 7.0

For more information, visit the following Microsoft Developer Network (MSDN) Web site: