WCF service may scale up slowly under load

Article ID: 2538826
Expand all | Collapse all

SYMPTOMS

When your WCF service receives a burst of requests, the default .Net I/O Completion Port (IOCP) thread pool may not scale up as quickly as desired and your WCF response time will increase as a result. Depending on the execution time and number of requests received you may notice the WCF execution time increase linearly by approximately 500ms for each request received until the process has created sufficient IOCP threads to service the requests or sustain the incoming load. The problem is more evident in services with longer execution times. The IOCP thread pool scalability problem is not typically observed upon the initial loading of the process.

CAUSE

Three variables that have an impact your WCF service ability to scale up at nearly the same rate as incoming requests.

1.       WCF Throttling

2.       .Net CLR Threadpool.GetMinThreads value

3.       .Net CLR IO Completion Port thread pool bug where IOCP threads are no longer created in a pattern corresponding to the incoming request volume prior to the Threadpool.GetMinThreads throttling value.

This article describes how to resolve the problem with the .Net IOCP Threadpool, #3. If you have throttling issues due to WCF throttling or the GetMinThreads value, this solution will not avoid those throttles. See the More Information section below for guidance in identifying your scenario. The IOCP thread creation bug should be addressed in the next post 4.0 release of the .Net Framework. This scalability problem does not exist in the .Net CLR Worker thread pool.

RESOLUTION

By moving the WCF service execution to another thread pool, you may incur a small amount of overhead implementing this solution. Performance results will vary per WCF Service. Test each WCF Service for individual results.

Note: Apply this solutionwhen using a WCF Listener which does not block the incoming thread while waiting on the WCF service code to complete.
Collapse this tableExpand this table
WCF ListenerRecommended solution
HTTP Sync Module (Default in 3.x) - used in Integrated Application PoolSwitch to the Async handler and then then apply the solution in this article or alternatively use a Private Threadpool (see links following this table)
HTTP Aync Module (Default in 4.x) - used in Integrated Application PoolApply the code solution in this article
ISAPI - used in Classic Mode Application PoolApply Private Threadpool (see links following this table)
tcp.NetApply the code solution in this article
If you are unable to apply the solution in this article following the above table, an example using a private threadpool can be found in an MSDN article:
Synchronization Contexts in WCF by Juval Lowy
http://msdn.microsoft.com/en-us/magazine/cc163321.aspx

Steps to switch from the Synchronous HTTP handler to use the Async HTTP handler:
http://blogs.msdn.com/b/wenlong/archive/2008/08/13/orcas-sp1-improvement-asynchronous-wcf-http-module-handler-for-iis7-for-better-server-scalability.aspx

Steps to implement this solution which will execute the WCF service on the .Net CLR Worker thread pool
1.       WCF throttling thresholds should high enough to handle anticipated burst volume within acceptable response times.

2.       If you use one of the .Net CLR default thread pools, Worker or IOCP for your WCF service, you must ensure the minimum thread count (value where thread creation throttling begins) to a number you anticipate to execute concurrently.

3.       Implement the following code in your service which will then execute your WCF service on the .Net CLR Worker thread pool.

This class is used to move the execution to the .Net CLR Worker thread pool.

   public class WorkerThreadPoolSynchronizer : SynchronizationContext
    {
        public override void Post(SendOrPostCallback d, object state)
        {
            // WCF almost always uses Post
            ThreadPool.QueueUserWorkItem(new WaitCallback(d), state);
        }
 
        public override void Send(SendOrPostCallback d, object state)
        {
            // Only the peer channel in WCF uses Send
            d(state);
        }
    }
Next we need to create a custom attribute class.

    [AttributeUsage(AttributeTargets.Class)]
    public class WorkerThreadPoolBehaviorAttribute : Attribute, IContractBehavior
    {
        private static WorkerThreadPoolSynchronizer synchronizer = new WorkerThreadPoolSynchronizer();
 
        void IContractBehavior.AddBindingParameters(
            ContractDescription contractDescription, 
            ServiceEndpoint endpoint, 
            BindingParameterCollection bindingParameters)
        {
        }
 
        void IContractBehavior.ApplyClientBehavior(
            ContractDescription contractDescription, 
            ServiceEndpoint endpoint, 
            ClientRuntime clientRuntime)
        {
        }
 
        void IContractBehavior.ApplyDispatchBehavior(
            ContractDescription contractDescription, 
            ServiceEndpoint endpoint, 
            DispatchRuntime dispatchRuntime)
        {
            dispatchRuntime.SynchronizationContext = synchronizer;
        }
 
        void IContractBehavior.Validate(
            ContractDescription contractDescription, 
            ServiceEndpoint endpoint)
        {
        }
 
    }
Now to apply the custom attribute to your WCF service. Example:

   [WorkerThreadPoolBehavior]
    public class Service1 : IService1
    {
        public string GetData(int value)
        {
            int iSleepSec = (value * 1000);
            System.Threading.Thread.Sleep(iSleepSec);
            return string.Format("You slept for: {0} seconds", value);
        }
    }

MORE INFORMATION

WCF uses the .Net CLR I/O Completion Port thread pool for executing your WCF service code. The problem is encountered when the .Net CLR IO Completion Port thread pool enters a state where it cannot create threads quickly enough to immediately handle a burst of requests. The response time increases unexpectedly as new threads are created at a rate of 1 per 500ms.

The problem may become more evident if your WCF service uses a technology which also utilizes the .Net CLR IOCP thread pool. For example, the Windows Server AppFabric Cache Client leverages this thread pool to a small extent.

If you are not hitting the WCF throttling limits described earlier, the following information will help you determine if you are experiencing the problem with the .Net CLR IOCP thread pool.

The .NET CLR thread pools use a value to determine when to begin throttling the creation of threads. This setting can be determined by calling the ThreadPool.GetMinThreads or when analyzing a process dump using the !SOS debugger extension !Threadpool.

0:000> !C:\windows\Microsoft.NET\Framework64\v4.0.30319\sos.threadpool
CPU utilization: 0%
Worker Thread: Total: 16 Running: 0 Idle: 16 MaxLimit: 250 MinLimit: 125
Work Request in Queue: 0
--------------------------------------
Number of Timers: 35
--------------------------------------
Completion Port Thread:Total: 26 Free: 0 MaxFree: 16 CurrentLimit: 28 MaxLimit: 1000 MinLimit: 125

The observed problem is when the .NET CLR IOCP thread pool enters a condition where a new thread is only created every 500ms (two per second) prior to the thread pool MinLimit value for the thread pool. Other expected factors which can also contribute to a thread creation delay would be memory pressure or high CPU.

Monitor the process hosting your WCF service. If you notice a problem scaling up threads prior to the minimum thresholds you have set, you may be encountering the problem with the .Net CLR IOCP thread pool. To determine if this is the case, Perfmon should be used to monitor the process thread creation rate in comparison to the incoming request rate. To accomplish this, log or view the following perfmon counters (below is an example for an IIS (WAS) hosted WCF 4.0 service using a HTTP binding):
Collapse this tableExpand this table
CounterInstance(s)
Process / Thread CountAll W3WP(x) instances
HTTP Service Request Queues / Arrival Rate<ApplicationPool Hosting the WCF Service(s)>
ASP.Net Apps v(4 or 2) / Requests Executing<WCF Application Instance(s)>
ASP.Net Apps v(4 or 2) / Request Execution Time<WCF Application Instance(s)>
You can use the WCF perfmon counters if you have them enabled as well:
http://msdn.microsoft.com/en-us/library/ms735098.aspx

It is normal to see a slowly increasing thread count when the arrival rate (client requests pattern) is following the same pattern. It is only when there is an immediate spike of incoming requests and the thread count slowly increases at a rate of 2 threads per second while the WCF response time increases that a problem exists.

This screenshot shows a worker process that after some time has encountered the .Net IOCP thread pool scalability issue. When the process first started, the IOCP threads are normally created in parallel to the incoming request load. In this AppPool (W3WP.EXE), there were two WCF services running. One service was using the default .Net IOCP thread pool which received a burst of 100 requests at 10:22:14 and again at 10:23:34. The second WCF service was using the above workaround to execute on the .Net Worker thread pool and received a burst of 100 requests at 10:22:54. After entering this state, a process recycle is required to restore the IOCP thread pool to a working, scalable state.


Collapse this imageExpand this image
2540178


Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.

Properties

Article ID: 2538826 - Last Review: April 26, 2011 - Revision: 1.1
Keywords: 
KB2538826

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com