Article ID: 2538826
When your WCF service receives a burst of requests, the default .Net I/O Completion Port (IOCP) thread pool may not scale up as quickly as desired and your WCF response time will increase as a result. Depending on the execution time and number of requests received you may notice the WCF execution time increase linearly by approximately 500ms for each request received until the process has created sufficient IOCP threads to service the requests or sustain the incoming load. The problem is more evident in services with longer execution times. The IOCP thread pool scalability problem is not typically observed upon the initial loading of the process.
Three variables that have an impact your WCF service ability to scale up at nearly the same rate as incoming requests.
1. WCF Throttling
2. .Net CLR Threadpool.GetMinThreads value
3. .Net CLR IO Completion Port thread pool bug where IOCP threads are no longer created in a pattern corresponding to the incoming request volume prior to the Threadpool.GetMinThreads throttling value.
This article describes how to resolve the problem with the .Net IOCP Threadpool, #3. If you have throttling issues due to WCF throttling or the GetMinThreads value, this solution will not avoid those throttles. See the More Information section below for guidance in identifying your scenario. The IOCP thread creation bug should be addressed in the next post 4.0 release of the .Net Framework. This scalability problem does not exist in the .Net CLR Worker thread pool.
By moving the WCF service execution to another thread pool, you may incur a small amount of overhead implementing this solution. Performance results will vary per WCF Service. Test each WCF Service for individual results.
Note: Apply this solutionwhen using a WCF Listener which does not block the incoming thread while waiting on the WCF service code to complete.
If you are unable to apply the solution in this article following the above table, an example using a private threadpool can be found in an MSDN article:
Collapse this tableExpand this table
Synchronization Contexts in WCF by Juval Lowy
Steps to switch from the Synchronous HTTP handler to use the Async HTTP handler:
Steps to implement this solution which will execute the WCF service on the .Net CLR Worker thread pool
1. WCF throttling thresholds should high enough to handle anticipated burst volume within acceptable response times.
2. If you use one of the .Net CLR default thread pools, Worker or IOCP for your WCF service, you must ensure the minimum thread count (value where thread creation throttling begins) to a number you anticipate to execute concurrently.
3. Implement the following code in your service which will then execute your WCF service on the .Net CLR Worker thread pool.
This class is used to move the execution to the .Net CLR Worker thread pool.
Next we need to create a custom attribute class.
Now to apply the custom attribute to your WCF service. Example:
WCF uses the .Net CLR I/O Completion Port thread pool for executing your WCF service code. The problem is encountered when the .Net CLR IO Completion Port thread pool enters a state where it cannot create threads quickly enough to immediately handle a burst of requests. The response time increases unexpectedly as new threads are created at a rate of 1 per 500ms.
The problem may become more evident if your WCF service uses a technology which also utilizes the .Net CLR IOCP thread pool. For example, the Windows Server AppFabric Cache Client leverages this thread pool to a small extent.
If you are not hitting the WCF throttling limits described earlier, the following information will help you determine if you are experiencing the problem with the .Net CLR IOCP thread pool.
The .NET CLR thread pools use a value to determine when to begin throttling the creation of threads. This setting can be determined by calling the ThreadPool.GetMinThreads
(http://msdn.microsoft.com/en-us/library/system.threading.threadpool.getminthreads.aspx)or when analyzing a process dump using the !SOS debugger extension !Threadpool
CPU utilization: 0%
Worker Thread: Total: 16 Running: 0 Idle: 16 MaxLimit: 250 MinLimit: 125
Work Request in Queue: 0
Number of Timers: 35
Completion Port Thread:Total: 26 Free: 0 MaxFree: 16 CurrentLimit: 28 MaxLimit: 1000 MinLimit: 125
The observed problem is when the .NET CLR IOCP thread pool enters a condition where a new thread is only created every 500ms (two per second) prior to the thread pool MinLimit value for the thread pool. Other expected factors which can also contribute to a thread creation delay would be memory pressure or high CPU.
Monitor the process hosting your WCF service. If you notice a problem scaling up threads prior to the minimum thresholds you have set, you may be encountering the problem with the .Net CLR IOCP thread pool. To determine if this is the case, Perfmon should be used to monitor the process thread creation rate in comparison to the incoming request rate. To accomplish this, log or view the following perfmon counters (below is an example for an IIS (WAS) hosted WCF 4.0 service using a HTTP binding):
You can use the WCF perfmon counters if you have them enabled as well:
Collapse this tableExpand this table
It is normal to see a slowly increasing thread count when the arrival rate (client requests pattern) is following the same pattern. It is only when there is an immediate spike of incoming requests and the thread count slowly increases at a rate of 2 threads per second while the WCF response time increases that a problem exists.
This screenshot shows a worker process that after some time has encountered the .Net IOCP thread pool scalability issue. When the process first started, the IOCP threads are normally created in parallel to the incoming request load. In this AppPool (W3WP.EXE), there were two WCF services running. One service was using the default .Net IOCP thread pool which received a burst of 100 requests at 10:22:14 and again at 10:23:34. The second WCF service was using the above workaround to execute on the .Net Worker thread pool and received a burst of 100 requests at 10:22:54. After entering this state, a process recycle is required to restore the IOCP thread pool to a working, scalable state.
Collapse this imageExpand this image
(http://go.microsoft.com/fwlink/?LinkId=151500)for other considerations.
Article ID: 2538826 - Last Review: April 26, 2011 - Revision: 1.1