IIS runs out of work items and causes RPC failures when connecting to a remote UNC path

In a Web farm environment, where one or more IIS Web servers are using a remote Windows NT or Windows 2000 file server to hold the content for the Web site (for example, the home directory or the virtual directories for the Web site are mapped to a UNC path), the IIS Web servers may run out of available work contexts at the redirector level. When this happens, the following symptoms can occur on your IIS Web servers:

• IIS appears to stop responding. If you examine a process dump of Inetinfo.exe, you will see that a large number of worker threads are waiting to retrieve content from the remote file share. In a Performance Monitor log, IIS appears largely idle during this time, as the majority of the threads are in a wait state, waiting to access the remote server.
• On the local desktop of one of the IIS Web servers, if you attempt to use the client redirector to access the remote file share through Network Neighborhood or My Network Places, or through Windows Explorer, it appears to stop responding indefinitely. If a UNC path is mapped to a local drive letter in Windows Explorer, and Windows Explorer is open, it appears to stop responding as well. This can give the appearance that the entire operating system has stopped responding, when in fact it has not.
• The following error messages may occur when the maximum number of work contexts are exhausted on a particular server:
  
  
  RPC 1792 - The remote procedure call failed and did not execute.
  Netlogon 5719 - Unable to find domain controller.
  These error messages may occur when you attempt to make any additional RPC connections to a server when the maximum number of work contexts is reached.
• The network BIOS command limit has been reached.
• The following event ID message is logged:

The server is running out of work contexts for a particular client. In Windows 2000, there is a hard-coded upper limit of 125 work contexts for all types of clients (Windows NT, Windows 95, and Windows 98).

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

1. If you are running Windows 2000, install the post-SP1 hotfix described in the following Microsoft Knowledge Base article to both the IIS servers and the file server:
   271148  (http://support.microsoft.com/kb/271148/ ) MaxMpxCt and MaxCmds limits in Windows 2000
2. Increase the values of MaxCmds on the IIS Servers and MaxMpxCT on the File Server by using the following registry scripts:
   
   
   • Save the following registry script as Client.reg, and then run it on the IIS servers:
     Windows Registry Editor Version 5.00
     
     [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters] "MaxCmds"=dword:00000800
   • Save the following registry script as Server.reg, and then run it on the file server:
     Windows Registry Editor Version 5.00
     
     [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters]
     "MaxMpxCT"=dword:00000800
     "MaxWorkItems"=dword:00002000
   By default, these scripts specify a MaxCmds and MaxMpxCT of 2,048, which should be sufficient for most situations. MaxWorkItems has been specified at 4x MaxMpxCt or 8,192. For more information on these settings, see the "More Information" section of this article.
3. Restart all of the IIS servers and the file server for these changes to take effect.

Note When you increase these values on the x86 platform, additional non-paged pool memory is consumed on the file server and the IIS clients. Non-paged pool memory has an upper limit of 256 megabytes (MB). Lots of clients that have lots of connections can consume all of the non-paged pool memory on the file server. Use Performance Monitor to watch this counter and to make sure that it is not approaching the limit. On the x64 platform, non-paged pool memory has an upper limit of 128 gigabytes (GB). Therefore, you can increase these values to the maximum value. For more information about the differences between non-paged pool memory on the x86 platform and on the x64 platform, click the following article number to view the article in the Microsoft Knowledge Base:

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

After you apply the hotfix described in Q271148, the upper limit for MaxMpxCt (the server setting) changes from 125 to 65,535, and on the client, the upper limit for MaxCmds (the client setting) changes from 255 to 65,535.

Additionally, the maximum number of concurrent SMB sessions that can be opened between the client and the server is the lower of MaxCmds and MaxMpxCT . However, if the connecting client is a Windows 95 or Windows 98 client, then the effective value of MaxMpxCt for that client is limited to 125.

These limits become important when you use IIS in a Web-farm scenario where the content for the Web sites is stored on a remote UNC share, because IIS uses the ReadDirectoryChangesW API to receive file change notifications. This is done so that if files change, IIS can un-cache the old files, and then re-read the new files from the disk or share. When you use a UNC path as the home directory, a persistent SMB connection remains open between the IIS server and the file server, which consumes a work context. If the directory structure is large enough, it is possible to run out of work contexts and encounter the symptoms listed previously.

A computer running IIS can have multiple virtual directories or Web sites pointing to shares on another Windows NT Server computer. The ASP Directory Monitor uses the ReadDirectoryChangesW API to monitor for any changes to those directories on the other server. Each pending ReadDirectoryChangesW requires a work context on the server, and there is only a limited number of work contexts available.

The number of work contexts is passed from the server to the client when the SMB level is negotiated. The redirector on the client keeps an internal count of the number of work contexts that it is using on the server. The default number of work contexts is 50.

The number of work contexts is limited to keep the server process from consuming all non-paged pool memory. This can be raised, but then there is a limit to how many work contexts a particular client can consume.

This problem is not limited to IIS. Windows NT Explorer uses the same mechanism to monitor for directory changes.

Note This issue does not occur on computers that are running an x64 version of Windows Server 2003. For more information about the MaxWorkItems and MaxMpxCT settings, click the following article numbers to view the articles in the Microsoft Knowledge Base: