Article ID: 317249 - Last Review: December 3, 2007 - Revision: 9.3 How to troubleshoot Event ID 2021 and Event ID 2022This article was previously published under Q317249 On This PageSYMPTOMS Any one of the operating systems that are included in the Applies to section, may stop responding for a short time and multiple events that are similar to the following may be logged to the System log: Event ID: 2022 Source: Srv Description: Server was unable to find a free connection n times in the last s seconds. Event ID: 2021 Source: Srv Description: Server was unable to create a work item %2 times in the last %3 seconds. Event ID: 3013 Source: Rdr Description: The redirector has timed out to Computer_Name. Note Many of the troubleshooting steps that are discussed in this article can also be used to resolve Event ID 3013 errors. Other components of the operating system may not work and may generate error messages that report a status code of 1450 in the data section of their event log message. That is, "Insufficient System Resources." These events may be found in the System event log or in the Application event log. These messages may apply to the issue that is described in this article only if the underlying event was a connection to the server service. However, this fact is not easily determined. For example, there is Event ID 1055 that is generated by CLUSSVC. This event is from the cluster service that usually reports a failed connection to the server service. CAUSE This problem occurs because the Server service cannot keep up with the demand for network work items that are queued by the network layer of the I/O stream. The Server service cannot process the requested network I/O items quickly enough to the hard disk and exhausts available resources. There can be many root causes for the Server service exhausting available resources. For example, any issue in the I/O path between the network adapter and the hard disk drive can cause the symptoms that are described in this article. This problem may also occur if an incorrect network adapter driver is installed. RESOLUTIONTo resolve this problem, first try to maximize the number of resources that are available to the Server service. For more information about how to do this, see the "Level 1. Maximize server resources" section. Important Maximizing the system resources for the Server service at this point does not make it more difficult to troubleshoot the root cause of this issue. Then, try to determine under what circumstances event id 2022 and 2021 errors appear. Many different issues can cause this problem. When you troubleshoot this issue, investigate all the components in the I/O path, from the network adapter to the hard disk drive array. For example, perhaps these errors occur only when the backup program is running. Perhaps they occur only early in the morning when users log on. If the errors occur only briefly, you may address this problem by tuning the server service together with applying any hotfix updates for the server service. General informationEvent 2021 is logged when there is accumulation of work items in the server service. But you must understand that the most common cause of the accumulation of work items in the server service is because the disk subsystem does not keep up with the number of requests. The server service will allow for a some tuning but this will not fully resolve the issue if the problem is caused by disk throughput. Therefore, the most important step is to determine whether the disk throughput has changed from the ordinary baseline or is at least within reasonable limits.Event 2022 is logged when there are too few connections available for users. The most common cause of insufficient free network connections is a wide swing in network load represented by many clients trying connections at the same time or by some clients trying many connections each. This may be because of a program on the client because users cannot make connections quickly. This will cause too few connections to be available for new users. Therefore, one of the most important steps is to determine whether the changes in network load are within reasonable limits. If there are minor changes in load that cause these messages to be logged in the event logs on an infrequent basis, you can modify the operation of the server service. If the disk subsystem is too slow or the if network loads are too high or too variable, modifying the server service will not resolve the issue. That is why it is you must use tools such as Performance Monitor and a network packet capture program, either from Microsoft or from a third-party vendor, in to determine the true root cause of these events. Specific troubleshooting steps in are described this section under the sub-section “Level 2. Identify the root cause.” Level 1. Maximize server resourcesTo eliminate the error messages, or to reduce the frequency of the error messages, you can try to maximize the number of resources that are available to the server service. For more information about how to increase server resources, click the following article number to view the article in the Microsoft Knowledge Base:228766
(http://support.microsoft.com/kb/228766/
)
How to change the Server service properties
Important
For more information about how to set up the server to create a dump file, click the following article numbers to view the articles in the Microsoft Knowledge Base: 254649
(http://support.microsoft.com/kb/254649/
)
Overview of memory dump file options for Windows 2000, for Windows XP, and for Windows Server 2003
244139
(http://support.microsoft.com/kb/244139/
)
Windows feature allows a Memory.dmp file to be generated with the keyboard
The server service is tuned automatically when the server starts. However, many of the default settings have not been upgraded as new releases of Microsoft Windows 2000 and Windows Server 2003 service packs have been released. Values specified in this section represent the best practices of Microsoft Product Support Services by using the currently available service packs. Besides applying to the current service pack, the values will all apply to older systems as long as the current hotfixes for the kernel, redirector and server services are installed. Before you install an upgraded server service we recommend that you install the current hotfixes for both the redirector and the kernel at the same time. If you cannot install the current hotfix for the kernel immediately, install the current hotfixes for the redirector when the server service hotfixes are installed. A. Change Registry settings to Maximize Server Service PerformanceImportant This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:322756
(http://support.microsoft.com/kb/322756/
)
How to back up and restore the registry in Windows Note When you add the registry values that are described in this section, make sure that you enter them in decimal format. 1. Reduce Event ID 2021 errorsTo specifically address Event ID 2021, start Registry Editor and locate the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters Then, add or modify the following DWORD registry values:Description: Maximum Work Items Value Name: MaxWorkItems Data Type: REG_DWORD Value data: 0x2000 or 8192 (decimal) Description: Size of a Requested Buffer Value Name: SizReqBuf Data Type: REG_DWORD Value data: 0x4104 or 16644 (decimal) Scope of Values A brief discussion is given here to clarify the scope of the allowed values. Although, other documentation exists that show large maximum defined values, these maximum values cannot be used. For the MaxWorkItems registry entry, the value of 8196 can only be used on servers that have more than 1.5 gigabytes (GB) of memory and without the /3gb switch present in the Boot.ini file. The maximum value is '65535' (decimal). But this high setting will cause the server to stop responding, if enough work items are consumed. Therefore, a greater value usually cannot be used without modification of the server service work item buffer size. A high setting of the MaxWorkItems registry entry may even cause the server to stop responding on startup and the cause of this will be exhaustion of non-paged memory. The tags observed will be LSwn, or, rarely, some other closely related tag. By default, the value for SizReqBuf is '16644', but the non-paged pool allocation will be '20480' per work item. The additional allocation is because of tracking overhead. Therefore, a value of 8196 for MaxWorkItems will enable the server service to use up to a maximum of 160 megabytes (MB) of the 256 MB non-paged pool limit in all versions of Microsoft x86 32-bit operating systems. But, even then the maximum limit will be reached only under very heavy load conditions. Microsoft Support Services do not recommend the use of more than 160 MB for the server service out of a possible maximum of the whole non-paged pool of 256 MB. (Divide by the value of 1024*1024, if you want to convert megabytes to bytes.) Microsoft Product Support Services have seen multiple issues where high performance file servers stopped responding when values greater than 8196 were used. These servers had other components using lots of non-paged pool, WITHOUT any pool leaks. Therefore, do not use values greater than 8196. If you want to use greater values for MaxWorkItems, reduce the SizReqBuf value to limit the maximum pool used by the server service. In some applications if we need more work items, such as on an IIS back-end server, Domain Controller, or file server where redirected user profiles are stored, we will have to reduce the maximum SMB buffer size with the following setting. Value Name: SizReqBuf Data Type: REG_DWORD Value data: 8452 (decimal) This causes a non-paged pool allocation of 12284 bytes. This will enable a maximum of 13824 work items that can be used on servers without the /3gb switch present. On servers where the /3gb switch is used we prefer to set the SizReqBuf to 8452 and MaxWorkItems to 6400 in order to limit the use of non-paged pool to 80 MB as a standard practice for highly loaded servers. Remember modification of settings is not required on a /3gb file server unless it is heavily loaded with SMB/CIFS (Common Internet File Services) traffic. Therefore, no changes would be made to an Exchange, SQL or IIS front-end server or on back-end servers that use WINSOCK. This would apply to back-end servers having RPC traffic on the standard SMB/CIFS hosted transport. On NON /3gb servers that experience work item shortages after MaxWorkItems is set to 8196, you can also set SizReqBuf to 8452 and MaxWorkItems to 13824 for short term relief while you try to identify the root cause. 2. Reduce Event ID 2022 errorsTo specifically address Event ID 2022, start Registry Editor, and move to the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters Then add or modify the following DWORD registry values: Description: Maximum Free Connections Value Name: MaxFreeConnections Data Type: REG_DWORD Value data: 0x1000 or 4096 (decimal) Description: Minimum Free Connections Value Name: MinFreeConnections Data Type: REG_DWORD Value data: 0x100 or 256 (decimal) Important These settings on Windows 2000 require the installation of current Service Pack 4 or later hotfixes. On Windows Server 2003, it requires Service Pack 1 or later hot fixes. Without the hotfixes or service pack the setting ranges are too low to be useful (100 and 32 respectively). Currently the maximum we recommend are, 4096 and 256 respectively. 3. Implement the changesTo implement these changes, exit Registry Editor, and restart the computer, or stop and then restart the Server service. To restart the Server service, follow these steps:
B. More informationThe MaxFreeConnections setting is the most important of all the server service settings.For more information, click the following article number to view the article in the Microsoft Knowledge Base: 245080
(http://support.microsoft.com/kb/245080/
)
Receiving multiple instances of Event ID 2022
The MaxFreeConnections value is most useful in resolving an Event ID 2022 error that contains many failures to find a free connection. For example, to resolve an event that has text that is similar to the following in the logged error message: Event ID: 2022 Source: Srv Description: Server was unable to find a free connection n times in the last s seconds. Many failures to find a free connection may also indicate a "network card flooding" situation where a non-network aware program floods the server by using connection attempts. If you experience many free connection failures, view the Network Card Flooding section. Note There are usually no adverse effects by adding the previous registry values. The following registry entries and values may be present in the same subkey. These setting are not directly related to the troubleshooting performed for this issue, but are listed underneath for completeness. Description: Server Size Value Name: Size Data Type: REG_DWORD Value data: 0x3 or 3 (decimal) This should have a value of 3. Do not change this value. Description: Maximum outstanding commands. Maximum Multiplex Count Value Name: MaxMpxCt Data Type: REG_DWORD Value data: 0x1000 or 4096 (decimal) Do not modify this value unless you have a specific scenario that requires it. A through discussion of this setting is found at the end of this document. For more information, click the following article number to view the article in the Microsoft Knowledge Base: 271148
(http://support.microsoft.com/kb/271148/
)
MaxMpxCt and MaxCmds Limits in Windows 2000
Level 2. Identify root causeThe key to resolution is to determine whether the root cause of the issue is the disk subsystem or if it is network load. Because it is difficult to quantitatively establish if network load is the issue, it is best to examine the disk systems. If there are consistent short periods of errors recorded in the event log you should also use a network packet capture program to capture all network traffic during the issue and examine it, in addition to the disk subsystem. Remember, that the disk subsystem may start with third-party volume managers and then move down the whole software stack. This will include all layers of all the filter drivers that are installed, and the device drivers that drive the hardware at the end. We realize that Volume Managers cannot be removed, but many other filter drivers can be uninstalled, at least temporarily.A. Verify that Windows is updated and stable1. Upgrade Network ComponentsThe following components must be the latest available from the hotfix tree. Therefore, upgrade to the latest version of the network components:
2. Examine the Event Logs for Errors Look for all event log errors that relate either to network or storage hardware. These must be corrected before you can troubleshoot the causes in the following list. Check device drivers for return errors, and also filter drivers which connect to these drivers, for any events that generally show an interoperability issue with any low level driver. 3. Identify free space errors Note If the drives have ever run of disk space this step must be done. Examine the event logs to see whether the drives have ever been out of free space. If the drives have ever run of disk space you must run a full chkdsk routine to make sure that the file system has not been damaged. 4. Scan for file fragmentation Examine the disk storage for file fragmentation. You may not be able to defragment your disk storage but you can at least run an analyze pass on the storage system. Note that for damaged or very fragmented drives running an 'analyze pass' could take the file system offline; however this is the least intrusive method currently known to scan for file system issues. Also note that it is not just the data returned from the defragmentation analysis that is important. The time the pass took to complete is very important also. You may find that some disk subsystems take very long to complete, or do not finish the process at all. Severely fragmented file systems will create the issues described in this article. Defragment the hard disks to increase Windows Read/Write performance. Note The requirement for defragmenting the disk can exceed the I/O bandwidth of the storage subsystem requiring other methods to address this issue. This may include removing files or expanding storage. For more information, click the following article number to view the article in the Microsoft Knowledge Base: 300978
(http://support.microsoft.com/kb/300978/
)
How to analyze and defragment a disk volume in Windows 2000
5. Verify hard disk corruption Hard disk corruption may cause Input/Output (I/0) bottlenecks when the operating system is reading from, or writing to the hard disk. You can run the "chkdsk /r" command at a command prompt to examine the hard disk for errors. This is not the preferred method to use to examine the file system. However, as these issues typically occur on large file systems that cannot be taken off-line, this method can be used. First, just run chkdsk without any parameters against the file system. It will report some bit oriented file system errors that are typical in a dynamic system but the report should not have any missing or corrupted directory, indexes or files. Again massive corruption may cause chkdsk to exit, or cause the utility to progress very slowly, or the utility may indicate extensive file corruption. In an interactive run of chkdsk, we can usually stop it after several minutes if it produces severe errors, as that indicates that the volume must be rebuilt. Again, you may have to use other methods to address this situation, but only running chkdsk /r can fix issues in the file systems. To do this, follow these steps:
B. Examine possible causesTo determine the root cause of the issue we present a detailed list of possible causes. To troubleshoot this issue more, you can use the following methods in the order that they are listed. These steps are not listed in any particular order.The root cause of these events can be summarized under the following two categories:
a. Server service overloadedTo determine whether the server service is overloaded, eliminate all the other possibilities, listed underneath.
b. Network Card FloodingEnterprise-level support configuration issues are a common cause of this issue, and can be classified as follows:
At the command prompt, by using the following commands you can receive additional information pointing to both offending workstation and program: net files > netfiles.txt net session > netsession.txt The following Event ID is usually listed on the offending workstation computer, especially if it is another server computer acting as a workstation: Event ID: 3013 You may be experiencing one other networking issue. This issue is seen in the following two environments:
For more information, click the following article number to view the article in the Microsoft Knowledge Base: 221790
(http://support.microsoft.com/kb/221790/
)
IIS runs out of work items and causes RPC failures when connecting to a remote UNC path
In these cases the server is used for a large number (thousands) of IIS virtual roots or for a large number (thousands) of remote profiles that contain links pointing back to the hosting server. You may also experience other abnormal symptoms, but not necessarily any other errors. To resolve this issue, follow these steps:
Increasing these values consumes additional non-paged pool memory on the file server and the IIS client server computers. Non-paged pool memory has an upper limit of 256 megabytes (MB). Many clients that use many connections can consume all the non-paged pool memory on the file server. Use Performance Monitor to watch this counter and to make sure that it is not approaching the limit. A computer that is running IIS can have multiple virtual directories or Web sites pointing to shares on other Windows NT 4.0 Server computers. The ASP Directory Monitor uses the ReadDirectoryChangesW API to monitor for any changes to those directories on the other server. Each pending ReadDirectoryChangesW item requires a work context on the server, and there are only a limited number of work contexts available. The number of work contexts is passed from the server to the client when the SMB level is negotiated. The redirector on the client keeps an internal count of the number of work contexts that it is using on the server. The default number of work contexts is 50. The number of work contexts is limited to keep the server process from exhausting all non-paged pool memory. This can be raised, but there is a limit to how many work contexts a particular client can consume. This problem is not limited to IIS. Windows NT Explorer uses the same mechanism to monitor for directory changes. Usually you will experience the issue with Explorer in a terminal server environment. For more information, click the following article number to view the article in the Microsoft Knowledge Base: 232476
(http://support.microsoft.com/kb/232476/
)
Terminal Server client connections and logon limited by MaxWorkItem and MaxMpxCt values
271148
(http://support.microsoft.com/kb/271148/
)
MaxMpxCt and MaxCmds Limits in Windows 2000
If you experience this issue with Windows Explorer, it is resolved by using the same steps as those used with the previous IIS issue. However, this solution quickly consumes the server's supply of work items, and greatly reduces the number of desktop profiles that a server can host. Note the values in the following registry subkeys: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders Or, use the following commands, at the command prompt, and view the output files. Having more than 5 files per user indicates a problem. net files > filelist.txt net session > netsession.txt Microsoft supports the redirection of only the following folders by using System Policies:
MORE INFORMATIONBoth Event ID 2021 and Event ID 2022 are caused by one of two situations.
In the first situation, pool memory cannot be allocated. Work items (2021) require non-paged pool memory. Connections (2022) require both non-paged and paged pool memory. The server can limit its usage of these pools, and the pools can also be exhausted. When this occurs, the allocations fail. Little can be done to resolve this. The computer is just running out of pool memory. The only effective solution in this case is to reduce the load on the pool, or to add more RAM up to 1.6 gigabytes (GB). At this point, the pools have reached the theoretical maximum. Note Adding additional RAM only helps if the pools are not already at their theoretical maximums. The second situation that can cause these two errors conditions is sudden load on the server. If too many receives (2021) or connection requests (2022) must be processed at the same time, the server may not find an available work item or connection, respectively. Because there are more work items, they are less susceptible to this than are connections. Additionally, work items can be taken from other processors. To resolve this situation increase the minimum work items (2021), the minimum free connections (2022), or both. The maximum free connections setting may have little effect in this case, because it is only looked at when the connection is no longer required. However, it must be greater than the minimum value. Another issue that may occur is the use of the /PAE and /3GB startup switches. If the /3GB switch is used on a Windows 2000-based computer, it sets the Paged Pool Memory back to the Windows NT 4.0 maximum amount of 192 MB. If the /PAE switch is used on a server, it may reduce available paged pool memory. The use of both switches configures the operating system with less available system resources than you would have if you use either switch alone, or none of the switches. Heavily stressed file servers should not use /3gb switch. The /PAE switch alone will not cause any issue. For truly difficult 2022 issues you can use an event monitoring program to stop a network capture when the event is recorded. Also you can contact PSS-CPR-US to obtain a diagnostic driver that will dump the server when it receives the errors listed earlier. A dump file will quickly lead to ROOT cause if you have gathered some background information about the type, use, and quantities of files opened. REFERENCES889100
(http://support.microsoft.com/kb/889100/
)
How to obtain the latest service pack for Windows Server 2003
260910
(http://support.microsoft.com/kb/260910/
)
How to obtain the latest Windows 2000 service pack
To obtain the latest Windows NT 4.0 service pack, visit the following Microsoft Web site: http://www.microsoft.com/downloads/details.aspx?FamilyID=e396d059-e402-46ef-b095-a74399e25737&DisplayLang=en
(http://www.microsoft.com/downloads/details.aspx?FamilyID=e396d059-e402-46ef-b095-a74399e25737&DisplayLang=en)
Note There is a post-Service Pack 6a (post SP6a) hotfix for Event ID
2022 issues that may apply.
APPLIES TO
| Article Translations
|
Back to the top
