Symptoms
You may experience one or more of the following symptoms. These symptoms may be intermittent or continuous. These symptoms are more likely and more widespread during "high usage" times, such as at the beginning of a business day when increased client load occurs on the servers in the environment.
You may experience the following issues in a web services scenario:
-
Web clients receive delayed responses from the web server.
-
Web clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issues in a web proxy scenario:
-
Web clients receive delayed responses from the web server.
-
Web clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issues in an Exchange client scenario:
-
Clients receive delayed responses from the server.
-
Clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issue in any scenario in which NTLM authentication is used for applications:
Line of business or custom applications that use NTLM authentication fail. Additionally, you may receive different errors that are intermittent and may include "access denied."You may experience the following issue in a remote file access scenario:
Windows clients receive "access denied" errors or delayed responses from the file server.You may experience the following issue in any scenario in which Kerberos delegation is being used in a middle-tier service:
The clients gain access successfully at first but then lose access to the same resources. Additionally, you may be repeatedly prompted for credentials or experience "access denied" errors.Notes
-
This issue is more likely to occur if one or more of the following conditions are true:
-
There are highly transactional and heavily used services in the environment.
-
There is heavy use of scripts that use the WINNT provider.
-
There are applications and services that are not configured (or are not configurable) to use Kerberos authentication.
-
When the following three conditions are true at the same time:
-
There are many "accounts" domains (in other words, domains that have user accounts in them) in the environment.
-
There are Windows Server 2003-based domain controllers (DCs) .
-
There are applications or services that may authenticate without providing the domain name. For example, there are applications or services that provide <null>\username instead of domainname\username.
-
-
-
The following symptoms indicate that this issue is occurring in the environment:
-
A Kerberos source event is logged in the System log of application servers. This event indicates that Kerberos PAC validation is failing. The event resembles the following:
-
Text in Netlogon service debug logs (Netlogon.log) matches the text "NlpUserValidateHigher: Can't allocate Client API slot." These entries may appear in any of the Netlogon debug logs of the following servers:
-
The application server
-
The domain controllers in the application servers domain
-
Trusting domain controllers
-
-
Perfmon performance logging of the Netlogon performance counter for Semaphore Timeouts during the time when the issue is occurring shows numbers greater than zero. This counter value may appear on any of the following servers in this scenario:
-
The application server
-
The domain controllers in the application servers domain
-
Trusting domain controllers
-
-
Cause
This issue occurs when a high volume of NTLM authentication or Kerberos PAC validation transactions (or both) occur on a Windows-based server, and that volume is greater than the volume that can be handled at one time by the member server or the domain controllers that are providing authentication. In other words, this is caused by an authentication resource bottleneck.
NTLM authentication and PAC validation are performed by dedicated threads in the Lsass.exe process on Windows-based computers. There is a maximum number of these threads that are available to handle these requests at the same time, and if the requests exceed the availability of the threads and the requests cannot wait any longer, this issue occurs.
By default, workstations have one of the threads available for use, and member servers have two of the threads available for use. Domain controllers have one available thread per security channel to trusted domains. This maximum number of threads that are dedicated to this purpose is known as "Maxconcurrentapi" and is configurable.
Resolution
To resolve the issue, use one or more of the following methods:
-
Install the following hotfix, and then follow the steps that are described in the "Registry information" section. After you install this hotfix on Windows Vista, Windows Server 2008, Windows 7, or Windows Server 2008 R2, the maximumlimit of concurrent connections between a client computer and another server or a domain controller for NTLM authentication or PAC validation may be changed up to 150. This should be done on all servers that exhibit Perfmon Netlogon “semaphore time-out” indications in their Performance logs or that have the “NlpUserValidateHigher: Can't allocate Client API slot” text in their Netlogon debug logs.
-
For applications and services that are using NTLM, just configure them to use Kerberos authentication instead. The methods to do that will be unique to those applications.
Note In order to decide what value to set for the MaxConcurrentApi setting in your environment refer to the Knowledge Base article below.
2688798 How to do performance tuning for NTLM authentication by using the MaxConcurrentApi setting
Hotfix information
A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem that is described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.
If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.
Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft website:
http://support.microsoft.com/contactus/?ws=supportNote The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.
Prerequisites
To apply this hotfix, your computer must be running Windows 7, Windows Server 2008 R2, Windows Vista Service Pack 2, or Windows Server 2008 Service Pack 2.
Registry information
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 How to back up and restore the registry in WindowsAfter you install the hotfix, increase the Maxconcurrentapi value to a larger number on all servers that have Perfmon Netlogon "semaphore timeout" indications in their Performance logs or that have "NlpUserValidateHigher: Can't allocate Client API slot" text in their Netlogon debug logs. To do this, follow these steps:
-
Start Registry Editor.
-
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters
-
Create the following registry entry:
Name: MaxConcurrentApi
Type: REG_DWORD
Value:Set the value to the larger number, which you tested (any number greater than the default value). -
At a command prompt, run net stop netlogon, and then run net start netlogon.
Notes
-
The maximum value that can be configured depends on the operating system version and whether a hotfix is available.
-
The maximum configurable setting in Windows Server 2003 is 10.
-
The maximum configurable setting in Windows Server 2008 (without the hotfix in this article) is 10. With the hotfix, the maximum is 150.
-
The maximum configurable setting in Windows Server 2008 R2 (without the hotfix in this article) is 10. With the hotfix, the maximum is 150.
-
-
If you decide to increase the MaxConcurrentApivalue to greater than 10, the load and the performance of the desired setting should be tested in a nonproduction environment before you implement in production. This is recommended to make sure that increasing this value does not cause other resource bottlenecks.
Restart requirement
You must restart the computer after you apply this hotfix.
Hotfix replacement information
This hotfix does not replace a previously released hotfix.
File information
The English (United States) version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.
-
The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed separately in the "Additional file information for Windows Vista and Windows Server 2008" section. MUM and MANIFEST files, and the associated security catalog (.cat) files, are very important to maintaining the state of the updated component. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.
File information for Windows Vista and for Windows Server 2008
File information for x86-based versions of Windows Server 2008 and of Windows Vista
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.0.6002.22289 |
592,896 |
16-Dec-2009 |
12:09 |
x86 |
Nlsvc.mof |
Not Applicable |
2,873 |
03-Apr-2009 |
21:24 |
Not Applicable |
File information for x64-based versions of Windows Server 2008 and of Windows Vista
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.0.6002.22289 |
716,800 |
16-Dec-2009 |
12:07 |
x64 |
Nlsvc.mof |
Not Applicable |
2,873 |
03-Apr-2009 |
20:58 |
Not Applicable |
File information for IA-64-based versions of Windows Server 2008
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.0.6002.22289 |
1,216,512 |
16-Dec-2009 |
12:05 |
IA-64 |
Nlsvc.mof |
Not Applicable |
2,873 |
03-Apr-2009 |
20:59 |
Not Applicable |
File Information for Windows 7 and for Windows Server 2008 R2
Windows 7 and Windows Server 2008 R2 file information notes
Important Windows 7 hotfixes and Windows Server 2008 R2 hotfixes are included in the same packages. However, hotfixes on the Hotfix Request page are listed under both operating systems. To request the hotfix package that applies to one or to both operating systems, select the hotfix that is listed under "Windows 7/Windows Server 2008 R2" on the page. Always refer to the "Applies To" section in articles to determine the actual operating system to which each hotfix applies.
-
The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed separately in the "Additional file information for Windows Server 2008 R2 and for Windows 7" section. MUM and MANIFEST files, and the associated security catalog (.cat) files, are very important to maintaining the state of the updated component. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.
For all supported x86-based versions of Windows 7
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.1.7600.20576 |
563,712 |
16-Nov-2009 |
06:40 |
x86 |
Nlsvc.mof |
Not Applicable |
2,873 |
10-Jun-2009 |
21:29 |
Not Applicable |
For all supported x64-based versions of Windows 7 and of Windows Server 2008 R2
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.1.7600.20576 |
692,736 |
16-Nov-2009 |
07:45 |
x64 |
Nlsvc.mof |
Not Applicable |
2,873 |
10-Jun-2009 |
20:47 |
Not Applicable |
For all supported IA-64-based versions of Windows Server 2008 R2
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Netlogon.dll |
6.1.7600.20576 |
1,148,416 |
16-Nov-2009 |
06:10 |
IA-64 |
Nlsvc.mof |
Not Applicable |
2,873 |
10-Jun-2009 |
20:52 |
Not Applicable |
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
More Information
This hotfix is included in Windows 7 Service Pack 1 (SP1) and in Windows Server 2008 R2 Service Pack 1 (SP1).
The MaxConcurrentApi setting and the default settings for it are a legacy of Windows 2000 and of the limited hardware capabilities of that time. With older hardware, allowing for additional threads and the RPC traffic they would generate was a serious concern, and there was a possibility of performance bottlenecks if too many threads were created. With newer hardware platforms and improved performance, that hardware performance limitation is less likely to occur. As always, it is important to gauge and understand the performance of the servers in an environment before you increase the potential load by using a high MaxConcurrentApi setting.
For more information about how to use Netlogon service debug logging (Netlogon.log), click the following article number to view the article in the Microsoft Knowledge Base:
109626 Enabling debug logging for the Net Logon service An additional lessening step can be performed on Windows Server 2003-based domain controllers that have entries in their Netlogon service debug log that indicate that clients are submitting <null>\username instead of domainname\username. The steps are described in the following Microsoft Knowledge Base article:
923241 The Lsass.exe process may stop responding if you have many external trusts on a Windows Server 2003-based domain controller More information about how to use the Netlogon Performance monitoring object is available, together with an update to add that Performance object in Windows Server 2003. There is an update for Windows Server 2003 that lets you monitor the speed and the throughput of NTLM authentications. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
928576 New performance counters for Windows Server 2003 let you monitor the performance of Netlogon authentication
There is an update for Windows Server 2008 R2 that introduces new events to track Netlogoan API overload:
New event log entries that track NTLM authentication delays and failures in Windows Server 2008 R2 are available
http://support.microsoft.com/default.aspx?scid=kb;EN-US;2654097
For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684
Description of the standard terminology that is used to describe Microsoft software updates
Additional file information
Additional file information for Windows Vista and for Windows Server 2008
Additional file information for x86-based versions of Windows Vista and of Windows Server 2008
File name |
Update.mum |
File version |
Not Applicable |
File size |
3,068 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
X86_d097c3e62c5fe28649de747cfa96d8cc_31bf3856ad364e35_6.0.6002.22289_none_8f4d35d9370e9c53.manifest |
File version |
Not Applicable |
File size |
705 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
X86_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_ffda50c84e769578.manifest |
File version |
Not Applicable |
File size |
22,701 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
14:05 |
Platform |
Not Applicable |
Additional file information for x64-based versions of Windows Server 2008 and of Windows Vista
File name |
Amd64_0fe0181b07108c9de21c48d7ff24c52a_31bf3856ad364e35_6.0.6002.22289_none_f87d1e5f85e49530.manifest |
File version |
Not Applicable |
File size |
1,060 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
Amd64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_5bf8ec4c06d406ae.manifest |
File version |
Not Applicable |
File size |
23,180 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
15:52 |
Platform |
Not Applicable |
File name |
Update.mum |
File version |
Not Applicable |
File size |
3,092 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_664d969e3b34c8a9.manifest |
File version |
Not Applicable |
File size |
18,332 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
14:00 |
Platform |
Not Applicable |
Additional file information for IA-64-based versions of Windows Server 2008
File name |
Ia64_93a1c37632c96e8463a7fa3608662f92_31bf3856ad364e35_6.0.6002.22289_none_f505daf555102feb.manifest |
File version |
Not Applicable |
File size |
1,058 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
Ia64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_ffdbf4be4e749e74.manifest |
File version |
Not Applicable |
File size |
23,156 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
16:08 |
Platform |
Not Applicable |
File name |
Update.mum |
File version |
Not Applicable |
File size |
2,247 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
21:16 |
Platform |
Not Applicable |
File name |
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_664d969e3b34c8a9.manifest |
File version |
Not Applicable |
File size |
18,332 |
Date (UTC) |
16-Dec-2009 |
Time (UTC) |
14:00 |
Platform |
Not Applicable |
Additional file information for Windows 7 and for Windows Server 2008 R2
Additional files for all supported x86-based versions of Windows 7
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Package_for_kb975363_rtm~31bf3856ad364e35~x86~~6.1.1.0.mum |
Not Applicable |
1,947 |
16-Nov-2009 |
09:45 |
Not Applicable |
X86_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_fe237c4db262181f.manifest |
Not Applicable |
35,541 |
16-Nov-2009 |
08:08 |
Not Applicable |
Additional files for all supported x64-based versions of Windows 7 and of Windows Server 2008 R2
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Amd64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_5a4217d16abf8955.manifest |
Not Applicable |
35,547 |
16-Nov-2009 |
08:11 |
Not Applicable |
Package_for_kb975363_rtm~31bf3856ad364e35~amd64~~6.1.1.0.mum |
Not Applicable |
2,181 |
16-Nov-2009 |
09:45 |
Not Applicable |
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_6496c2239f204b50.manifest |
Not Applicable |
16,596 |
16-Nov-2009 |
08:01 |
Not Applicable |
Additional files for all supported IA-64-based versions of Windows Server 2008 R2
File name |
File version |
File size |
Date |
Time |
Platform |
---|---|---|---|---|---|
Ia64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_fe252043b260211b.manifest |
Not Applicable |
35,544 |
16-Nov-2009 |
09:06 |
Not Applicable |
Package_for_kb975363_rtm~31bf3856ad364e35~ia64~~6.1.1.0.mum |
Not Applicable |
1,683 |
16-Nov-2009 |
09:45 |
Not Applicable |
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_6496c2239f204b50.manifest |
Not Applicable |
16,596 |
16-Nov-2009 |
08:01 |
Not Applicable |