Memory leak in LSASS process on Windows Server 2012 R2-based domain controllers and AD LDS server

Symptoms

A new arena heap memory allocation was introduced in the Windows Server 2012 R2 version of Directory Services. It causes a memory leak in the Lsass.exe (domain controller role) and DsaMain.exe of Lightweight Directory Service (LDS) process that may consume all the available memory on your Windows Server 2012 R2 domain controller or LDS server.

Cause

This issue occurs in the following situations:

• Domain controller promotion or adding LDS replica to configuration set
  
  After adding a replica to a configuration by domain controller promotion or adding a LDS instance on Windows Server 2012 R2 or Windows 8.1, the replication engine has to replicate every object in the naming contexts needed on the new instance. During the task, Local Security Authority Server Service (LSASS) process or DsaMain.exe may cause a severe leak when it allocates virtual committed bytes although the real usage is very low. Additionally, DCpromo.log shows the following error message:

  
  Date Time[INFO] Replicating data DC=Contoso,DC=com: Received XXXXXX out of approximately XXXXXX objects and XXXXXX out of approximately XXXXXX distinguished name (DN) values...
  
  Date Time[WARNING] Non critical replication returned 14
  

  
  Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.
  
  The following event is logged in the event log during or shortly after dcpromo finishes:

  Event Log	Event Source	ID	Description
  Directory Services	Replication	2094	Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes. Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com Object GUID: GUID Partition DN: DC=Contoso,DC=com Server: _msdcs DNS record of replication partner Elapsed Time (secs): XX User Action A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions. If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check. Additional Data Warning Limit (secs): XX Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
  Directory Services	KCC	1308	The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed. Attempts: 2 Directory service: CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com Period of time (minutes): XXXXXXX The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed. Additional Data Error value: 14 Not enough storage is available to complete this operation.

  
  Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

• The following event is logged in the event log during or shortly after dcpromo finishes:

  Event Log	Event Source	ID	Description
  Directory Services	Replication	2094	Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes. Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com Object GUID: GUID Partition DN: DC=Contoso,DC=com Server: _msdcs DNS record of replication partner Elapsed Time (secs): XX User Action A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions. If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check. Additional Data Warning Limit (secs): XX Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
  Directory Services	KCC	1308	The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed. Attempts: 2 Directory service: CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com Period of time (minutes): XXXXXXX The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed. Additional Data Error value: 14 Not enough storage is available to complete this operation.

  
  Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

• Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.The following event is logged in the event log during or shortly after dcpromo finishes:

  Event Log	Event Source	ID	Description
  Directory Services	Replication	2094	Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes. Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com Object GUID: GUID Partition DN: DC=Contoso,DC=com Server: _msdcs DNS record of replication partner Elapsed Time (secs): XX User Action A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions. If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check. Additional Data Warning Limit (secs): XX Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
  Directory Services	KCC	1308	The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed. Attempts: 2 Directory service: CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com Period of time (minutes): XXXXXXX The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed. Additional Data Error value: 14 Not enough storage is available to complete this operation.

  
  Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

• Security descriptor (SD) propagation
  
  A memory leak issue may occur when a security change on a container object (domain root or organizational unit (OU)) is inherited on many child objects or subordinate OUs though SD propagation. Depending on the size of the access control entry (ACE) to be changed and the number of objects (for example, 500,000), the propagation may take a long time. During this time, the LSASS process or DsaMain process may constantly allocate committed bytes.

• Long-running queries
  
  Unexpectedly high virtual memory consumption during long-running Lightweight Directory Access Protocol (LDAP) queries on Windows Server 2012 R2 or Windows 8.1-based LDAP servers may result in memory outages. The long-running queries consume memory by obtaining a heap for the security descriptor of each object that is touched.

In these situations, when the committed bytes reach the number of bytes that are available, the system becomes unusable and may even crash.

Resolution

To fix this issue, apply the hotfix that is mentioned in the following section.

To use the hotfix in the context of Domain Controller Promotions (DCPROMO), follow these steps:

1. Install the Active Directory Domain Services or AD LDS role.

2. Install this update or newer Windows Server 2012 R2 updates and security updates that contain the same Ntdsai.dll binary as they are always cumulative.

3. Promote the computer to a status of domain controller.

Important If you install a language pack after you install this hotfix, you must reinstall this hotfix. Therefore, we recommend that you install any language packs that you need before you install this hotfix. For more information, see Add language packs to Windows.

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

More Information

If you enable NTDS\Diagnostics "9 Internal Processing" level 2 as mentioned in Microsoft Knowledge Base article 314980, you may see the following ActiveDirectory_DomainService events. These events show the long-running SD propagation task.

Event 1257 - Indicating the start of the SD Propagation
Internal event: The security descriptor propagation task is processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com

Event 2007 - Indicating the progress of the SD Propagation, logged every 5 minutes
Internal event: The security descriptor propagation task has reached the following container and will continue with the propagation.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed so far: XXXXXX

Event 1258 - Indicating the end of the SD Propagation, after some hours
Internal event: The security descriptor propagation task has finished processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed: ZZZZZZ

After Active Directory replication to any other Windows Server 2012 R2-based domain controller or LDS instances in the domain, the same issue may occur because the SD propagation is performed locally.

You won't see the memory leak in the Active Directory Collector Set Report because it shows only used bytes. However, you can use the created performance monitor data BLG file-checking counter Object:Process, Instance:Lsass, Counter:Private Bytes, and Object:Memory, Counter:Committed Bytes.

For longer observation of the leak development, you can use a "Graph with Performance Monitor Properties, Graph Elements, Sample" every 60 seconds. Duration: 15,000 seconds (> 4 hours).

The increasing allocation can also be observed in Task Manager, tab Performance, item Memory, Committed. This is displayed under Committed/Available gigabytes (GB).

Indicators of the error condition are as follows:

repadmin /showrepl returns:

Directory Service logs Error event 1699:

Application popup:

References

Learn about the terminology that Microsoft uses to describe software updates.

File Information

The English (United States) version of this software update installs files that have the attributes that are listed in the following tables. The dates and times for these files are listed in Coordinated Universal Time (UTC). Be aware that dates and times for these files on your local computer are displayed in your local time and with your current daylight saving time bias. The dates and times may also change when you perform certain operations on the files.