Memory leak in LSASS process on Windows Server 2012 R2-based domain controllers and AD LDS server

This article describes a memory leak issue that occurs on Windows Server 2012 R2-based domain controllers and on a Windows 8.1 or Windows Server 2012 R2-based computer that has the Active Directory Lightweight Directory Services (AD LDS) server role installed. A hotfix is available to fix this issue. The hotfix has a prerequisite.

Symptoms

A new arena heap memory allocation was introduced in the Windows Server 2012 R2 version of Directory Services. It causes a memory leak in the Lsass.exe (domain controller role) and DsaMain.exe of Lightweight Directory Service (LDS) process that may consume all the available memory on your Windows Server 2012 R2 domain controller or LDS server.

Cause

This issue occurs in the following situations:
  • Domain controller promotion or adding LDS replica to configuration set

    After adding a replica to a configuration by domain controller promotion or adding a LDS instance on Windows Server 2012 R2 or Windows 8.1, the replication engine has to replicate every object in the naming contexts needed on the new instance. During the task, Local Security Authority Server Service (LSASS) process or DsaMain.exe may cause a severe leak when it allocates virtual committed bytes although the real usage is very low. Additionally, DCpromo.log shows the following error message:

    Date
    Time[INFO] Replicating data DC=Contoso,DC=com: Received XXXXXX out of approximately XXXXXX objects and XXXXXX out of approximately XXXXXX distinguished name (DN) values...

    Date Time[WARNING] Non critical replication returned 14

    Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.

    The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation
    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation
    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation
    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • Security descriptor (SD) propagation

    A memory leak issue may occur when a security change on a container object (domain root or organizational unit (OU)) is inherited on many child objects or subordinate OUs though SD propagation. Depending on the size of the access control entry (ACE) to be changed and the number of objects (for example, 500,000), the propagation may take a long time. During this time, the LSASS process or DsaMain process may constantly allocate committed bytes.
  • Long-running queries

    Unexpectedly high virtual memory consumption during long-running Lightweight Directory Access Protocol (LDAP) queries on Windows Server 2012 R2 or Windows 8.1-based LDAP servers may result in memory outages. The long-running queries consume memory by obtaining a heap for the security descriptor of each object that is touched.
In these situations, when the committed bytes reach the number of bytes that are available, the system becomes unusable and may even crash.

Resolution

To fix this issue, apply the hotfix that is mentioned in the following section.

To use the hotfix in the context of Domain Controller Promotions (DCPROMO), follow these steps:
  1. Install the Active Directory Domain Services or AD LDS role.
  2. Install this update or newer Windows Server 2012 R2 updates and security updates that contain the same Ntdsai.dll binary as they are always cumulative.
  3. Promote the computer to a status of domain controller.
Important If you install a language pack after you install this hotfix, you must reinstall this hotfix. Therefore, we recommend that you install any language packs that you need before you install this hotfix. For more information, see Add language packs to Windows.

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that's described in this article. Apply this hotfix only to systems that are experiencing this specific problem.

If the hotfix is available for download, there's a "Hotfix Download Available" section at the top of this Knowledge Base article. If this section doesn't appear, submit a request to Microsoft Customer Service and Support to get the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that don't qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, go to the following Microsoft website: Note The "Hotfix Download Available" form displays the languages for which the hotfix is available. If you don't see your language, it's because a hotfix isn't available for that language.

Prerequisites

To apply this hotfix, you must have the April 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (2919355) installed on Windows 8.1 or Windows Server 2012 R2.

Registry information

To apply this hotfix, you don't have to make any changes to the registry.

Restart requirement

You may have to restart the computer after you apply this hotfix.

Hotfix replacement information

This hotfix doesn't replace a previously released hotfix.

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

More Information

If you enable NTDS\Diagnostics "9 Internal Processing" level 2 as mentioned in Microsoft Knowledge Base article 314980, you may see the following ActiveDirectory_DomainService events. These events show the long-running SD propagation task.
After Active Directory replication to any other Windows Server 2012 R2-based domain controller or LDS instances in the domain, the same issue may occur because the SD propagation is performed locally.

You won't see the memory leak in the Active Directory Collector Set Report because it shows only used bytes. However, you can use the created performance monitor data BLG file-checking counter Object:Process, Instance:Lsass, Counter:Private Bytes, and Object:Memory, Counter:Committed Bytes.

For longer observation of the leak development, you can use a "Graph with Performance Monitor Properties, Graph Elements, Sample" every 60 seconds. Duration: 15,000 seconds (> 4 hours).

The increasing allocation can also be observed in Task Manager, tab Performance, item Memory, Committed. This is displayed under Committed/Available gigabytes (GB).

Indicators of the error condition are as follows:

repadmin /showrepl returns:
Not enough storage is available to complete this operation.

Directory Service logs Error event 1699:
This directory service failed to retrieve the changes requested for the following directory partition. As a result, it was unable to send change requests to the directory service at the following network address.
Extended request code: 0
Additional Data
Error value: 8446 The replication operation failed to allocate memory.

Application popup:
Windows - Out of Virtual Memory: Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.


References

Learn about the terminology that Microsoft uses to describe software updates.

File Information

The English (United States) version of this software update installs files that have the attributes that are listed in the following tables. The dates and times for these files are listed in Coordinated Universal Time (UTC). Be aware that dates and times for these files on your local computer are displayed in your local time and with your current daylight saving time bias. The dates and times may also change when you perform certain operations on the files.
Windows 8.1 and Windows Server 2012 R2
Additional file information
Eigenschaften

Artikelnummer: 3083038 – Letzte Überarbeitung: 07.07.2016 – Revision: 1

Feedback