This article describes a memory leak issue that occurs on Windows Server 2012 R2-based domain controllers and on a Windows 8.1 or Windows Server 2012 R2-based computer that has the Active Directory Lightweight Directory Services (AD LDS) server role installed. A hotfix is available to fix this issue. The hotfix has a prerequisite.

Symptoms

A new arena heap memory allocation was introduced in the Windows Server 2012 R2 version of Directory Services. It causes a memory leak in the Lsass.exe (domain controller role) and DsaMain.exe of Lightweight Directory Service (LDS) process that may consume all the available memory on your Windows Server 2012 R2 domain controller or LDS server.

Cause

This issue occurs in the following situations:

  • Domain controller promotion or adding LDS replica to configuration set

    After adding a replica to a configuration by domain controller promotion or adding a LDS instance on Windows Server 2012 R2 or Windows 8.1, the replication engine has to replicate every object in the naming contexts needed on the new instance. During the task, Local Security Authority Server Service (LSASS) process or DsaMain.exe may cause a severe leak when it allocates virtual committed bytes although the real usage is very low. Additionally, DCpromo.log shows the following error message:


    Date
    Time[INFO] Replicating data DC=Contoso,DC=com: Received XXXXXX out of approximately XXXXXX objects and XXXXXX out of approximately XXXXXX distinguished name (DN) values...

    Date Time[WARNING] Non critical replication returned 14


    Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.

    The following event is logged in the event log during or shortly after dcpromo finishes:

    Event Log

    Event Source

    ID

    Description

    Directory Services

    Replication

    2094

    Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)

    Directory Services

    KCC

    1308

    The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation.


    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

  • The following event is logged in the event log during or shortly after dcpromo finishes:

    Event Log

    Event Source

    ID

    Description

    Directory Services

    Replication

    2094

    Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)

    Directory Services

    KCC

    1308

    The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation.


    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

  • Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.The following event is logged in the event log during or shortly after dcpromo finishes:

    Event Log

    Event Source

    ID

    Description

    Directory Services

    Replication

    2094

    Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)

    Directory Services

    KCC

    1308

    The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation.


    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.

  • Security descriptor (SD) propagation

    A memory leak issue may occur when a security change on a container object (domain root or organizational unit (OU)) is inherited on many child objects or subordinate OUs though SD propagation. Depending on the size of the access control entry (ACE) to be changed and the number of objects (for example, 500,000), the propagation may take a long time. During this time, the LSASS process or DsaMain process may constantly allocate committed bytes.

  • Long-running queries

    Unexpectedly high virtual memory consumption during long-running Lightweight Directory Access Protocol (LDAP) queries on Windows Server 2012 R2 or Windows 8.1-based LDAP servers may result in memory outages. The long-running queries consume memory by obtaining a heap for the security descriptor of each object that is touched.

In these situations, when the committed bytes reach the number of bytes that are available, the system becomes unusable and may even crash.

Resolution

To fix this issue, apply the hotfix that is mentioned in the following section.

To use the hotfix in the context of Domain Controller Promotions (DCPROMO), follow these steps:

  1. Install the Active Directory Domain Services or AD LDS role.

  2. Install this update or newer Windows Server 2012 R2 updates and security updates that contain the same Ntdsai.dll binary as they are always cumulative.

  3. Promote the computer to a status of domain controller.

Important If you install a language pack after you install this hotfix, you must reinstall this hotfix. Therefore, we recommend that you install any language packs that you need before you install this hotfix. For more information, see Add language packs to Windows.

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that's described in this article. Apply this hotfix only to systems that are experiencing this specific problem.

If the hotfix is available for download, there's a "Hotfix Download Available" section at the top of this Knowledge Base article. If this section doesn't appear, submit a request to Microsoft Customer Service and Support to get the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that don't qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, go to the following Microsoft website:

http://support.microsoft.com/contactus/?ws=supportNote The "Hotfix Download Available" form displays the languages for which the hotfix is available. If you don't see your language, it's because a hotfix isn't available for that language.

Prerequisites

To apply this hotfix, you must have the April 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (2919355) installed on Windows 8.1 or Windows Server 2012 R2.

Registry information

To apply this hotfix, you don't have to make any changes to the registry.

Restart requirement

You may have to restart the computer after you apply this hotfix.

Hotfix replacement information

This hotfix doesn't replace a previously released hotfix.

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

More Information

If you enable NTDS\Diagnostics "9 Internal Processing" level 2 as mentioned in Microsoft Knowledge Base article 314980, you may see the following ActiveDirectory_DomainService events. These events show the long-running SD propagation task.

Event 1257 - Indicating the start of the SD Propagation
Internal event: The security descriptor propagation task is processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com

Event 2007 - Indicating the progress of the SD Propagation, logged every 5 minutes
Internal event: The security descriptor propagation task has reached the following container and will continue with the propagation.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed so far: XXXXXX

Event 1258 - Indicating the end of the SD Propagation, after some hours
Internal event: The security descriptor propagation task has finished processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed: ZZZZZZ

After Active Directory replication to any other Windows Server 2012 R2-based domain controller or LDS instances in the domain, the same issue may occur because the SD propagation is performed locally.

You won't see the memory leak in the Active Directory Collector Set Report because it shows only used bytes. However, you can use the created performance monitor data BLG file-checking counter Object:Process, Instance:Lsass, Counter:Private Bytes, and Object:Memory, Counter:Committed Bytes.

For longer observation of the leak development, you can use a "Graph with Performance Monitor Properties, Graph Elements, Sample" every 60 seconds. Duration: 15,000 seconds (> 4 hours).

The increasing allocation can also be observed in Task Manager, tab Performance, item Memory, Committed. This is displayed under Committed/Available gigabytes (GB).

Indicators of the error condition are as follows:

repadmin /showrepl returns:

Not enough storage is available to complete this operation.


Directory Service logs Error event 1699:

This directory service failed to retrieve the changes requested for the following directory partition. As a result, it was unable to send change requests to the directory service at the following network address.
Extended request code: 0
Additional Data
Error value: 8446 The replication operation failed to allocate memory.


Application popup:

Windows - Out of Virtual Memory: Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.


 

References

Learn about the terminology that Microsoft uses to describe software updates.

File Information

The English (United States) version of this software update installs files that have the attributes that are listed in the following tables. The dates and times for these files are listed in Coordinated Universal Time (UTC). Be aware that dates and times for these files on your local computer are displayed in your local time and with your current daylight saving time bias. The dates and times may also change when you perform certain operations on the files.

Important Windows 8.1 hotfixes and Windows Server 2012 R2 hotfixes are included in the same packages. However, hotfixes on the Hotfix Request page are listed under both operating systems. To request the hotfix package that applies to one or both operating systems, select the hotfix that is listed under "Windows 8.1/Windows Server 2012 R2" on the page. Always refer to the "Applies To" section in articles to determine the actual operating system that each hotfix applies to.

Notes

  • The files that apply to a specific product, milestone (RTM, SPn), and service branch (LDR, GDR) can be identified by examining the file version numbers as shown in the following table:

    Version

    Product

    Milestone

    Service branch

    6.3.960 0.18 xxx

    Windows 8.1 and Windows Server 2012 R2

    RTM

    GDR

  • GDR service branches contain only those fixes that are widely released to address widespread, critical issues. LDR service branches contain hotfixes in addition to widely released fixes.

  • The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed in the "Additional file information" section. MUM, MANIFEST, and the associated security catalog (.cat) files, are very important to maintain the state of the updated components. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.

x86 Windows 8.1

File name

File version

File size

Date

Time

Platform

Ntdsa.mof

Not applicable

227,765

18-Jun-2013

12:21

Not applicable

Ntdsai.dll

6.3.9600.18116

2,583,552

03-Nov-2015

02:41

x86

x64 Windows 8.1 and Windows Server 2012 R2

File name

File version

File size

Date

Time

Platform

Ntdsa.mof

Not applicable

227,765

18-Jun-2013

14:45

Not applicable

Ntdsai.dll

6.3.9600.18116

3,676,160

03-Nov-2015

02:54

x64


x86 Windows 8.1

File property

Value

File name

Update.mum

File version

Not applicable

File size

1,600

Date (UTC)

04-Nov-2015

Time (UTC)

05:53

Platform

Not applicable

File name

X86_532408894ea01e6280e568d78cbfbc21_31bf3856ad364e35_6.3.9600.18116_none_70de56a241809c7b.manifest

File version

Not applicable

File size

712

Date (UTC)

04-Nov-2015

Time (UTC)

05:53

Platform

Not applicable

File name

X86_microsoft-windows-d..toryservices-ntdsai_31bf3856ad364e35_6.3.9600.18116_none_85b3851fd48201e8.manifest

File version

Not applicable

File size

3,352

Date (UTC)

03-Nov-2015

Time (UTC)

05:09

Platform

Not applicable

x64 Windows 8.1 and Windows Server 2012 R2

File property

Value

File name

Amd64_1c835b965fc6e60c22f413386606599e_31bf3856ad364e35_6.3.9600.18116_none_b800a1506ce79afa.manifest

File version

Not applicable

File size

716

Date (UTC)

04-Nov-2015

Time (UTC)

05:53

Platform

Not applicable

File name

Amd64_microsoft-windows-d..toryservices-ntdsai_31bf3856ad364e35_6.3.9600.18116_none_e1d220a38cdf731e.manifest

File version

Not applicable

File size

3,356

Date (UTC)

03-Nov-2015

Time (UTC)

06:04

Platform

Not applicable

File name

Update.mum

File version

Not applicable

File size

2,061

Date (UTC)

04-Nov-2015

Time (UTC)

05:53

Platform

Not applicable


Need more help?

Expand your skills
Explore Training
Get new features first
Join Microsoft Insiders

Was this information helpful?

What affected your experience?

Any additional feedback? (Optional)

Thank you for your feedback!

×