Deleting Active Directory objects that have many links causes replication failures

Summary
This article discusses an issue that occurs when you delete Active Directory objects that contain many forward and backward links.

The registry key that is discussed in this article should be applied only to domain controllers (DCs) that are experiencing the issue that is described in the "Symptoms" section. This issue is likely to occur on Windows Server 2012 and Windows Server 2012 R2 DCs. By following the recommendations that are given here, you may decrease Active Directory replication performance but increase the reliability of correctly processing the deletion of such large objects.
Symptoms
When you delete Active Directory objects that contain many forward and backward links, you encounter replication failure. For example, you delete objects that contain large group membership sets, or you demote some RODC computer accounts that have many permission settings.

The following conditions are the key indicators that this solution applies to the issue:

  • The forest functional level is Windows Server 2003 or later version of Windows Server.
  • Event 2094 (replication delay) occurs several times, referencing the same deleted object.
  • Event 1083 (Write conflict) occurs around the same time the 2094 event referencing the same deleted object.
  • The affected domain controller (DC) may also report that the version store is exhausted (Event ID 623). Exhaustion of version store does not always occur in this scenario. Other factors that increase the likelihood of version store exhaustion include a high rate of changes to Active Directory objects, both local and replicated, as well as other long running operations such as deep queries.
If the Active Directory recycle bin is enabled, the replication errors may not occur for 60 to 180 days (deleted object lifetime) after the object is deleted.

Event log entries

When the issue occurs, the following events are logged:

Event ID: 2094 
Log Name: Domain Service
Task Category: Replication
Level: Warning
Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes. Object DN: CN=cm12847026\0ADEL:bf70880b-3d7f-4c1f-b43d-bbca00fd8f91,CN=Deleted Objects,DC=<dcname>,DC=com
Object GUID: <objectGUID>
Partition DN: DC=<dcname>,DC=com
Server: <NTDSA>._msdcs.contoso.com
Elapsed Time (secs): 13


Event ID: 1083
Log Name: Domain Service
Source: NTDS ISAM
Level: Warning
Description:
"Active Directory Domain Services could not update the following object with changes received from the directory service at the following network address because Active Directory Domain Services was busy processing information.
Object:
CN=cm12847026\0ADEL:bf70880b-3d7f-4c1f-b43d-bbca00fd8f91,CN=Deleted Objects,DC=<dcname>,DC=com
Network address:
0248b610-69f4-44a0-bb73-589165a0184d._msdcs.contoso.com
This operation will be tried again later."


Event ID: 623
Log Name: Directory Service
Source: NTDS ISAM
Task Category: Backup
Level: Error
Description:
NTDS (812) NTDSA: The version store for this instance (0) has reached its maximum size of <version store size>Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.
Possible long-running transaction:

More information
By default, when you run multiple passes to delete Active Directory objects that have an exceptionally large number of forward and backward links, 10,000 links are deleted at a time. During this time, if other threads have to update the target objects of these links, the link deletion transaction is suspended until the objects are available again. This suspension can cause the whole deletion transaction to take a long time to finish.

During this time, users may see write conflicts and transaction failure events. Also, as additional objects are processed by replication, more and more version store is allocated because the pending large transaction does not release its allocated versions store until the deletion transaction is finished. This can cause version store errors and replication warnings events.

Notes

  • Garbage collection is not related to the processing of group membership link deletions.
  • The legacy value for Links process batch size is 1,000 in versions before Windows Server 2008 R2. In later versions, the batch size is increased to 10,000 to improve the performance of undeleting in forests that have the Recycle Bin enabled.

Active Directory services check for the following registry key.

For AD DS:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Links process batch size

For AD LDS:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\<adam instance>\Parameters\Links process batch size

Type: DWORD

Min Value: 1000

Max Value: 10000

This value overrides the default value of 10,000 as the number of atomic links to process at one time. After each atomic operation, the corresponding version store is released. The version store is reacquired only during the next atomic operation that continues to process the same object.
Workaround
To work around this issue, set the value of links process batch size lower than 10,000. This decreases the potential for an object access collision to occur. By doing this, you make the replication process of large object deletion more reliable. Also, it now takes a longer time to complete the whole transaction. This helps you avoid version store depletion.
Properties

Article ID: 3149779 - Last Review: 11/09/2016 21:36:00 - Revision: 7.2

Windows Server 2012 R2 Standard, Windows Server 2012 Standard

  • kbexpertiseadvanced kbsurveynew kbtshoot KB3149779
Feedback