ADREPL: Troubleshooting AD Replication error 8461

Slow AD Replication troubleshooting

This issue requires advanced knowledge of multiple troubleshooting tools. It may be necessary to seek the help of Microsoft Support to assist with the data analysis process.
 

Terminology:  Slow vs. latent

In slow AD replication, changes are committed to the Active Directory database slowly or replication tasks take a long time to process.

Common causes include:
 

• DC / OS performance problems
   
  • Resource depletion
  • Disk bottleneck
• AD database problems
   
  • Logical or physical corruption
  • Object or attribute issues
• DC load issues
   
  • Handling too many clients
  • Replication storm
     
    • High rate of change to objects and attributes
    • Full partition syncs

In latent AD replication, the DC is not notified about changes for a long time:

Common causes include:
 

• AD Topology Configuration (schedule, site links, replication schedule, disconnected topology)

This document and data collection strategy is meant to be used for troubleshooting Slow AD Replication.
 

Symptoms of slow AD Replication

Data collection

Repadmin Data

Use Repadmin /queue to document the queued replication tasks.  Monitor the queue to see if there is a delay in processing replication tasks.  Log all repadmin /queue output to the same text file so you have good historical data.
 

Repadmin /queue DCName >DCNameReplQueue.txt Choice /d y /t 300 Repadmin /queue DCName >>DCNameReplQueue.txt Choice /d y /t 300 Repadmin /queue DCName >>DCNameReplQueue.txt Choice /d y /t 900 Repadmin /queue DCName >>DCNameReplQueue.txt Choice /d y /t 900 Repadmin /queue DCName >>DCNameReplQueue.txt Choice /d y /t 1800Repadmin /queue DCName >>DCNameReplQueue.txt 

Review the output to see whether replication tasks are processed in a timely manner. The top of the file contains the currently running task and the length of time it has been running. If the same task is always at the top of the output, you can use verbose output of repadmin /showrepl to monitor the progress.
 

Repadmin changes

Repadmin /showrepl

Use Repadmin /showrepl and the /verbose option to monitor the last replication status and the number of changes that remain to be replicated.

Repadmin /showrepl /verbose DCNameDomainDN

Repadmin /showrepl /verbose 5thwardCorpDC dc=corp,dc=contoso,dc=com 

To limit the output so that only the desired Source DC is displayed, use the following:

Repadmin /showrepl /verbose DCNameDomainDN SourceDcDSAObjectGUID

Repadmin /showrepl /verbose 5thwardCorpDC c7f89ed3-0f4a-4ae7-8a20-b025b4556358 dc=corp,dc=contoso,dc=com 

Repadmin /showutdvec

Use Repadmin /showutdvec on the source and destination DC to determine replication delta.

Repadmin /showutdvec DCName DomainDN
 

repadmin /showutdvec LiverpoolDC1 dc=north,dc=fabrikam,dc=com 

Obtain Repadmin /showutdvec /latency from the source and destination DC for the partition in question.

 In the output, document the following:
 

1. From source DCs showutdvec:
   1. USN # next to Source DCName
2. From destination DCs showutdvec
   1. USN# next to SOURCE DCNanme

Source DC
 

Dallas\2008DC1                       @ USN    451265 @ Time 2013-07-10 13:30:43 Dallas\SourceDC                      @ USN   1126541 @ Time 2013-07-12 08:17:16 Houston\2012DC3                      @ USN    469842 @ Time 2013-07-10 13:14:27 66a1b264-80b8-44f8-8356-9ebcd7a34f15 @ USN     32811 @ Time 2013-06-06 08:26:56 Dallas\2012DC2                       @ USN    460219 @ Time 2013-07-10 13:30:44 Dallas\DestinationDC                 @ USN   1353465 @ Time 2013-07-11 14:16:40 Dallas\2003DC1                       @ USN     15556 @ Time 2013-07-10 13:21:50 

Destination DC
 

Dallas\2008DC1                       @ USN    451265 @ Time 2013-07-10 13:30:43 Dallas\SourceDC                      @ USN    801224 @ Time 2013-07-11 14:17:04 Houston\2012DC3                      @ USN    469842 @ Time 2013-07-10 13:14:27 66a1b264-80b8-44f8-8356-9ebcd7a34f15 @ USN     32811 @ Time 2013-06-06 08:26:56 Dallas\2012DC2                       @ USN    460219 @ Time 2013-07-10 13:30:44 Dallas\DestinationDC                 @ USN   1359087 @ Time 2013-07-12 07:59:37 Dallas\2003DC1                       @ USN     15556 @ Time 2013-07-10 13:21:50 

Source DC
 

SourceDC @ USN 1126541 

Destination DC
 

SourceDC @ USN 801224 

1126541-801224 = 325317

The destination DC is 325,317 USNs behind.

Note You can also get the Source DCs highestcommitedUSN from the Source DC by using this command:

repadmin /showattr SourceDC . /atts:highestcommittedusn DN: 1>  highestCommittedUSN: 1126541 

Performance Data

Create a new User Defined Data Collector set In Performance Monitor that uses the AD Diagnostics template.

The steps here detail how to set up a good set of baseline DC performance counters.  You will need to modify some of the settings, such as duration and sample interval to fit your specific scenario. 

How to Create a User Defined Data Collection Set

1. Open Server Manager on a Full version of Windows Server 2008 or a later version.
2. Expand Diagnostics > Performance > Data Collector Sets.
3. Right-click User Defined and select New > Data Collector Set.
4. Type in a name such as AD DS Diagnostics, and leave the default selection of Create from a template (Recommended) selected. Then, select Next.
5. Select Active Directory Diagnostics from the list of templates and then select Next and follow the Wizard prompts (making any changes you think are necessary).
6. Right-click the new User Defined data collector set and view the Properties.
7. To change the run time, modify the Overall Duration settings in the Stop Condition tab, and then click OK to apply the changes.
   
   Options to consider:
    
   • Overall Duration –you can have the data collector stop after collecting for a set amount of time
   • Limits –you can have the data collect stop after a time or size threshold is reached (with the option to have it automatically restart)
     
     Setting limits is advantageous when you want to limit the log size.
      
     • Restart the data collector set at limits
     • Duration
     • Maximum Size
     
     

View Performance Counter Properties for the User Defined Data Collector Set
 

1. Select your new User Defined data collector set.
2. Right-click Performance Counter, and then select Properties.

Here you have the options of changing the Sample interval and adding or removing additional counters.

For this scenario, the default-sampling interval of 3 seconds should be sufficient. However, for much longer sampling times, 3 seconds is too frequent an interval.

All recommended counters are included in the default AD Diagnostic’s collector set with three exceptions:
 

• Database ==>Instances(lsass/NTDSA)\ *
• LogicalDisk(*)\*
  
  For LogicalDisk: all instances is not required - System drive and drives where database and logs are stored should be included at minimumSecurity System-Wide Statistics\*
• Security System-Wide Statistics\*

To add the AD DS database counters to the User Defined Data Collection Set
 

1. In Performance Counter Properties, select Add.
2. Expand Database ==>Instances (all counters should be highlighted).
3. Under Instances of selected object, select lsass/NTDSA
4. Select Add, and then click OK.
   
5. Add the LogicalDisk and Security System-Wide Statistics objects also.
   

After the settings are configured to your liking, you can run the new data collector set directly from Server Manager or export it and deploy it to specific servers.

When you are ready to begin collecting performance data, start your newly defined Collection Set:

To start your newly defined Data Collection Set from the MMC:
 

1. In the Performance Monitor mmc, navigate to Performance / Data Collector Sets / User Defined
2. Right click the new collection set AD DS Diagnostics and select Start.
3. You can verify that it has started by selecting User Defined
   AD DS Diagnostics should have a status of Running

After testing is complete, stop the AD DS Diagnostics Collection Set:
 

1. In the Performance Monitor mmc, navigate to Performance / Data Collector Sets / User Defined.
2. Right-click the new collection set AD DS Diagnostics, and select Stop. You can verify that it has stopped by selecting User Defined.
   
   AD DS Diagnostics should go from a status of Running to Compiling and finally Stopped

Please note that the MMC is not great about refreshing its view. So, if it seems to be stuck in Running or Compiling after you click Stop, try refreshing the screen.

The compiled report is viewable under Performance / Reports / User Defined / AD DS Diagnostics

The default path in Windows Explorer is here: %systemdrive%\PerfLogs\Admin\AD DS Diagnostics

 
Command Line instructions:  Gather AD Diagnostics from the command line:
 

To START a collection of data from the command line issue this command from an elevated command prompt:logman start "user defined\AD DS Diagnostics" –ets To STOP the collection of data before the default 5 minutes, issue this command: (get at least one full five minute sample for this issue) logman stop "user defined\AD DS Diagnostics" –ets 

 

Diagnostic data is logged here:  C:\PerfLogs\Admin\AD DS Diagnostics\DateStamp

Sample Data Collection Script

logman start "system\Active Directory Diagnostics" -ets time /t >slow.txt repadmin /showrepl /verbose LiverpoolDC1 dc=north,dc=fabrikam,dc=com >slowrepl.txt repadmin /queue LiverpoolDC1 >>slow.txt repadmin /showutdvec LiverpoolDC1 dc=north,dc=fabrikam,dc=com >>slow.txt :start ping 127.0.0.1 -n 60 >NUL time /t >>slow.txt time /t >>slowrepl.txt repadmin /showrepl /verbose LiverpoolDC1 dc=north,dc=fabrikam,dc=com >>slowrepl.txt repadmin /queue LiverpoolDC1 >>slow.txt repadmin /showutdvec LiverpoolDC1 dc=north,dc=fabrikam,dc=com >>slow.txt goto start 

Event log data

The default logging enabled in the directory services event log is useful to monitor for events that indicate slow application of changes. (EVENT X)  Verbose diagnostic logging can be enabled to see what changes are currently being applied.  Enabling diagnostic logging at the level mentioned in this article will cause the log to fill up rather quickly, so only enable it while actively troubleshooting this condition.  To give an idea of rate of events logged with this level of verbosity:

A Directory Services event log configured to 100 MB wrapped in less than two minutes (1 minute 27 seconds).  The log contained 195,728 events. Of all events, 189,340 were event ID 1412 (attribute addition).
 

• Increase the size of the Directory Services event log
• Size the event log so that the enhanced logging doesn't cause the log to wrap in too short of a period
• Enable AD Replication Diagnostic logging to 0x5:

  HKLM\System\CurrentControlSet\Services\NTDS\Diagnostics5 Replication Events = 5 

Export the Directory Services event log shortly after you receive the status 8461, and reduce the diagnostic logging to a suitable level. 

Review the event log for the following:

How quickly are attribute values are written to the database? ->Directory Services event log Event ID 1412 or even better, use performance counter:DirectoryServices/DRA Inbound Properties Applied/sec

At diagnostic level 5 for Replication Events, for user object creation, around 25 or so event 1412's (depending on what was written at time of user creation) are written (one per attribute value).  When all attributes have been added, the object creation event is logged (Event ID 1365).

The description section of the event contains the following data:

Internal event: The following object changes were applied to the local Active Directory Domain Services database. Property: 900dd (sAMAccountName) Object: CN=xtestingusers14167,CN=Users,DC=north,DC=fabrikam,DC=com Object GUID: 984a8f06-aa4b-4f80-8b40-0f78b5066ae0 Remote version: 1 Remote timestamp: 2013-07-10 14:24:31 Remote Originating USN: 540828 

The Property section contains both the attributeID and the lDAPDisplayName of the attribute.

One event is written per value at this debug log level.  Filter on the events and determine how many entries occur in a given period.  Review the event details in order to determine if we are writing values for multiple attributes in order to instantiate an object or if we are writing to the same attribute across multiple objects.  While this level of analysis can seem cumbersome, it can be useful in determining root cause.  As an example, if you see that we are only writing a few events per second then that could indicate that transactions are being written to the database slowly or perhaps we have too many partners that are sending redundant changes (event ID 1239).

Notice that it is perfectly normal to see event ID 1239 when replication diagnostics is set to 0x5.  If you filter out event 1239 and you see nothing else and you have a fairly long event log history then that may indicate a problem.  This issue was observed by a customer with a large Active Directory environment that had inter-site change notification enabled. If you determine that there is a large number of events per second, replication is probably not affected by a performance problem.

Object Metadata

If an event is logged that indicates a change is taking a long time to process Event ID,  record the objectGUID, and then get the following output:

Replication metadata:

Repadmin /showobjmeta * "<GUID=ObjectGUID>" >objectmeta.txt 

Review the output for recently modified attributes.  Pay particular attention to attributes with frequently modified version numbers.  An attribute that has a very high version count could indicate that frequent changes are being made to the attribute. If you suspect this, you can either view the attribute value to get some context as to why the attribute was changed, or you can let some time pass, and then get addition repadmin /showobjmeta output in order to check whether the version of the same attribute on the same object has increased further.

Object and attribute data:

Use a utility to output the object and attribute values.  Then, review the attribute data for the attributes that have recently modified data. The following examples present two methods to do this.

Object attribute values from repadmin:
 

Repadmin /showattr DCName "<GUID=ObjectGUID>" /allvalues /long >objectByGuid.txt 

Object attribute values from LDP

Connect and bind to the server in LDP and copy all the output for the object to a text file

ldpoutput.txt

 Network related data

Tasklist /svc >nets.txt Netstat –anob >>nets.txt 

Data AnalysisKey AD Replication specific Performance Counters
 

• DRA Inbound Full Sync Objects Remaining
• DRA Inbound Objects Applied/Sec
• DRA Inbound Objects / Second (Inbound Replication)
• DRA Inbound Objects Filtered / Sec (Suggests all New Attributes)
• DRA Outbound Bytes Total Since Boot

Replication Queue:
 Use this counter to determine the replication queue.  Repadmin /queue DCName also reports this information.


Gauging Current Performance:

You can use the two counters to monitor how quickly changes are being applied to the database.
 

Database:

Server Performance: 

Network:

Testing

Scenario
Two DCs were isolated (no client or other server activity)
15,000 users were created from script with the minimal attributes populated on one DC
Enabled the connection between the two DCs.

To give an idea of rate of events logged with this level of verbosity:

A Directory Services event log configured to 100 MB in size wrapped in less than two minutes (1 minute 27 seconds).  The log contained 195,728 events.  Of all events, 189,340 were event ID 1412 (attribute addition). The number of event 1412s per second:

01: 2400
02: 3570
03: 3540
04: 2160
05: 1170
06: 1890
07: 2225
08: 1435
09: 4233
10: 2817

Event ID 1365 (Object creation) was logged 6,312 times in 1 minute 27 seconds.

In one minute, 4,630 user objects were created, consisting of 138,900 attributes) or about 77 objects per second.

An understanding of NTDS performance counters is needed in order to troubleshoot this issue effectively. 

Object creations per second is obtained via the following performance counters:

• NTDS / DRA Inbound Objects Applied/sec
• Database adds/sec
• NTDS / DRA Inbound Values (DNs only)/sec
  This number includes objects that reference other objects. Values for distinguished names, such as group or distribution list memberships, are more expensive to apply than other kinds of values because a group or distribution list object can include hundreds or thousands of members. In contrast, a simple object might have only one or two attributes. A high number from this counter might explain why inbound changes are slow to be applied to the database.

Attribute creations per second is:
 

• NTDS / DRA Inbound Properties Applied/sec

Special Condition Frequently Encountered

Repadmin /rehost results in status 8461:

This issue occurs when the GC being rehosted is busy accepting updates for other partitions. The sourcing of writable domain partitions including schema, configuration and domain partition are by nature higher priority work-items than the rehosting of a read-only domain partition.

Repadmin /queue output should show that the request to add the partition has been queued and will eventually be processed.  However, sometimes it is necessary to use an alternative method of partition rehost:
 

1. Repadmin /unhost
2. Wait for event ID 1660
3. Disable KCC connection translation
4. Repadmin /add

If the process is preempted before /add is complete, you can disable inbound replication and use repadmin /replicate and the /readonly and /force options to get the partition re-hosted before you re-enable inbound replication.