Troubleshooting AD Replication error 8477: The replication request has been posted; waiting for reply


Notice


Home users: This article is only intended for technical support agents and IT professionals. If you're looking for help with a problem, please ask the Microsoft Community.

Symptoms


This article describes the symptoms, cause and resolution steps involved in troubleshooting Active Directory Replication error 8477: "The replication request has been posted; waiting for reply".

The symptoms discussed in this article are commonly related to the occurrence of event 8477, however may also be observed with other events related to slow or delayed replication. When troubleshooting such issues, consideration should be given to all factors that may cause replication delays and be remediated accordingly.

1.  Output from repadmin.exe /showreps /verbose may report the replication attempt has failed with error 8477 – "The replication request has been posted; waiting for reply"


DC=Contoso,DC=com
    Default-First-Site-Name\DomainController via RPC
        DSA object GUID: <source DCs ntds settings object object guid>
        Address: <source DCs ntds settings object object guid>._msdcs.Contoso.Com
        DSA invocationID: <source DCs NTDS DB invocation id>
        DO_SCHEDULED_SYNCS WRITEABLE COMPRESS_CHANGES NO_CHANGE_NOTIFICATIONS PREEMPTED
        USNs: 1158544/OU, 111052/PU
        Last attempt @ <Date Time> was delayed for a normal reason, result 8477 (0x211d):
    The replication request has been posted; waiting for reply.
        Last success @ <Date Time>

 
2.  Output from repadmin.exe /queue may report a large number of queued inbound replication requests and an unprecedentedly long execution time

Repadmin /queue
repadmin running command /queue against server localhost
 
Queue contains 26 items.
Current task began executing at <Date Time>.
Task has been executing for 86 minutes, 12 seconds.
 
[6485] Enqueued <Date Time> at priority 590
    SYNC FROM SOURCE
    NC DC=Contoso,DC=com
    DC Default-First-Site-Name\DomainController
    DC object GUID <source DCs ntds settings object object guid>
    DC transport addr <source DCs ntds settings object object guid>._msdcs.Contoso.com
  ASYNCHRONOUS_OPERATION WRITEABLE PERIODIC NEVER_NOTIFY PREEMPTED

 Note: The occurrence of event 8477 when inbound replication requests are queuing is generally observed following a pre-empted replication task. This event is indicated by event 8461 - The replication operation was pre-empted.


3.  When using dcdiag.exe, initialization of a recently promoted Domain Controller may be delayed, awaiting the completion of an initial synchronization


Directory Server Diagnosis
 
Performing initial setup:
   Trying to find home server...
   Home Server = DomainController
The directory service on DomainController has not finished initializing.
In order for the directory service to consider itself synchronized, it must
attempt an initial synchronization with at least one replica of this
server's writeable domain. It must also obtain Rid information from the Rid
FSMO holder.
The directory service has not signalled the event which lets other services
know that it is ready to accept requests. Services such as the Key
Distribution Center, Intersite Messaging Service, and NetLogon will not
consider this system as an eligible domain controller.
   * Identified AD Forest.
   Done gathering initial info.
The directory service on BOULDERDC01 has not finished initializing.
In order for the directory service to consider itself synchronized, it must
attempt an initial synchronization with at least one replica of this
server's writeable domain. It must also obtain Rid information from the Rid
FSMO holder.
The directory service has not signalled the event which lets other services
know that it is ready to accept requests. Services such as the Key
Distribution Center, Intersite Messaging Service, and NetLogon will not
consider this system as an eligible domain controller.
Done gathering initial info.

Note: The output from dcdiag.exe described above is normal behaviour during the promotion of a new, replica domain controller into an environment. The occurrence of 8477 in this case should be given time to clear through normal replication cycles prior to remedial activities being considered or conducted.

4.  When using dcdiag.exe, the 'Replications' test case may issue a 'REPLICATION LATENCY WARNING'


Starting test: Replications
   REPLICATION LATENCY WARNING
   DomainController: A long-running replication operation is in progress
      The job has been executing for 84 minutes and 22 seconds.
      Replication of new changes along this path will be delayed.
      This is normal for a new connection, or for a system
      that has been down a long time.
      Enqueued <Date Time> at priority 90
      Op:             SYNC FROM SOURCE
      NC DC=Contoso,DC=com
      DSADN <source DCs ntds settings object object guid>
 
      DSA transport addr <source DCs ntds settings object object guid>._msdcs.Contoso.com
   ......................... DomainController passed test Replications


5.  'NTDS Replication' event 1580 may be logged in the directory service event log:

 

Event Source

Event ID

Event String

NTDS Replication

1580

A long-running Active Directory Domain Services inbound replication task has finished with the following parameters.

Elapsed time (minutes):

84

Operation:

Synchronize Replica

Options:

0x21000051

Parameter 1:

DC=Contoso,DC=com

Parameter 2:

<source DCs ntds settings object object guid>

Parameter 3:

 

Parameter 4:

 

A long-running replication task may also occur when a system has been unavailable or a directory partition has been unavailable for an extended period of time. A long running replication task may indicate a large number of updates, or a number of complex updates occurring at the source directory service. Performing these updates during non-critical times may prevent replication delays.

 

A long running replication task is normal in the case of adding a new directory partition to Active Directory Domain Services. This can occur because of a new installation, global catalog promotion, or a connection generated by the Knowledge Consistency Checker (KCC).

 

Additional Data

Error value:

The operation completed successfully.

Cause


The 8477 (The replication request has been posted; waiting for reply) status is informational and represents normal Active Directory replication operation, indicating that replication is currently in progress from the source and has not yet been applied to the destination Domain Controllers database replica.


The presence of this event for an extended period may, however, may be indicative of issues with Active Directory Replication on the Destination Domain Controller. Possible causes may include:

  1. Low Available Physical Memory, High Latency or Low Bandwidth Network Links, Excessive CPU Utilization or Disk I/O for the amount of data that the domain controller is replicating
  2. High rate of change for objects in Active Directory Domain Services (AD DS)
  3. Third-Party utilities that may hamper or delay normal replication function
  4. Large Groups (Greater than 5000 Members) where Linked-value Replication is not enabled
  5. A recent modification to the Active Directory Schema where a sizeable number of schema attributes have been modified or indexed

Resolution


Investigation of replication error 8477 should be conducted, at least initially, on the Destination Domain Controller within the replication partnership. The resolution section of this article will discuss techniques an Active Directory Administrator should utilize to resolve the possible causes for error 8477 as per the "Cause" section.

1.  Low Available Physical Memory, High Latency or Low Bandwidth Network Links, Excessive CPU Utilization or Disk I/O for the amount of data that the domain controller is replicating
 
Active Directory Domain Controllers should be configured with as few additional roles and applications as possible, ideally a Domain Controller should be a single purpose server utilized only to service queries and authentication requests. When troubleshooting Active Directory performance issues, it is often best to first analyse the system for any additional applications, services or tasks that are unnecessary or redundant.
 
As an initial step, an IT Professional should check Task Manager (and Resource Monitor where appropriate) for overall CPU and Memory utilization to determine if it is uncharacteristically high or deviates from a predefined baseline. If no baseline exists, Microsoft Best Practice suggests that sustained and repeated CPU utilization greater than 80% would be considered high and should be investigated further.

http://technet.microsoft.com/en-us/library/bb727054.aspx - Troubleshooting High CPU Usage on a Domain Controller

 
In regards to disk utilization, on Active Directory Domain Controllers the Ntds.dit and edb.log files should be the most active files on the system. To determine whether this is the case, performance logging and analysis (as per the below) should be conducted.
 
Low Bandwidth, High Latency or Busy Network Connectivity may also cause error 8477 to occur on Active Directory Domain Controllers. Network Performance data metrics should be collected from a Domain Controller utilizing Performance Monitor, Network Monitor or other such tools. Guidance around monitoring and measuring replication traffic is detailed in http://technet.microsoft.com/en-us/library/bb742457.aspx.
 
For Windows Server 2003 based Domain Controllers:
An IT Professional may choose to use Performance Monitor with Processor, Physical Disk, Network Interface and Directory Services counters added to determine and investigate performance issues on the system. Alternatively, Server Performance Advisor (http://www.microsoft.com/en-us/download/details.aspx?id=15506) may reduce the complexity in analysing performance metrics obtained from Performance Monitor. The tool is a free download and will look at CPU, Network, Memory and Disk I/O giving detail around overall utilization and a determination as to whether the performance data is seen as idle, normal or potentially problematic.
 
 
For Windows Server 2008 and above based Domain Controllers:
Performance Monitor in Windows Server 2008 and beyond includes the key functionality of Server Performance Advisor straight out of the box. Within the System Data Collector Sets, the Active Directory Diagnostics set will, similarly to Server Performance Advisor, produce a report with key metrics for Active Directory Performance investigation and provide warnings for uncharacteristic behaviour on Active Directory Domain Controllers. Any warnings are included at the top of the report an example of which is below: 

 

Warning

Symptom:

The system is experiencing excessive paging

Cause:

Available memory on the system is low.

Details:

The total physical memory on the system is not capable of handling the load.

Resolution:

Upgrade the physical memory or reduce system load

Related:

Memory Diagnosis

Symptom:

On directory servers, Ntds.dit and Edb.log should be the most active files. In this case, C:\Windows\NTDS\ntds.dit has the highest I/O rate (84.622 operations/sec).

 
2.  High rate of change for objects in Active Directory Domain Services (AD DS)
 
In a very busy Active Directory Environment, a large number of changes may occur to objects between each scheduled replication. If this is expected within the environment, all aspects of the organizations infrastructure should be sized appropriately to facilitate effective replication.
 
In the event that an IT Administrator is not aware or not expecting a large number of changes and is receiving error 8477, investigation should be conducted to determine what changes are occurring in the environment and whether these are expected or the result of an issue with an application or service. An effective means for performing this task is to determine what objects are changing through the use of the uSNChanged attribute, the highest uSNChanged value on a specific Domain Controller represents the High-Watermark USN.
 
Running repadmin /showreps /verbose against a domain controller with event 8477 being displayed will show the current High-Watermark and is followed by /OU:


 
DC=Contoso,DC=com
    Default-First-Site-Name\DomainController via RPC
        DSA object GUID: <source DCs ntds settings object object guid>
        Address: <source DCs ntds settings object object guid>._msdcs.Contoso.Com
        DSA invocationID: <source DCs NTDS DB invocation id>
        DO_SCHEDULED_SYNCS WRITEABLE COMPRESS_CHANGES NO_CHANGE_NOTIFICATIONS PREEMPTED
        USNs: 1158544/OU, 111052/PU
        Last attempt @ <Date Time> was delayed for a normal reason, result 8477 (0x211d):
    The replication request has been posted; waiting for reply.
        Last success @ <Date Time>


 
Depending on the number of changes occurring, the highest USN present on a Domain Controller may increment quite quickly, in order to determine the objects that are being updated, it is suggested that on the Source Replication Partner, the below command is run for the relevant Naming Context:

 
Repadmin /showattr DomainControllerName DC=Contoso,DC=com /subtree /filter:"(uSNChanged>=highestcommitedusn)" /atts:objectclass

 
Specific Applications or Services, such as the Exchange Recipient Update Service may make regular changes to Active Directory Attributes and may need to be tuned if resulting replication traffic is unmanageable.

http://support.microsoft.com/kb/914036 - Windows Server 2003-based Domain Controller's show a decrease in performance when they process certain Active Directory objects.

3.  Third-Party utilities that may hamper or delay normal replication function
 
Third Party applications should be configured and tuned around performance and supportability best practices. Certain applications, if not configured around best practices for Active Directory Performance, may cause impact to normal Active Directory Domain Controller function.
 
Applications of note include (but are not limited to):

a. Virus Scanning Software
b. Monitoring Agents
c. Identity Synchronization and Management Applications
d. Backup Software and Agents

http://support.microsoft.com/?id=822158 - Virus scanning recommendations for Enterprise computers that are running currently supported versions of Windows

4.  Large Groups ( Greater than 5000 Members) where Linked-value Replication is not enabled
 
Linked-value replication is not available in Windows 2000 Server forests. Because an originating update must be written in a single database transaction, and because the practical limit for a single transaction is 5,000 values, membership of more than 5,000 values is not supported in Windows 2000 Server Active Directory. A group of this size represents a limitation both in terms of the database write operation that is required to record a change to an attribute of that size and the transfer of that much data over the network. This is different in forests at Windows 2003 or higher functional level where linked-valued replication (LVR) for group memberships is enabled.
 
When a Domain is upgraded from a 2000 functional level, the memberships of any groups carried are considered legacy and can still cause replication issues:

i.  Forests prior to the 2003 functional level should divide the groups into smaller groups that do not exceed 5,000 members. Group nesting can also be used, providing the applications and services using the groups can support it 
 
ii.  Forests at the 2003 functional level can remove and reinstate group members to make them LVR-enabled. Over time, as security principals are added and removed from groups, the members are slowly enabled for LVR

Note: Events 1479 and 1519 are also commonly logged in the Directory Service Event Log when large groups are causing replication issues and delays.

Using repadmin /showobjmeta legacy members in a group can be determined and converted to LVR enabled members if required to resolve the issue, these users are denoted with 'Type' of value LEGACY:

repadmin /showobjmeta DomainControllerName "CN=Administrators,CN=Builtin,DC=Contoso,DC=Com"
 
3 entries.
Type    Attribute     Last Mod Time                            Originating DSA  Loc.USN Org.USN Ver
======= ============  =============                           ================= ======= ======= ===
        Distinguished Name
        =============================
PRESENT       member <Date Time>                    Default-First-Site-Name\DomainController    8203    8203   1
        CN=Administrator,CN=Users,DC=Contoso,DC=Com
PRESENT       member <Date Time>                    Default-First-Site-Name\DomainController   12398   12398   1
        CN=Enterprise Admins,CN=Users,DC=Contoso,DC=Com
PRESENT       member <Date Time>                    Default-First-Site-Name\DomainController   12378   12378   1
        CN=Domain Admins,CN=Users,DC=Contoso,DC=Com
LEGACY       member <Date Time>                    Default-First-Site-Name\DomainController   198458   198458   1
        CN=mjordan,CN=Users,DC=Contoso,DC=Com

 

5.  A recent modification to the Active Directory Schema where a sizeable number of schema attributes have been modified or indexed
 
Attribute definitions are stored in attributeSchema objects in the schema directory partition. Changes to attributeSchema objects block other replication until the schema changes are performed. During replication of any directory partition other than the schema directory partition, the replication system first checks to see whether the schema versions of the source and the destination domain controllers are in agreement. If the versions are not the same, the replication of the other directory partition is rescheduled until the schema directory partition is synchronized.
 
Modification of the schema through the use of LDIFDE or customized binaries included with applications (ie Microsoft Exchange, Operations Manager etc) may add indexed attributes to the schema directory partition. Depending on the size of the update, replication delays can be caused in large databases.
 
In cases such as these, the 8477 state may appear as a transient issue and should be expected to self-resolve once the higher priority schema updates are successfully replicated and all other replication catches up on backlogged changes within the directory.