Active Directory replication delayed when indexed attributes rebuilt during schema upgrade

Selected attributes in Active Directory databases are indexed to enhance performance for LDAP searches and internal operations in the operating system. A schema change that indexes existing attributes in a Windows 2000 forest or adds new indexed attributes may delay Active Directory replication until the indexing process has completed.

This replication delay applies to any schema change that adds a large number of indexed attributes to a Windows 2000 forest that does not have the hotfix that is mentioned in the following article in the Microsoft Knowledge Base: This also includes the addition of:

• Windows Server 2003-based computers to a Windows 2000 forest using Adprep.exe.
• Microsoft Exchange 2000.
• Third-party applications that add indexed attributes.

This article uses the addition of indexed attributes by the Windows Server ADPREP utility to illustrate the events encountered when indexed attributes are added to a Windows 2000 forest.

The Adprep.exe utility, which is located on the Windows Server compact disc (CD), prepares a Windows 2000 forest and its domains for the addition of Windows Server domain controllers.

One of the operations performed by Adprep is a schema change that adds 25 indexed attributes to the Active Directory database.
In terms of time-to-task, it costs just as much to index a rare attribute, such as invocation-id, as a common one so the number of entries in the database that pertain to the index is not relevant. The database engine does not know that a column will only be filled in on certain easy-to-identify records, so it must walk all records in the table and examine each of them.

Creating an index in Active Directory is an input/output (I/O) intensive operation, rather than being CPU-intensive because the database engine, ESE, must read every record in the database. Domain controllers in the process of building indexes will exhibit very high I/O rates even if the computer does not otherwise appear busy.

The building of indexes, schema cache update, and finally the replication of Active Directory occurs synchronously. Adding indexed attributes to the Window forests with a large number of objects delays Active Directory replication until indexes and the schema cache are updated. The duration of the delay depends on the number of objects pertaining to the index being added, performance of the disk subsystem and existing load on the domain controller.

Performance Metrics for a Windows Server schema upgrade

A test forest consisting of an empty root and 3 child domains was built to test "dry-run" schema upgrades. Hardware consisted of four processor 500 Mhz Compaq Proliant 5500's with 2 GB of RAM. Ntds.dit files were located on three disk Raid 5 arrays connected to Compaq Proliant 4200 controllers. Domain Controllers were promoted in production domains, and then placed on a private network. The Windows Server schema was added to the Windows 2000 forest by using the adprep /forestprep and adprep /domainprep switches for the root domain. Adprep /domainprep was run on each domain controller in three child domains. The Ntds.dit file on Global Catalogs in the forest was approximately 15 GB in size.

Note: Active Directory database size can be measured accurately only when the domain controller is booted in Dsrepair mode.

As a result of rebuilding indexes, Active Directory replication did not occur for some six to eight hours after the schema update. During the upgrade, no client authentication or application load was present so domain controllers were free to dedicate all resources to the index operation. Upgrades in production environments with slow links, replication latency or slower disk subsystems could take significantly longer.

The following formula can be used to roughly estimate the replication delay once the schema change has been replicated by a given domain controller: On computers with slower disk subsystems, consider this change in the formula: Applying the first formula to determine the replication delay when adding the Windows Server schema, which adds 25 indexed attributes to a Windows 2000 forest with a 15 GB database, you get the following estimate: Post SP2 and Windows Server Enhancements

A post SP2 hotfix for Windows 2000, Q307219, and built-into server editions of Windows Server, allows the indexing of newly added attributes to finish immediately so that Active Directory replication is not blocked.
Customers seeking to avoid replication delays when schema changes containing indexed attributes are added to an applicable Windows 2000 forest (this or inclusive hotfix not already installed on domain controllers containing a large number of Active Directory objects (> 1GB) in a forest where large number of indexed attributes are being added) should deploy this hotfix (or an inclusive fix or service pack) to all domain controllers in the forest prior to adding the schema change.

A more limited deployment strategy is to install the fix only on replication bridgeheads in the forest. This method prevents inter-site replication from being stalled twice: Once while the left-hand bridgehead rebuilds, and then again while the right-hand one does the same.

Install this fix if:

1. Dry-runs of schema upgrade or obviously large database indicate your environment is susceptible to replication delays
2. Addition of indexed attributes to the forest cannot wait until the weekend.
3. Business operations require replication to be functional on a 24x7 basis.
4. The pain of installing the hotfix and rebooting all the domain controllers is less than the pain of the replication delay.

Events logged during a Windows Server schema upgrade

The following examples illustrate benign events logged when the Windows Server schema changes were added to a Windows 2000 forest by using Adprep.exe from a server edition of Windows Server. The forest used for this test consisted of a single Windows 2000 SP2 domain controller in an empty root domain and two Windows 2000 SP2 domain controllers in a single child domain. This is just an example of a forest used to reproduce the problem. The actual problem is related to the sequence and complexity of schema changes.

The domain controller in the root domain logged the following: Active Directory replication with domain controllers in the child domain failed with the error: Domain controllers in the child domain logged the following error: Active Directory replication from the domain controller of the root domain to the domain controller in the child domain failed with the following error: The two domain controllers in the child domain logged the following replication events when trying to replicate with each other: For more information, click the following article number to view the article in the Microsoft Knowledge Base: