When you install a schema extension that adds attributes to the set of attributes included in the global catalog, you may see multiple NTDS Replication Event ID 1988 events and possibly also Event ID 1388 on various domain controllers.
Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1988
The local domain controller has attempted to replicate the following object from the following source domain controller. This object is not present on the local domain controller because it may have been deleted and already garbage collected.
Source domain controller:
Replication will not continue with the source domain controller until the situation has been resolved.
Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1388
This destination system received an update for object which should have been present locally, but was not. The attribute set included in the packet is not sufficient to create the object. A full copy of the object will be requested...
A few hours after the events are first seen they stop being logged. The maximum duration of events is 12 hours, which is the default interval for garbage collection.
If Strict Replication is enabled, you will also see replication errors reported during this interval. When you check whether the objects referenced are still present, you may find the objects are not found in any database.
To identify whether you are affected by this problem, please follow these steps:
- Identify the setting for the tombstoneLifetime attribute using the information towards the end of the Microsoft Knowledge Base article below. The tombstoneLifetime attribute specifies the number of days a deleted object is kept as a tombstone.
910205 Information about lingering objects in a Windows 2000-based forest or in a Windows Server 2003-based forest
- Copy the object GUID of the object cited in Event ID 1988 on the destination domain controller (text between DEL: and ,cn=Deleted), for example A32C892F-E03F-48F7-A6EB-8E35CAA52743.
- Run the following command against the object GUID of the object on the source domain controller:
repadmin /showobjectmeta <source DC> <GUID=A32C892F-E03F-48F7-A6EB-8E35CAA52743>
|| Attribute |
|| 2001-05-02 16:55:32
|| 2008-03-31 10:10:18
|| 2009-09-22 12:32:04
The time stamp you are interested in is on the same line as IsDeleted. The time stamp should be past tombstone expiry (current time minus days of tombstone lifetime) by a short period of time.
- Run repadmin /showobjectmeta against the object GUID of the object on the destination domain controller. You should receive an object not found error or similar because garbage collection has fully purged the deleted object.
Schema extensions add attributes to object classes, or change the presence settings of attributes on tombstoned objects that reside in writable and read-only directory partitions. All existing objects, both live and tombstoned objects subject to the schema update are modified in each local copy of the Active Directory database. The database is updating the tombstoned objects with the additional attributes so it has the correct set of attributes in case the object is reanimated.
Live objects are not a problem here. All domain controllers should have the same set of objects and attributes. However, for tombstoned objects that are close to removal from the database, the object population may vary amongst domain controllers. Although each copy of a tombstone has the same deletion time stamp and thus could be removed from all domain controllers at the same time, garbage collection runs at a different time on each domain controller.
Some tombstones may still exist on the source of a replication agreement and have an attribute added to the partial attribute set of the object which needs to be replicated out. However, the same object was garbage collected on the target domain controller when it is replicated. When this happens, the destination domain controller logs Event ID 1988 and possibly Event ID 1388.
Garbage collection runs every 12 hours by default and is independent of other tasks such as Active Directory replication. The error condition should remove itself as the problem objects are also garbage collected on the source domain controller the next time it runs. Subsequent attempts to replicate the naming context should succeed.
You cannot completely resolve the issue. You can only work on minimizing the impact of the varying object population. The reference on how changes are implemented is in the More Information section. Some of the different approaches include:
- Let the errors occur and reactively triggering garbage collection.
From the time of the schema update until the last domain controller has removed the overdue tombstones, ignore the errors that your monitoring system may log, based on the Event ID 1988 and Event ID 1388 events. You may reduce the frequency by manually starting garbage collection on the source domain controllers of the events that are reported.
This approach should work best if your replication monitoring is reporting errors quickly and you are replicating frequently, so for a certain source domain controller the problem would be cleared on the next replication cycle.
- Minimize divergence by running garbage collection more often.
Shorten the garbage collection interval to maybe three hours a day before the schema extension. This should reduce the number of objects that are a potential problem, and also the time until the set of objects converges. The default interval is 12 hours. Changing the interval to three hours should not be a problem for most deployments. You can make the change several days before the schema extension and create a performance baseline to see whether this has adverse impact on domain controllers.
- Synchronize garbage collection execution and extend tombstone lifetime.
One day before the schema extension, do not allow any domain controller reboots. Then use scripts to create scheduled tasks on each domain controller in the forest. The scheduled task should run on all domain controllers at the same time (UTC), during a time of day when historically little or no Active Directory management occurs.
The schedule task should:
- Trigger garbage collection
- Increase tombstone lifetime by five days
If you were to skip triggering garbage collection, domain controller reboots would result in garbage collection being unsynchronized again, and you would again have to trigger it across all domain controllers. Changing tombstone lifetime on each domain controllers prevents replication latency from causing any inconsistency.
After the schema is extended, the set of objects present on all domain controllers should be the same, as all domain controllers run garbage collection at the same time. Therefore the objects should not differ much between domain controllers.
If you still see Event ID 1988, you may choose to trigger garbage collection on the source domain controller manually as described in the first method above. When the schema extension and object cleanup is done, you can reset tombstone lifetime to its original value.
You can use any of the above approaches. They are ranked by effectiveness, but also by impact to the environment. Therefore the third method will have the least impact on Active Directory replication and yield the lowest number of events, but it also has the highest impact on forest operations. What approach you choose will also depend on how you are monitoring Active Directory replication.
For example, given a monitoring data collection interval of eight hours, and a default garbage collection interval of 12 hours, chances are the flood of events is already over until you can take steps to control it (e.g. as described in the first method). If your monitoring system is reporting problems quickly, the first method may be a good approach.
The second method works best if you have low agility in replication monitoring and you cannot easily trigger garbage collection on all domain controllers in the forest, or run other commands. Shortening the garbage collection interval is a change distributed through replication of the configuration naming context, so all domain controllers will get it.
When your goal is to have the absolute minimum of false alarms as the number of alarms seen is part of your service level agreement, the third method is likely the best option for you.
for other considerations.
Article ID: 2005074 - Last Review: August 12, 2010 - Revision: 13.0
- Microsoft Windows Server 2003 Service Pack 2
- Windows Server 2008 Standard
- Windows Server 2008 Enterprise
- Windows Server 2008 R2 Standard
- Windows Server 2008 R2 Enterprise