Symptoms
On a domain controller (DC) that is running Windows Server 2003, Windows Server 2008, or Windows Server 2008 R2, you may experience the following symptoms:
-
The server runs slower than expected every several hours.
-
The server stops responding every several hours.
This issue lasts for several minutes and then disappears.
Note If you use Performance Monitor (Perfmon.exe) to monitor the domain controller, you find that the CPU usage and the disk I/O are very high when this issue occurs.
Note If you enable the garbage collection event logging according to KB314980, you find that the garbage collection process runs when this issue occurs. However, tombstone objects are not deleted.
When this issue occurs, one or more of the following events are added in the Directory Service log:
Event Type: Error
Event Source: NTDS ISAM
Event Category: (14)
Event ID: 623
Date: <date>
Time: <time>
User: N/A
Computer: <computer name>
Description: NTDS (432) NTDSA: The version store for this instance (0) has reached its maximum size of <number>. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.
Possible long-running transaction:
SessionId: <id>
Session-context: <context> Session-context ThreadId: <id> Cleanup: 1
Event Type: Error
Event Source: NTDS General
Event Category: Internal Processing
Event ID: 1519
Date: <date>
Time: <time>
User: NT AUTHORITY\LOCAL SERVICE
Computer: <computer name>
Description: Internal Error: Active Directory could not perform an operation because the database has run out of version storage.
Additional Data Internal ID: 2080490
Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1479
Description: Active Directory could not update the following object on the local domain controller with changes received from the following source domain controller. Active Directory does not have enough database version store to apply the changes.
Object: <object DN>
Object GUID: <object GUID>
Source domain controller: <DC GUID-based DNS name>
User Action
Restart this domain controller. If this does not solve the problem, increase the size of the database version store. If you are populating the objects with a large number of values, or the size of the values is especially large, decrease the size of future changes.
Additional Data
Error value: 8573 The database is out of version store.
Cause
This issue occurs because there are many inconsistent objects or reference phantoms in the Active Directory. The DELTIME attribute of these objects is set. However, the isDeleted attribute of these objects is NULL. Because of the DELTIME attribute, garbagecollector includes these objects in its database scan. Therefore, these objects cannot be deleted.Many inconsistent objects can cause the garbage collection process to use up version storage space.
This problem occurs during the garbage collection process of the tombstone objects. The tombstone cleanup code in the garbage collection process will update the DELTIME for these inconsistent objects, because it considers them reference phantoms. In this state, the tombstone cleanup code builds one database transaction for these updates and for many consecutive objects. Therefore, the size of the transaction exceeds the size of Version Store.
The Garbage Collector treats this as a fatal error and stops. Any database records that require cleanup will not be looked at, for example tombstones that are due for removal. The procedure is retried on the next run of the garbage collector. Therefore, the next attempt may also fail.
These inconsistent objects are not easy to identify, as DELTIME is an internal database column. They do not appear as such objects in the User Interface or in a database export. If you suspect you are affected by this problem but you want to be positive that applying this hotfix will help you, we recommend that you contact Microsoft Customer Technical Support to investigate the state of the affected databases.
In addition, these events can be caused bytoo many reference phantoms. These database records are placeholders for one-way object links (that have no backlink) in which the target object was deleted and the tombstone removed from the database. However, the reference is not changed. These references have to be cleaned up. You can do this by deleting the attribute value or by changing the reference to a valid object. One example attribute in this class is the "secretary" attribute.
Werecommend that you contact Microsoft Customer Technical Support to help you identify and remove these reference phantoms.
Resolution
The hotfix for Windows Server 2003 is available now. If you encounter this issue on a computer that is running Windows Server 2008 or Windows Server 2008 R2, you can follow the steps provided in the Workaround section. It is recommended that you contact Microsoft Customer Technical Support for further assistance.
Hotfix information
A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.
If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.
Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft Web site:
http://support.microsoft.com/contactus/?ws=supportNote The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.
Prerequisites
To apply this hotfix, you must be running Windows Server 2003 Service Pack 2 (SP2). Additionally, Active Directory must be installed.For more information about how to obtain a Windows Server 2003 service pack or a Windows XP Professional x64 Edition service pack, click the following article number to view the article in the Microsoft Knowledge Base:
889100 How to obtain the latest service pack for Windows Server 2003
Restart requirement
You do not have to restart the computer after you apply this hotfix.
Hotfix replacement information
This hotfix does not replace a previously released hotfix.
Installation instruction
After you install this hotfix, follow these steps.
-
Restart the DC into Directory Service Restore mode.
-
Log on as an Administrator.
-
Open a command prompt.
-
At a command prompt, type the following commands, and then press Enter after each command:
ntdsutil
semantic database analysis
go fixup
-
After the command is executed, you find a file "dsdit.dmp.x" in the folder from which NTDSUTIL was executed that lists all the objects that were corrected. "x" is the execution number of the semantic database analysis, please inspect the newest file.
-
Restart your computer in normal mode.
File information
The English (United States) version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.
Windows Server 2003 file information notes
-
In addition to the files that are listed in these tables, this hotfix also installs an associated security catalog file (KBnumber.cat) that is signed with a Microsoft digital signature.
For all supported x86-based versions of Windows Server 2003 SP2
|
File name |
File version |
File size |
Date |
Time |
Platform |
|---|---|---|---|---|---|
|
Ntdsutil.exe |
5.2.3790.4650 |
312,832 |
15-Jan-2010 |
12:42 |
x86 |
For all supported x64-based versions of Windows Server 2003 SP2
|
File name |
File version |
File size |
Date |
Time |
Platform |
SP requirement |
Service branch |
|---|---|---|---|---|---|---|---|
|
Ntdsutil.exe |
5.2.3790.4650 |
503,296 |
15-Jan-2010 |
15:14 |
x64 |
SP2 |
Not Applicable |
|
Wntdsutil.exe |
5.2.3790.4650 |
312,832 |
15-Jan-2010 |
15:14 |
x86 |
SP2 |
WOW |
For all supported IA-64-based versions of Windows Server 2003 SP2
|
File name |
File version |
File size |
Date |
Time |
Platform |
SP requirement |
Service branch |
|---|---|---|---|---|---|---|---|
|
Ntdsutil.exe |
5.2.3790.4650 |
807,424 |
15-Jan-2010 |
15:14 |
IA-64 |
SP2 |
Not Applicable |
|
Wntdsutil.exe |
5.2.3790.4650 |
312,832 |
15-Jan-2010 |
15:14 |
x86 |
SP2 |
WOW |
Workaround
To work around the issue, increase the size of version storage by changing the value of the following registry item:
Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Name: EDB max ver pages (increment over the minimum)
Type: REG_DWORD
Value: <value>
Note The value for the setting is the number of 16KB memory chunks (or 32KB memory chunks on a 64-bit processor) that will be reserved. For example:
9600 = ~150 MB 32-bit, ~600 MB 64-bit
12800 = ~200 MB 32-bit, ~800 MB 64-bit
16000 = ~250 MB 32-bit, ~1000 MB 64-bit
19200 = ~300 MB 32-bit, ~1200 MB 64-bit
(Note The machine's native pointer size (4 bytes on 32-bit CPUs, 8 bytes on 64-bit CPUs,) is also a factor in calculating version store size. Therefore, using a 64-bit CPU and OS effectively quadruples the default size of the version store.)
Note You cannot set an arbitrary high value, especially on an x86 operating system. You may run out of memory in the LSASS process space for other tasks.
Note Setting the value of "EDB max ver pages (increment over the minimum)" to anything less than 6400 will have no effect. 6400 is the default and the minimum value.
We recommend that you approach the required value in increments of 50MB. The value that enables garbage collection to work should then be the base value tried for additional affected DCs. If the garbage collector does not run successfully when the value is "19200" (302 MB), Werecommend that you contact Microsoft Customer Support Services for help.
Increasing the size of version storage is only a temporary method. It enables the garbage collection code to pass these inconsistent objects. However, this issue may occur again after a tombstone lifetime interval, or when the objects are in scope of the Garbage Collector scan again.
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
More Information
This issue may occur after you conduct an authoritative restore. The problem occurs with databases with 100,000 affected objects or more. You may proactively run the procedure in the resolution section to fix the affected databases, or you may use it reactively when you encounter the Version store errors in the Symptoms section.
For more information about the Active Directory database garbage collection process , click the following article number to view the article in the Microsoft Knowledge Base:
198793 The Active Directory database garbage collection process
For more information about how to enable the garbage collection logging level, click the following article number to view the article in the Microsoft Knowledge Base:
314980 How to configure Active Directory diagnostic event logging in Windows Server 2003 and in Windows 2000 Server
For more information about how to enable the garbage collection logging level, visit the following Web site:
How to enable the garbage collection logging level
For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684 Description of the standard terminology that is used to describe Microsoft software updates
