The event ID 2104 is logged in the DFS Replication log on a downstream server when the DFS Replication service stops

Article ID: 979295 - View products that this article applies to.
Expand all | Collapse all

On This Page

SYMPTOMS

Consider the following scenario:
  • You build a Distributed File System (DFS) Replication network environment. The servers in this network are running Windows Server 2003 R2 or Windows Server 2008.
  • You have a downstream server that functions as a hub server. This server replicates the data from the upstream servers.
In this scenario, the DFS Replication service may stop on the downstream server. Additionally, the event 2104 is logged in the DFS Replication log on the downstream server:
Log Name: DFS Replication
Source: DFSR
Event ID: 2104
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: <computer name>
Description:
The DFS Replication service failed to recover from an internal database error on volume <drive letter>:. Replication has been stopped for all replicated folders on this volume.

Additional Information:
Error: 9214 (Internal database error (-1605))
Volume: <GUID>
Database: <drive letter>:\System Volume Information\DFSR
Note Error code "-1605" is the "JET_errKeyDuplicate" Jet error.

At the same time, the event 2212 is logged in the DFS Replication log on an upstream server:
Log Name: DFS Replication
Source: DFSR
Event ID: 2212
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: <computer name>
Description:
The DFS Replication service has detected an unexpected shutdown on volume <drive letter>:. This can occur if the service terminated abnormally (due to a power loss, for example) or an error occurred on the volume. The service has automatically initiated a recovery process. The service will rebuild the database if it determines it cannot reliably recover. No user action is required.

Additional Information:
Volume: <drive letter>
GUID: <GUID>
After a while, the DFS Replication service restarts automatically. However, the same problem may occur again.
Back to the top | Give Feedback

CAUSE

The DFS Replication service error occurs because the DFS Replication database recovers from a dirty shutdown state on the upstream server that logged an instance of the event ID 2122. However, a database inconsistency on the downstream server is introduced.

The upstream server populates a global version sequence number for an update to its downstream server before the upstream server commits the global version sequence number to its database. Then, the upstream server crashes or it encounters a power loss or a hard disk failure before it commits the latest global version sequence number to its database.

After the upstream server resumes, it reuses the global version sequence number that is already published for another update. The downstream server detects an inconsistency because there are various updates that have the same global version sequence number.

However, the downstream server cannot resolve the inconsistency because two ID_RECORDs in the same content set cannot have the same global version sequence number. The attempt to resolve the issue fails and generates a "-1605" error code that represents the "JET_errKeyDuplicate" error.

After the hotfix is installed, DFS Replication calls another Jet database function. This function makes sure that the database transaction is committed to the disk of the upstream server before the upstream server populates a global version sequence number for an update to the downstream server.
Back to the top | Give Feedback

RESOLUTION

To resolve this issue, follow these steps:
  1. To resolve the database inconsistency issue on the downstream server, perform the steps in the "Workaround" section.
  2. To prevent this issue from occurring again, install the appropriate hotfix on the upstream server:
    • On an upstream server that is running Windows Server 2003 R2, install this hotfix.
    • On an upstream server that is running Windows Server 2008, install the hotfix from the following Microsoft Knowledge Base article:
      979247
      (http://support.microsoft.com/kb/979247/ )
      The DFS Replication service crashes on a Windows Server 2008-based downstream server that is configured to replicate data from many upstream servers

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.

If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft Web site:
http://support.microsoft.com/contactus/?ws=support
(http://support.microsoft.com/contactus/?ws=support)
Note The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.

Prerequisites

Apply this hotfix on the upstream server that is running Windows Server 2003 R2 Service Pack 2 (SP2).

For more information about how to obtain a Windows Server 2003 service pack, click the following article number to view the article in the Microsoft Knowledge Base:
889100
(http://support.microsoft.com/kb/889100/ )
How to obtain the latest service pack for Windows Server 2003

Registry information

To use the hotfix in this package, you do not have to make any changes to the registry.

Restart requirement

You must restart the computer after you install this hotfix.

Hotfix replacement information

This hotfix does not replace a previously released hotfix.

File information

The English (United States) version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.
Windows Server 2003 R2 file information note
  • In addition to the files that are listed in these tables, this hotfix also installs an associated security catalog file (KBnumber.cat) that is signed with a Microsoft digital signature.
For all supported x86-based versions of Windows Server 2003 R2 SP2
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatformSP requirement
W03a3409.dll5.2.3790.459043,52016-Sep-200912:16x86SP2
Dfsr.exe5.2.3790.46511,705,98418-Jan-201020:30x86SP2
For all supported x64-based versions of Windows Server 2003 R2 SP2
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatformSP requirementService branch
Dfsr.exe5.2.3790.46513,595,26420-Jan-201001:57x64SP2Not applicable
W03a3409.dll5.2.3790.459044,03220-Jan-201001:57x64SP2Not applicable
Ww03a3409.dll5.2.3790.459043,52020-Jan-201001:57x86SP2WOW
For all supported IA-64-based versions of Windows Server 2003 R2 SP2
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatformSP requirementService branch
Dfsr.exe5.2.3790.46515,501,44020-Jan-201001:53IA-64SP2Not applicable
W03a3409.dll5.2.3790.459042,49620-Jan-201001:53IA-64SP2Not applicable
Ww03a3409.dll5.2.3790.459043,52020-Jan-201001:53x86SP2WOW
Back to the top | Give Feedback

WORKAROUND

Workaround for the DFS Replication error

To work around this issue, manually perform a tombstone operation on the old ID_RECORD record and creates a new global version sequence number. To do this, follow these steps:
  1. Determine the originating DFSR server and the file path. This consists of the file GVSN's GUID and the RG Name, where the CSID's match.

    So in the above case, for example:

    20091213 01:55:56.862 8820 RDCX 808 Rdc::SeedFile::Initialize The hashes coincide. The download is redundant, uid:{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357 gvsn:{58A90409-3866-4DC1-B474-BE1C8DE41B84}-v22461 fileName:4-5269.dot csId:{B890770A-6043-4004-A149-41E3A1935AC1

    20091213 01:55:57.705 8820 DOWN 5186 [ERROR] DownstreamTransport::RdcGet Failed on connId:{7D927665-24BF-4F16-B503-708AABDBC9AE} csId:{B890770A-6043-4004-A149-41E3A1935AC1} rgName:SOME_RG update:

    Run:

    DFSRDIAG.EXE GUID2NAME /rgname:"SOME_RG" /guid:{58A90409-3866-4DC1-B474-BE1C8DE41B84}
  2. Now that you know the server, you can go to that computer.
    • If the upstream server is running Windows Server 2008 R2, run the following command:
      DFSRDIAG IDRECORD
    • If the upstream server is running Windows Server 2003 or Windows Server 2008, run the following command:
      WMIC.EXE /namespace:\\root\microsoftdfs path dfsridrecordinfo.Uid="<UID of the duplicated GVSN on the upstream server side>" call getfullfilepath
  3. Locate the full path of file having a problem.
    1. If using Windows Server 2008 R2, use DFSRDIAG IDRECORD. For example:

      DFSRDIAG.EXE IDRECORD /UID:{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357
    2. If using Win2003 or Win2008, use WMIC. For example:

      WMIC.EXE /namespace:\\root\microsoftdfs path dfsridrecordinfo.Uid="{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357" call getfullfilepath
  4. Make a copy of the file (in the example above "4-5269.dot") in the same folder and call it some random name like "fixed.ned".
  5. Delete the old file (in the example above "4-5269.dot").
  6. Rename the copied file back to the original name (so from "fixed.ned" back to "4-5269.dot").
  7. This will tombstone the old ID_RECORD and create a new GVSN, and all will be well.

Workaround example

For example, assume that you receive the following debug log on the affected downstream server: 
20091213 01:55:56.862 8820 RDCX   808 Rdc::SeedFile::Initialize The hashes coincide. The download is redundant, uid:{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357 gvsn:{58A90409-3866-4DC1-B474-BE1C8DE41B84}-v22461 fileName:4-5269.dot csId:{B890770A-6043-4004-A149-41E3A1935AC120091213 01:55:57.705 8820 DOWN  5186 [ERROR] DownstreamTransport::RdcGet Failed on connId:{7D927665-24BF-4F16-B503-708AABDBC9AE} csId:{B890770A-6043-4004-A149-41E3A1935AC1} rgName:SOME_RG update:
In this scenario, the "{58A90409-3866-4DC1-B474-BE1C8DE41B84}-v22461" GUID and the "SOME_RG" RgName have the same "{B890770A-6043-4004-A149-41E3A1935AC1}" CSID. Therefore, follow these steps to work around the delay issue in this scenario:
  1. Run the following command to obtain the problematic upstream server name:
    DFSRDIAG.EXE GUID2NAME /rgname:"SOME_RG" /guid:{58A90409-3866-4DC1-B474-BE1C8DE41B84}
  2. On the upstream server, find the GUID of the duplicated global version sequence number in its debug log. Then, run one of the following commands to obtain the full path of the file that has the duplicated global version sequence number:
    • For Windows Server 2008 R2, use the following command:
      DFSRDIAG.EXE IDRECORD /UID:{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357
    • For Windows Server 2003 R2 or for Windows Server 2008, use the following command:WMIC.EXE /namespace:\\root\microsoftdfs path dfsridrecordinfo.Uid="{8BB53DB2-9446-47D0-94E0-F93A56589D18}-v2357" call getfullfilepath
  3. Assume that you obtain the "4-5269.dot" file name. Copy the "4-5269.dot" file, and then rename the copied file to the "fix.dot" file name. Make sure that you put the "4-5269.dot" and "fix.dot" files in the same folder.
  4. Delete the "4-5269.dot" file.
  5. Rename the "fix.dot" file to the "4-5269.dot" file name.
By following these steps, you can work around the DFS Replication database issue in the example.
Back to the top | Give Feedback

STATUS

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
Back to the top | Give Feedback

MORE INFORMATION

For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684
(http://support.microsoft.com/kb/824684/ )
Description of the standard terminology that is used to describe Microsoft software updates
Back to the top | Give Feedback

Properties

Article ID: 979295 - Last Review: March 24, 2010 - Revision: 3.1
APPLIES TO
  • Microsoft Windows Server 2003 Service Pack 2
  • Microsoft Windows Server 2003 R2 Datacenter Edition (32-Bit x86)
  • Microsoft Windows Server 2003 R2 Datacenter x64 Edition
  • Microsoft Windows Server 2003 R2 Enterprise Edition (32-Bit x86)
  • Microsoft Windows Server 2003 R2 Enterprise x64 Edition
  • Microsoft Windows Server 2003 R2 Standard Edition (32-bit x86)
  • Microsoft Windows Server 2003 R2 Standard x64 Edition
Keywords: 
kbexpertiseadvanced kbfix kbhotfixserver kbqfe kbautohotfix kbsurveynew kbprb KB979295
Back to the top | Give Feedback

Give Feedback

 
Back to the topBack to the top