FRS: Disk defragmentation causes excessive FRS replication traffic

View products that this article applies to.

This article was previously published under Q282791

SYMPTOMS

When you use disk defragmentation utilities to optimize logical drives that host File Replication service (FRS) replicated content, the following symptoms may occur:

Files in System Volume (SYSVOL) and Distributed File System (DFS) shares are replicated excessively and there is no apparent change to the files.
Files may replicate at offpeak hours but at regularly occurring times if you schedule disk defragmentation to run during specific times or periods of low server usage.
The number of files in the staging folder constantly grows, and then empties after the disk defragmentation utility is completed, or the FRS schedule opens to allow replication.
The number of files in the staging folder constantly grows but never empties if changes to downstream partners cannot be replicated either because of network connectivity or some other error condition.
Network traffic between replication partners is consuming excessive network bandwidth as a result of FRS.

Back to the top

In Windows 2000, the file system APIs that support defragmentation use the file cache to do a mapped file write of the file data to the new set of assigned disk clusters. Because a mapped file may not be associated with any particular file handle, the NTFS journal code forces the SourceInfo field in the USN record to zero. This behavior occurs because defragmentation is an on-line operation (the volume is not locked) and the file cache may actually contain modified data for a given byte range of the file if a user was writing to the file while the defragmentation process was underway. However, NTFS cannot tell that this operation has occurred. FSCTL_MARK_HANDLE becomes a no-op. When you stop FRS while Diskkeeper is being run, NTFS still writes the journal records and FRS reads them when it is later restarted.

In Windows Server 2003, NTFS no longer passes the data through the file cache and no USN records are generated.

Back to the top

RESOLUTION

You can use the FRS debug log files and the outbound log of the FRS database to locate the source of FRS replicating files. To determine if disk defragmentation utilities are causing excessive FRS replication traffic in your environment, use either of the methods described in this section.

Back to the top

Method One: Searching the FRS Debug Logs

To search the FRS debug logs to determine the reason for replication:

Open a Windows NT CMD prompt, and then type: cd /d %systemroot%\debug
From the %SystemRoot%\Debug folder, use either of the following steps:
- If you are using FRS in Windows 2000 or Windows 2000 Service Pack 1 (SP1), run the following command:
  findstr /i "contentcmd" %systemroot%\debug\ntfrs_00??.log
- If you are using FRS in Windows 2000 Service Pack 2 (SP2), Windows Server 2003, increase log level verbosity to 4, and then let the service run for a sufficient period of time to capture replication events. To filter the FRS change events, run the following command:
  findstr /i "content" %systemroot%\debug\ntfrs_00??.log
  NOTE: Run this command if you are using the hotfix that is described in the following Microsoft Knowledge Base article:
  272567 (http://support.microsoft.com/kb/272567/EN-US/ ) File Replication Service Improvements in Windows 2000 Service Pack 2
  For additional information about increasing the logging levels, click the article number below to view the article in the Microsoft Knowledge Base:
  221111 (http://support.microsoft.com/kb/221111/EN-US/ ) Description of FRS Entries in the Registry

Use the Layout tab in the CMD sessions properties to set the window size width and height to accommodate FINDSTR output. To start, set the width at 110 characters, the height at 45, and the resolution to 1024 x 768.

The "Content|ContentCmd" string in the FINDSTR output displays the change that took place for files on NTFS formatted partitions. The FINDSTR output for files replicated by disk defragmentation utilities have lines at the "[DatOvrWrt ]" or the [StrmOvrWrt] string. For example:

d:\>FINDSTR /i "ContentCmd" ntfrs_00??.log 

---------- D:\WINNT\DEBUG\NTFRS_0001.LOG
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ]   
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]   
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker:   648: 11585: S4: HH:MM:SS> ContentCmd

Flags set in the NTFS change log describe modifications to FRS-replicated files. FRS replication is predicated on closed files that reside on NTFS 5.0 formatted partitions in FRS replicated folders. The reasons for changes in FRS are displayed in the following table:

           Close		- Change log close record
           Create		- File or folder was created
           Delete		- File or folder was deleted
           RenNew		- File or folder was renamed
           DatOvrWrt	- Main file data stream was overwritten
           DatExt		- Main file data stream was extended
           DatTrunc	- Main file data stream was truncated
           Info		- Basic info change (attrib, last write time, etc.)
           Oid		- Object Id change
           StreamNam	- Alternate data stream name change
           StrmOvrWrt	- Alternate data stream was overwritten
           StrmExt		- Alternate data stream was extended
           StrmTrunc	- Alternate data stream was truncated
           EAChg		- Extended file attribute was changed
           Security	- File access permissions changes
           IndexableChg	- File change requires re-indexing.
           HLink		- Hardlink change
           CompressChg	- File commpression attribute changed
           EncryptChg	- File encryption changed
           Reparse		- Reparse point changed

To increase the log endurance and verbosity for FRS, modify the following registry entries:

Set the Debug Maximum Log Messages (REG_DWORD) registry entry to 20,000 (do not enter the comma).
Set the Debug Log Files (REG_DWORD) registry entry to a value between 20 and 50.
For Windows 2000 SP2 and Windows Server 2003 versions of FRS, set the Debug Log Severity (REG_DWORD) registry entry to 4 to see Content: strings. If you do not to set the log verbosity to a high enough value, you may be able to filter the debug logs against the "UsnReason:" string to locate the the type of modification that took place.

NOTE: Follow the procedure in this step if you used the hotfix described in the following Microsoft Knowledge Base article:
272567 (http://support.microsoft.com/kb/272567/EN-US/ ) File Replication Service Improvements in Windows 2000 Service Pack 2

For additional information about increasing the logging levels, click the article number below to view the article in the Microsoft Knowledge Base:

221111 (http://support.microsoft.com/kb/221111/EN-US/ ) Description of FRS Entries in the Registry

Back to the top

Method Two: Searching the NTFRS Outbound Log

To search the NTFRS Outbound Log by using the Ntfrsutl (Ntfrsutl.exe) utility and Iologsum.cmd:

Identify the computer that is originating excessive updates, and then use Ntfrsutl from the Windows 2000 Resource Kit to dump the FRS outbound log.

To do so, from the Windows NT CMD prompt, type:
ntfrsutl outlog >outlog.txt

Use the Iologsum.cmd FRS troubleshooting utility to structure the Outlog file that is created in the preceding step. Iologsum supports the following arguments:

c:\>iologsum.cmd /?
Usage:  C:\iologsum.cmd  [-sort=xxx]  datafile

Process the output of "ntfrsutl inlog" or "ntfrsutl  outlog"
and summarize the pending change orders.

   Sort Keyword        Sort the output by:
  -sort=seqnum         Sequence Number (default)
  -sort=version        File Version Number
  -sort=filename       File Name
  -sort=size           File Size
  -sort=fileguid       File Guid
  -sort=origguid       Originator Guid
  -sort=eventtime      Event Time

You can use the Iologsum switches to identify the sources of replication. Use the -sort=eventtime switch to identify patterns of replication changes, for example, changes that occur in a scheduled job or process. Use the following command line to sort the Outlog.txt file by event time:

c:\>iologsum -sort=eventtime outlog.txt

The following text is an example of output from an outlog that is sorted by eventtime:

Event Time       Ver       OrigiGUID File    Replication Reason

11/11/2000 3:03	4	646d2289	FILE001.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	5	646d2289	FILE002.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE003.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE004.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	6	646d2289	FILE005.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	5	646d2289	FILE006.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE007.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	5	646d2289	FILE008.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE009.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE010.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE011.TXT   [Content  DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03	4	646d2289	FILE012.TXT   [Content  DIR MD5/ StrmOvrWrt ]
...
11/11/2000 6:00	2	0c88c766	FILE101.TXT   [Content  MD5/ DatOvrWrt ]
11/11/2000 6:00	4	0c88c766	FILE102.TXT   [Content  MD5/ DatOvrWrt ]
11/11/2000 6:00	10	0c88c766	FILE103.TXT   [Content  MD5/ DatOvrWrt ]
11/11/2000 6:00	2	0c88c766	FILE104.TXT   [Content  MD5/ DatOvrWrt ]
11/11/2000 6:00	3	0c88c766	FILE105.TXT   [Content  MD5/ DatOvrWrt ]
11/11/2000 6:00	2	0c88c766	FILE106.TXT   [Content  MD5/ DatOvrWrt ]

The Iolog output shows a pattern of replication at 3:03AM and 6:00 AM that originates from replica members 646d2289 and 0c88c766 (these numbers can be resolved to computer names in the Ntfrsutil sets output). The reason for the changes are [StrmOvrWrt ] and [DatOvrWrt]. The version columns show that the same files have been replicated anywhere from two to ten times. Files that have high version numbers may illustrate that excessive updates are occurring.

Back to the top

Workarounds to Excessive Replication by Disk Defragmentation Utilities

To prevent disk defragmentation problems from causing excessive replication of FRS files:

Do not run defragmentation utilities on volumes that contain FRS replicated files. In the short term, disable disk defragmentation routines to stop FRS updates from generating and allow the staging folders to empty.
Exclude FRS-replicated folders from being optimized by defragmentation utilities. Diskkeeper allows administrators to exclude folders from fragmentation.
Remove the member computer from all replica sets hosted by the volume that is being defragmented, run the defragmentation utility, and then add the member computer back to the replica sets. When you complete this procedure, you force a non-authoritative restore (D2) but no file data is moved over the network.

FRS can generate the staging file along with a new MD5 checksum. If the checksum matches the value in the IDTable, then abort the local change order. FRS does a lot of work to generate a staging file for each file that gets defragmented, but it does not replicate the file if the file has not changed.

Back to the top

MORE INFORMATION

FRS is a multiple-threaded, multiple-master replication engine that replaces the LanMan Replication (LMREPL) service in Microsoft Windows NT versions 3.x and 4.0. Windows 2000 domain controllers and servers use FRS to replicate system policy and logon scripts for Windows 2000 and down-level clients that reside in the SYSVOL.

FRS can also replicate content between Windows 2000-based servers that host the same fault-tolerant DFS roots or child node replicas. FRS replicates files based on version changes or changes to permissions on files and replicated folders.

Back to the top