How to locate and correct disk space problems on NTFS volumes

The NTFS file system supports many volume and file-level features that may lead to what appears to be lost or misreported free disk space. This behavior may be apparent when an NTFS volume suddenly becomes very full for no reason, and yet an administrator cannot find the cause or locate the offending folders and files. This sometimes occurs because of malicious or unauthorized access to an NTFS volume on which very large files or a high quantity of small files are secretly copied, which then have their NTFS permissions removed or restricted. This behavior may also occur after a system malfunction or power outage occurs, causing volume corruption.

This article discusses how to check an NTFS file systems disk space allocation to discover offending files and directories or check for volume corruption. This article is intended for Windows 2000-and-later operating systems that support some advanced storage features and troubleshooting methods; however Windows NT users may also find this article useful. The following is a short list of reasons why the disk space allocation of an NTFS volume may appear to be misreported:

• The NTFS volume's cluster size is too large for the average-sized files being stored.
• File attributes or NTFS permissions prevent files or folders from being displayed or accessed by Windows Explorer or from a Windows command prompt.
• The folder path exceeds 255 characters.
• Directories or files contain invalid or reserved file names.
• NTFS metafiles (such as the Master File Table) have grown and cannot be de-allocated.
• Files or folders contain alternate data streams.
• NTFS File system corruption causes free space to be reported as being in use.
• Other NTFS features may cause file-allocation confusion.

Please use the following information to help optimize, repair, or gain a better understanding of how disk space is used on your NTFS volume or volumes.

Cluster size is too large

Disk space can be consumed only by files and directories that include internal NTFS metafiles like the Master File Table (MFT), directory indexes, and so forth. All file space allocation is consumed by using multiples of a cluster. A cluster is a collection of contiguous sectors. The cluster size is determined at the time the volume is formatted, and is further determined by the partition size. For more information about clusters, click the following article number to view the article in the Microsoft Knowledge Base: When a file is first created, it consumes a minimum of a single cluster of disk space, depending on the initial file size. When data is later added to a file, NTFS increases the file's allocation in multiples of the cluster size.

To determine the current cluster size and volume statistics, run a read-only chkdsk command from a command prompt, as follows and then view the resulting output. For example: NOTE: Multiply each value that is reported in kilobytes (KB) by 1024 to determine accurate byte counts. For example: 2906360 x 1024 = 2,976,112,640 bytes.

By using this output, you can determine how your disk space is being used, along with the default cluster size. To see if this is the optimal cluster size, determine the amount of wasted space by following these steps:

1. Double-click My Computer on the desktop, and then double-click the drive letter (for example, D) of the volume in question.
   
   This opens the volume and displays folders and files contained in the root.
2. Click any file or folder, and then click the Select All option on the Edit menu.
3. With all files and folders selected, right-click any file or folder, and then click the Properties option.
   
   This opens a Properties dialog box that contains a General tab. It also tabulates the total number of files and directories on the entire volume and provides two file size statistics: SIZE and SIZE ON DISK.

If you are not using NTFS compression for any files or folders contained on the volume, the difference between SIZE and SIZE ON DISK is wasted space because of a larger-than-necessary cluster size. You should attempt to use an optimal cluster size so that the SIZE ON DISK value is as close to the SIZE value as possible. An excessive discrepancy between the SIZE ON DISK and the SIZE value is an indication that the default cluster size is too large for the average file size that you are storing on the volume, and that it should be decreased. This can be done only by backing up the volume and then reformatting the volume by using the format command and the /a switch to specify the appropriate allocation size: (This example uses a 2-KB cluster size).

NOTE: Alternately, you can enable NTFS compression to regain space lost because of an incorrect cluster size; however, this may result in a slight decrease in performance.

File attributes or NTFS permissions

Both Windows Explorer and the following directory list command will total file and directory statistics for only those files and directories that you have permissions to access. Files always excluded by default include all hidden files and protected operating system files. This behavior may result in inaccurate file and folder totals and size statistics to be displayed by Windows Explorer or DIR command outputs. To include these types of files in the overall statistics, change Folder Options by following these steps:

1. Double-click My Computer on the desktop, and then double-click the drive letter (D) of the volume in question. This opens the volume and displays folders and files contained in the root.
2. On the Tools menu, click Folder Options, and then click the View tab.
3. Select the Show Hidden Files and Folders option, and then click to clear the Hide protected operating system files check box.
4. Acknowledge the warning message, and then click the Apply button.

This allows Windows Explorer and the DIR /A /S command to total all files and directories that are contained on the volume that the user has permissions to.

To determine which directories and files cannot be accessed, follow these steps:

1. From a command prompt, pipe the output of a DIR /A /S to a text file.
   
   For example: DIR D: /A /S >C:\D-DIR.TXT
2. Run ntbackup.exe, and then select the Detailed option on the Backup Log tab under Options on the Tools menu.
   
   This enables detailed backup logs.
3. On the Backup tab, back up the entire volume that is affected (D), and then start the backup.
4. After backup is complete, open the backup report and compare directory for directory the NTBackup log output with the D-DIR.TXT output that you saved in step 1.

Because backup can access all files, its report may contain folders and files not seen or counted by Windows Explorer and the DIR command. You may find it easier to use the NTBackup GUI to navigate the volume when you are looking for large files or directories that otherwise are inaccessible through Windows Explorer without actually backing up the volume.

After you locate files that you don't have access to, you should be able to add or change permissions by using the Security tab while you view the properties of the file or directory in Windows Explorer. By default, you will not have access to the System Volume Information directory. You will need to add the appropriate permissions for the directory to be included in the DIR /A /S command.

NOTE: You may run across directories or files that have no security tab, or find that you cannot re-assign permissions to the affected folders and files. You may receive the following error message while you try to access them: If you have any such directories, contact Product Support Services for additional assistance at the following Microsoft Web site:

Invalid file names

Directories or files that contain invalid or reserved file names may also be excluded from file and directory statistics. Directories or files that contain leading or trailing spaces are perfectly legal from an NTFS file system perspective, but are not legal from a Win32 subsystem point of view, and therefore, neither Windows Explorer nor a command prompt can reliably handle them. For more information, click the following article number to view the article in the Microsoft Knowledge Base: Often times it is not possible to rename or delete these files or folders. When you attempt to rename or delete them, you may receive one of the following error messages: If you have directories or files that cannot be deleted or renamed, contact Product Support Services at the following Microsoft Web site:

NTFS Master File Table (MFT) expansion

When an NTFS volume is first created and formatted, NTFS metafiles are created. One of these metafiles is called the Master File Table (MFT). It is very small when first created (approximately 16 KB), but it grows as files and directories are created on the volume. When a file is first created, it is entered into the MFT as a File Record Segment (FRS), which is always 1024 bytes (1 KB) in size. As files are added to the volume, the MFT grows as required. However, when files are deleted, the associated FRSs are marked as free to be reused, but the total FRSs and associated MFT allocation remains. This explains why, after deleting a large number of files, you don't regain the space used by the MFT.

To see exactly how large the MFT is, you can use the built-in defrag utility to analyze the volume. The resulting defrag report provides detailed information about the size and number of fragments in the MFT.

EXAMPLE:

However, for a more complete picture of how much space (overhead) is being used by the entire NTFS file system, perform a chkdsk, and then look at the resulting output for the following line: Currently, only third-party defrag utilities consolidate unused MFT FRS records and reclaim unused MFT allocated space.

Alternate data streams

NTFS allows files and directories to contain alternate data streams. This feature allow multiple data allocations to be associated with a single file or directory. Please be aware of the following limitations when you use alternate data streams on files and directories.

When a file or directory contains alternate data streams:

• Windows Explorer and the DIR command do not report the data in alternate data streams as part of the file size or volume statistics. Rather, they show only the total bytes for the primary data stream.
• The output from chkdsk accurately reports space used by a user's data files, including alternate data streams.
• Disk quotas accurately track and report all data stream allocations that are part of a user's data files.
• NTBackup records the number of bytes backed up in the backup log report. However it does not show which files contain alternate data streams, nor does it show accurate file sizes for files that include data in alternate streams.