How to Troubleshoot Black Hole Router Issues

This article defines the term "black hole" router, describes a method of locating black hole routers, and suggests three ways to avoid the data loss that can occur because of a black hole router.

On a TCP/IP-based wide area network (WAN), communication over some routes may fail if an intermediate network segment has a maximum packet size that is smaller than the maximum packet size of the communicating hosts--and if the router does not send an appropriate Internet Control Message Protocol (ICMP) response to this condition or if a firewall on the path drops such a response. Such a router is sometimes known as a "black hole" router.

You can locate a black hole router by using the Ping utility, which is a standard utility that is installed with the Microsoft Windows TCP/IP protocol. You can then use one of three methods of fixing or working around black hole routers.

When a network router receives a packet that is larger than the size of the Maximum Transmission Unit (MTU) of the next segment of a communications network, and that packet's IP layer "don't fragment" bit is flagged, the router is expected to send an ICMP "destination unreachable" message back to the sending host.

If the router does not send a message, the packet might be dropped, causing a variety of errors that vary with the program that is communicating over the unsuccessful link. (These errors do not occur if a program connects to a computer on a local subnet.) The behavior may seem intermittent, but closer examination shows that the behavior can be reproduced, for example, by having a client read a large file that is sent from a remote host.

Client-side Error

The client could not establish a connection to the remote computer. The most likely causes for this error are:

• Remote connections cannot be enabled at the remote computer.
• The maximum number of connections is exceeded at the remote computer.
• A network error occurs while establishing the connection.
  Server-side Error: Event ID 1004
  Source: TermService
  Description: "The terminal server cannot issue a client license. It was unable to issue the license due to a changed (mismatched) client license, insufficient memory, or an internal error. Further details for this problem may have been reported at the client's computer."

Locating a Black Hole Router

You can use the Ping utility to locate a black hole router, by setting the -f and -l parameters when you type the ping command.

• The -f parameter causes the Ping utility to send an ICMP echo packet that has the IP "do not fragment" bit set.
• The -l parameter sets the buffer, or payload, size of the ICMP echo packet. You specify this size by typing a number after the -l parameter.

The largest buffer that can be sent unfragmented is equal to the smallest MTU that exists along a route, minus the IP and ICMP headers (in other words, the smallest MTU minus 28). For example, Ethernet has an MTU of 1,500 bytes, so under the best circumstances, the Ping utility can echo an unfragmented packet, plus an ICMP buffer, of 1,472 bytes (1,500 minus 28). The syntax for the ping command in this case is: For all local IP addresses, the expected results are as follows:

• If the MTU of every segment of a routed connection is at least 1,500, the packet is successfully returned.
• If there are intermediate segments that have smaller MTUs, and the routers return the appropriate ICMP "destination unreachable" packet, the Ping utility displays the message, "Packet needs to be fragmented but DF set."
• If there are intermediate segments that have smaller MTUs, and the routers do not return the appropriate ICMP "destination unreachable" packet, the Ping utility displays the message, "Request timed out."

By increasing the -l parameter on successive pings, you can identify how large an unfragmented packet can travel a specific route. The smallest MTU that is in general use is 576 bytes, so you can safely start with an ICMP buffer of 548 and then work up from there. For example, if the command Ping computer_name or IP_address -f -l 972 returns packets but Ping computer_name or IP_address -f -l 973 does not return packets, the largest MTU on that route is 1,000 (972 plus 28). The default MTUs of common network media are described in the following article in the Microsoft Knowledge Base:

Fixing or Working Around a Black Hole Router

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
The following three methods are ways to either fix or work around a black hole router.

Method 1

Enable PMTU Black Hole Detection on the Windows-based hosts that will be communicating over a WAN connection. Follow these steps:

1. Start Registry Editor (Regedit.exe).
2. Locate the following key in the registry:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tcpip\parameters
3. On the Edit menu, click Add Value, and then add the following registry value:
   Value Name: EnablePMTUBHDetect
   Data Type: REG_DWORD
   Value: 1
4. Quit Registry Editor, and then restart the computer.

Method 2

Configure intermediate routers to send ICMP Type 3 Code 4 messages ("destination unreachable, don't fragment (DF) bit sent and fragmentation required"). This might require a router software or firmware upgrade, router reconfiguration, or router replacement.

Method 3

Set the MTU of the host interface to be the largest size that the black hole router can handle, to guarantee that the largest possible packet size is sent over that connection. However, note that local traffic then uses smaller packets than necessary, as will traffic that uses the routed connections without problems.

This workaround assumes that you have identified the MTU and the state of all possible links that the host might use. After you identify the largest MTU size that is supported, manually set the MTU. Follow these steps:

1. Click Start, and then click Control Panel.
2. Double-click Network and Internet Connections, and then click to open the Network Connections folder.
3. If more than one network connection is listed, for each connection, double-click the connection and then click the Support tab of the Status interface that opens. The connection that shows a Default Gateway entry is probably the network connection that is used to connect to the Internet. Note the name of the connection (for example, "Local Area Connection 2").
4. Start Registry Editor (Regedit.exe).
5. Under the HKEY_LOCAL_MACHINE tree, go to the following key:
   SYSTEM\CurrentControlSet\Control\Network\{4D36E972-E325-11CE-BFC1-08002BE10318}\
6. Under that key are one or more keys that have numeric identifiers. Each of these keys has a Connection subkey. Examine each of the keys that look like this:
   ID_for_Adapter\Connection
   The Name value in the Connection subkey provides the network connection name that is used in the Network Connections folder. When you find the one that matches the name that you found in step 3, make a note of the ID_for_Adapter that the network connection name is under.
7. Return to HKEY_LOCAL_MACHINE, and then locate the following key
   
   SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\ID_for_Adapter
   where ID_for_Adapter is the number that you noted in step 6. When you highlight this key, several values appear on the right side of the screen, including DefaultGateway and EnableDHCP.
8. Right-click the right side of the screen, click New, and then click DWORD Value. Name the value MTU.
9. Double-click the value so that you can edit the value, change Base to Decimal, and then enter the largest acceptable MTU size, which is the size that you identified by using the Ping tests.
10. Quit Registry Editor.

Note that if you still encounter problems with some servers, you might need to set the MTU lower than the Ping tests indicate because of other routers in that specific path. Repeatedly lower the MTU by 10 until access to those sites is successful.

For additional information manually setting the MTU, click the article number below to view the article in the Microsoft Knowledge Base:
For additional information, see Internet RFC 1191 and RFC 1435, which are available from the following Internic Web site: