Diagnoses and treatment of black hole routers

On a TCP/IP wide area network (WAN), communication over some routes may fail if intermediate network segments have packet sizes smaller than the communicating hosts, and routers do not send appropriate ICMP responses to this condition. Alternatively, the firewall on the path may drop such responses. A router that causes this condition is sometimes known as a "black hole" router. The Ping utility, a standard utility installed with the Microsoft Windows TCP/IP protocol, can be used to find black hole routers. Some recommendations are provided to work around or fix problems with black hole routers.

When a network router receives a packet larger than the Maximum Transfer Unit (MTU) of the next network segment, and that packet's IP layer "don't fragment" bit is flagged, the router should send an ICMP destination unreachable message back to the sending host. When this does not happen, packets can be dropped, causing a variety of errors that will vary with the application that is communicating over the failed link. These errors will not occur when an application connects to a computer on a local subnet. The problem may seem intermittent, but on closer examination, it can be duplicated, such as in having a client read a large file from a remote host.

The Ping utility can be used to find black hole routers by using the -f (do not fragment) and the -l (buffer size) parameters. Setting the -f parameter will cause the Ping utility to send an ICMP echo packet with the IP "do not fragment" bit set. The -l parameter sets the buffer, or payload, size of the ICMP Echo packet. The largest buffer that can be sent unfragmented equals the MTU minus the IP and ICMP headers (MTU-28) of the smallest MTU along a route. For example, because Ethernet has an MTU of 1500 bytes, under the best circumstances, Ping could echo an unfragmented packet with an ICMP buffer of 1472 bytes. The syntax for Ping in this case would be:

Ping <computer name or IP address> -f -l 1472

This should work on all local IP addresses. If the MTU of all segments of a routed connection are 1500 or larger, the packet should be returned as well. If there are intermediate segments with smaller MTUs, and routers return the appropriate ICMP Destination Unreachable packet, the utility should display "Packet needs to be fragmented but DF set". If there are segments along the route with smaller MTUs, and the appropriate ICMP packet is not returned, the Ping utility should display "Request timed out." The default MTUs of common network media are described in Knowledge Base article 314496  (http://support.microsoft.com/kb/314496/EN-US/ ) .

By changing the -l parameter on successive Pings, the largest unfragmented packet that will travel a specific route can be found. The smallest MTU in general use is 576 bytes, so you should be able to safely start with an ICMP buffer of 548, then work up from there. For example, if Ping <host name or address> -f -l 972 returns packets and Ping <host name or address> -f -l 973 fails, the largest MTU that can be used over that route is 1000 (972+28).

To fix or work around black hole routers, there are four possible solutions:

1. Enable PMTU Black Hole Detection on Windows hosts that will be communicating over a wide area connection, as documented in Microsoft Knowledge Base article 136970  (http://support.microsoft.com/kb/136970/EN-US/ ) . In this case, Windows NT 3.51 Service Pack 2 or later or Windows NT 4.0 should be used.
2. Configure intermediate routers to send ICMP type 3 code 4 (destination unreachable don't fragment (DF) bit sent and fragmentation required) messages. This may require upgrading router software or firmware, router configuration or router replacement.
   
3. Disable PMTU discovery on Windows hosts that communicate over troublesome routes. This will configure the default MTU to 576 bytes. This could cause significant degradation in network performance.
   
4. Set the MTU of the host interface to be the largest the black hole router can handle. This guarantees the largest possible packet size will be sent over that connection, but will cause local traffic, and traffic over routed connections without problems, to use smaller packets than they would otherwise. This workaround assumes that you have determined the MTU and the state of all possible links that could be used by the host in question.

For more information, click the following article numbers to view the articles in the Microsoft Knowledge Base: For more information, see Internet RFC 1191 and RFC 1435. To do this, visit the following Web site:See also chapter 6, "TCP/IP Implementation Details," in the Windows NT Server Resource Kit. To do this, visit the following Microsoft Web site: