How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003

Denial of service (DoS) attacks are network attacks that are aimed at making a computer or a particular service on a computer unavailable to network users. Denial of service attacks can be difficult to defend against. To help prevent denial of service attacks, you can use one or both of the following methods:

• Keep your computer updated with the latest security fixes. Security fixes are located on the following Microsoft Web site:
  http://www.microsoft.com/security

  (http://www.microsoft.com/security)
• Harden the TCP/IP protocol stack on your Windows Server 2003 computers. The default TCP/IP stack configuration is tuned to handle standard intranet traffic. If you connect a computer directly to the Internet, Microsoft recommends that you harden the TCP/IP stack against denial of service attacks.

TCP/IP Registry Values That Harden the TCP/IP Stack

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
The following list explains the TCP/IP-related registry values that you can configure to harden the TCP/IP stack on computers that are directly connected to the Internet. All of these values should be created under the following registry key, unless otherwise noted:

NOTE: All values are in hexadecimal unless otherwise noted.

• Value name: SynAttackProtect
  Key: Tcpip\Parameters
  Value Type: REG_DWORD
  Valid Range: 0,1
  Default: 0
  
  This registry value causes Transmission Control Protocol (TCP) to adjust retransmission of SYN-ACKS. When you configure this value, the connection responses time out more quickly during a SYN attack (a type of denial of service attack).
  
  The following parameters can be used with this registry value:
  • 0 (default value): No SYN attack protection
  • 1: Set SynAttackProtect to 1 for better protection against SYN attacks. This parameter causes TCP to adjust the retransmission of SYN-ACKS. When you set SynAttackProtect to 1, connection responses time out more quickly if the system detects that a SYN attack is in progress. Windows uses the following values to determine whether an attack is in progress:
    • TcpMaxPortsExhausted
    • TCPMaxHalfOpen
    • TCPMaxHalfOpenRetried
  Note In Windows Server 2003 Service Pack 1 and later, the default value for the SynAttackProtect registry entry is 1.
  
  Note The TcpMaxPortsExhausted registry key is obsolete in Windows XP SP2 and in later versions of Windows operating systems.
• Value name: EnableDeadGWDetect (Applies to Windows 2003 only)
  Key: Tcpip\Parameters
  Value Type: REG_DWORD
  Valid Range: 0, 1 (False, True)
  Default: 1 (True)
  
  The following list explains the parameters that you can use with this registry value:
  • 1: When you set EnableDeadGWDetect to 1, TCP is permitted to perform dead-gateway detection. When dead-gateway detection is enabled, TCP may ask the Internet Protocol (IP) to change to a backup gateway if a number of connections are experiencing difficulty. Backup gateways are defined in the Advanced section of the TCP/IP configuration dialog box in the Network tool in Control Panel.
  • 0: Microsoft recommends that you set the EnableDeadGWDetect value to 0. If you do not set this value to 0, an attack may force the server to switch gateways and cause it to switch to an unintended gateway.
• Value name: EnablePMTUDiscovery
  Key: Tcpip\Parameters
  Value Type: REG_DWORD
  Valid Range: 0, 1 (False, True)
  Default: 1 (True)
  
  The following list explains the parameters that you can use with this registry value:
  • 1: When you set EnablePMTUDiscovery to 1, TCP tries to discover either the maximum transmission unit (MTU) or the largest packet size over the path to a remote host. TCP can remove fragmentation at routers along the path that connect networks with different MTUs by discovering the path MTU and limiting TCP segments to this size. Fragmentation adversely affects TCP throughput.
  • 0: Microsoft recommends that you set EnablePMTUDiscovery to 0. When you do so, an MTU of 576 bytes is used for all connections that are not hosts on the local subnet. If you do not set this value to 0, an attacker may force the MTU value to a very small value and overwork the stack.
    
    Important Setting EnablePMTUDiscovery to 0 negatively affects TCP/IP performance and throughput. Even though Microsoft recommends this setting, it should not be used unless you are fully aware of this performance loss.
• Value name: KeepAliveTime
  Key: Tcpip\Parameters
  Value Type: REG_DWORD-Time in milliseconds
  Valid Range: 1-0xFFFFFFFF
  Default: 7,200,000 (two hours)
  
  This value controls how frequently TCP tries to verify that an idle connection is still intact by sending a keep-alive packet. If the remote computer is still reachable, it acknowledges the keep-alive packet. Keep-alive packets are not sent by default. You can use a program to configure this value on a connection. The recommended value setting is 300,000 (5 minutes).
• Value name: NoNameReleaseOnDemand
  Key: Netbt\Parameters
  Value Type: REG_DWORD
  Valid Range: 0, 1 (False, True)
  Default: 0 (False)
  
  This value determines whether the computer releases its NetBIOS name when it receives a name-release request. This value was added to permit the administrator to protect the computer against malicious name-release attacks. Microsoft recommends that you set the NoNameReleaseOnDemand value to 1.

Troubleshooting

When you change the TCP/IP registry values, you may affect programs and services that are running on the Windows Server 2003-based computer. Microsoft recommends that you test these settings on nonproduction workstations and servers to confirm that they are compatible with your business environment.