CPU use may be more than 50 percent when an ISA Server 2004 computer is operating under heavy load conditions

Symptoms

When a Microsoft Internet Security and Acceleration Server (ISA) 2004-based computer is operating under heavy load conditions, you may experience high CPU use. For example, CPU use on the ISA Server computer may be more than 50 percent.

Cause

This behavior may occur because of the TCP/IP maximum transmission unit (MTU) setting that is applied during ISA Server installation.

To prevent an attacker from changing the MTU value, ISA Server 2004 disables path MTU (PMTU) discovery. This setting is documented in Microsoft security bulletin MS05-019. To see this bulletin, visit the following Microsoft Web site:

http://www.microsoft.com/technet/security/Bulletin/MS05-019.mspxNotes

By default, Windows uses an MTU setting of 1,480 bytes and accepts Internet Control Message Protocol (ICMP) messages that request smaller packet sizes.
If MTU discovery is disabled on a Windows-based server, the server uses an MTU setting of 576 bytes.

Resolution

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

To resolve the behavior that is caused by the MTU setting that is configured during ISA Server installation, follow these steps.

Important You must make this registry change if you have installed Windows Server 2003 Service Pack 1 (SP1) or the hotfix that is described in the following Microsoft Knowledge Base article:

898060 Network connectivity between clients and servers may fail after you install security update MS05-019 or Windows Server 2003 Service Pack 1

Click Start, click Run, type regedit, and then click OK.
Locate and then right-click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnablePMTUDiscovery
Value Type: REG_DWORD
Value: 0, 1 (False, True)
ISA default value: 0 (False)

Note If the EnablePMTUDiscovery entry is unavailable, create the entry.
If appropriate, set or change the value according to the following information:
- Value = 1
  When you set EnablePMTUDiscovery to 1, TCP tries to discover either the MTU or the largest packet size in the path of a remote host. TCP can remove fragmentation at routers in the path that connects networks that use different MTUs. TCP does this by discovering the path MTU and by limiting TCP segments to this size. Fragmentation adversely affects TCP throughput.
- Value = 0
  When you set EnablePMTUDiscovery to 0, an MTU of 576 bytes is used for all connections that are not related to hosts on the local subnet. If you do not set this value to 0, an attacker may force the MTU value to a very small value and overwork the stack.
  
  Notes
  - When you set EnablePMTUDiscovery to 0, TCP/IP performance and throughput are affected. You must be fully aware of the performance throughput before you set this value to 0.
  - When you install ISA Server 2004 or ISA Server 2004 Service Pack 1, this value also resets to 0.
  - ISA Server 2004 Service Pack 2 does not change the value of EnablePMTUDiscovery.
To participate in the discovery process, create an ICMP MTU access rule for ISA Server. To do this, follow the steps that are described in the following sections, depending on your configuration.
Exit Registry Editor, and then restart the computer.

ISA Server 2004, Standard Edition

Click Start, point to Programs, point to Microsoft ISA Server, and then click ISA Server Management.
In the left pane, expand ArrayName, and then click Firewall Policy.
In the task pane, click the Toolbox tab, and then click Protocols.
Under Protocols, click New, and then click Protocol.
In the Protocol definition name box, type ICMP MTU Discovery, and then click Next.
Click New, and then click ICMP in the Protocol type list.
In the Direction list, click Send Receive.
Type 4 in the ICMP Code box, type 3 in the ICMP Type box, and then click OK.
Click Next, click Finish, and then click Apply.
In the left pane, right-click Firewall Policy, click New, and then click Access Rule.
In the Access rule name box, type Allow ICMP MTU Discovery, and then click Next.
Click Allow, and then click Next.
In the This rules applies to list, click Selected protocols, and then click Add.
In the Protocols list, expand User-Defined.
Click ICMP MTU Discovery, click Add, click Close, and then click Next.
Click Add.
In the Network entities list, expand Networks.
Click External, and then click Add.
Click Internal, click Add, click Close, and then click Next.
Click Add.
In the Network entities list, expand Networks.
Click Local Host, click Add, click Close, and then click Next two times.
Click Finish, and then click Apply.

ISA Server 2004, Enterprise Edition

For ISA Server 2004, Enterprise Edition to participate in the ICMP MTU discovery process, create the ICMP protocol at the enterprise level, and then create the ICMP MTU access rule at the array level.

How to create the ICMP protocol at the enterprise level

Click Start, point to Programs, point to Microsoft ISA Server, and then click ISA Server Management.
In the left pane, expand Enterprise, and then click Enterprise Policies.
In the task pane, click the Toolbox tab, and then click Protocols.
Under Protocols, click New, and then click Protocol.
In the Protocol definition name box, type ICMP MTU Discovery, and then click Next.
Click New, and then click ICMP in the Protocol type list.
Type 4 in the ICMP Code box, type 3 in the ICMP Type box, and then click OK.
Click Next, click Finish, and then click Apply.

Note An Enterprise access rule cannot be created for user-defined ICMP protocols when you use the rule creation wizard.

For more information about how to use user-defined ICMP protocols in ISA Server 2004, Enterprise Edition policies, click the following article number to view the article in the Microsoft Knowledge Base:

902348 User-defined ICMP protocols are not displayed in the New Access Rule Wizard in ISA Server 2004 Enterprise Edition

How to manually create the ICMP MTU access rule if you have a single array in the network

Click Start, point to Programs, point to Microsoft ISA Server, and then click ISA Server Management.
In the left pane, expand Arrays, and then expand ArrayName.
Right-click Firewall Policy, click New, and then click Access Rule.
In the Access rule name box, type Allow ICMP MTU Discovery, and then click Next.
Click Allow, and then click Next.
In the This rules applies to list, click Selected protocols, and then click Add.
In the Protocols list, expand User-Defined.
Click ICMP MTU Discovery, click Add, click Close, and then click Next.
Click Add.
In the Network entities list, expand Networks.
Click External, and then click Add.
Click Internal, click Add, click Close, and then click Next.
Click Add.
In the Network entities list, expand Networks.
Click Local Host, click Add, click Close, and then click Next two times.
Click Finish, and then click Apply.

How to create the ICMP MTU access rule if you have multiple array members in the network

Method 1

Manually create the ICMP MTU access rule on the first array. To do this, follow the steps that are described to create the access rule for a single array.
Copy the ICMP MTU access rule. To do this, follow these steps:
1. Click Start, point to Programs, point to Microsoft ISA Server, and then click ISA Server Management.
2. In the left pane, expand Arrays, and then expand ArrayName.
  ArrayName is the first array for which you created the ICMP MTU access rule.
3. Click Firewall Policy, right-click the Allow ICMP MTU Discovery rule under Firewall Policy Rules, and then click Copy.
Paste the ICMP MTU access rule into each array. To do this, follow these steps:
1. In the ISA Server Management Microsoft Management Console (MMC), expand the array where you want to paste the ICMP MTU access rule, and then click Firewall Policy.
2. In the center pane, right-click the array policy rule that you want to immediately follow the ICMP MTU access rule, and then click Paste.
  
  Note If no array policy rules exist, you must create a new array policy rule before you can paste the ICMP MTU access rule into the array policy.
3. Click Apply.
Repeat steps 3a through 3c until you copy the ICMP MTU access rule for all the array members.

Method 2

Manually create the ICMP MTU access rule on the first array. To do this, follow the steps that are described to create the ICMP MTU access rule for a single array.
Export the ICMP MTU access rule. To do this, follow these steps:
1. In the ISA Server Management MMC, expand the array for which you created the ICMP MTU access rule, and then click Firewall Policy.
2. Right-click the Allow ICMP MTU Discovery rule under Firewall Policy Rules, and then click Export Selected.
3. In the Export Wizard, click Next.
4. On the Export Preferences page, click Next.
5. On the Export File Location page, click Browse.
6. Select a location where you will export the ICMP MTU Discovery rule, type MtuDiscoveryRule in the File name box, and then click Open.
7. Click Next, and then click Finish to exit the wizard.
8. Click OK when the progress indicator displays a message that the configuration has been successfully exported.
Import the ICMP MTU access rule into each array. To do this, follow these steps:
1. In the ISA Server Management MMC, expand the array where you want to import the ICMP MTU access rule, and then right-click Firewall Policy.
2. Click Import.
3. In the Import Wizard, click Next.
4. Click Browse, and then locate the folder where you saved the MtuDiscoveryRule file in step 2f.
5. Click the MtuDiscoveryRule file, and then click Open.
6. Click Next two times.
7. Click Finish.
8. Click OK when the progress indicator displays a message that the configuration has been successfully imported.
9. Click Apply.

References

For more information about PMTU discovery registry settings in Microsoft Windows 2000, click the following article number to view the article in the Microsoft Knowledge Base:

315669 How to harden the TCP/IP stack against denial of service attacks in Windows 2000

For more information about PMTU discovery registry settings in Microsoft Windows Server 2003, click the following article number to view the article in the Microsoft Knowledge Base:

324270 How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003