"Error 792: The L2TP connection attempt failed because security negotiation timed out." error message when VPN clients try to complete a connection to ISA Server or to Microsoft Forefront Threat Management Gateway, Medium Business Edition

Virtual private network (VPN) clients may be unable to connect to a network through a VPN server that is running Microsoft Internet Security and Acceleration (ISA) Server 2006, ISA Server 2004, or Microsoft Forefront Threat Management Gateway, Medium Business Edition. In this scenario, the VPN clients may receive the following error message:

This issue may occur if both the following conditions are true:

• The VPN clients use Layer 2 Tunneling Protocol (L2TP) to create the VPN connection.
• ISA Server or Microsoft Forefront Threat Management Gateway, Medium Business Edition is configured to block IP fragments.

To resolve this issue, turn off the option that blocks fragmented IP packets. In Microsoft Forefront Threat Management Gateway, Medium Business Edition, follow these steps:

1. Start the Microsoft Forefront Threat Management Gateway, Medium Business Edition Management tool.
2. Expand ServerName, where ServerName is the name of the computer that is running Microsoft Forefront Threat Management Gateway, Medium Business Edition.
3. Click Firewall Policy, and then in the task pane, click Configure IP Preferences.
4. Click the IP Fragments tab, click to clear the Block IP fragments check box, and then click OK.
5. Click Apply to update the firewall policy, and then click OK.

In ISA Server, follow these steps:

1. Start the ISA Server Management tool.
2. Expand ServerName, where ServerName is the name of your ISA Server computer.
3. Expand Configuration, and then click General.
4. Under Additional Security Policy, click Define IP Preferences.
   
   Note In ISA Server 2006, click Configure IP Protection.
5. Click the IP Fragments tab, click to clear the Block IP fragments check box, and then click OK.
6. Click Apply to update the firewall policy, and then click OK.

IPSec uses the Internet Key Exchange (IKE) protocol for mutual computer authentication and for the exchange of session keys in an L2TP VPN connection. The IKE negotiation information cannot fit inside a Maximum Transmission Unit (MTU). Because of this, the IKE negotiation packet is fragmented or broken into smaller multiple datagrams. When you filter fragmented packets in ISA Server or in In Microsoft Forefront Threat Management Gateway, Medium Business Edition, the IKE negotiation packets are dropped by ISA Server. Therefore, the VPN connection cannot be completed successfully.

Note IKE negotiation is always used regardless of your IPSec authentication mechanism, such as preshared keys, Kerberos protocol, or certificates.

For additional information about why you might want to filter IP fragments, search on "packet fragments" in ISA Server Help.