Basic L2TP/IPSec Troubleshooting in Windows XP

This article provides information to help you troubleshoot Layer 2 Tunneling Protocol (L2TP) and Internet Protocol Security (IPSec) in Windows XP.

L2TP is a standard that allows the transfer of Point to Point Protocol (PPP) traffic between different networks (described in Request for Comments [RFC] #2661). L2TP is combined with IPSec to provide both tunneling and security for Internet Protocol (IP), Internetwork Packet eXchange (IPX), and other protocol packets across any IP network.

L2TP encapsulates original packets inside a PPP frame (performing compression when possible) and inside a User Datagram Protocol (UDP)-type packet assigned to port 1701. Because the UDP packet format is an IP packet, L2TP automatically uses IPSec to secure the tunnel, in accord with the security settings in the user configuration of the L2TP tunnel. The IPSec Internet Key Exchange (IKE) protocol negotiates security for the L2TP tunnel; certificate-based authentication is the default. This authentication process uses computer certificates, not user certificates, to verify that the source and destination computers both trust each other. If IPSec transport security is successfully established, L2TP negotiates the tunnel (including compression and user authentication options) and performs access control that is based on the user identity.

The L2TP/IPSec packet structure looks like the following example. The PPP Payload contains the original IP datagram, and the italicized text represents what is encrypted with IPSec. Microsoft Point-to-Point Encryption Protocol (MPPE), which can be used to secure the PPP payload when the Extensible Authentication Protocol Transport Layer Security (EAP-TLS) or Microsoft Challenge-Handshake Authentication Protocol (MS-CHAP) is used, is negotiated by Windows when the L2TP peer (client or server) requests it.

MPPE uses the Rivest-Shamir-Adleman (RSA) RC4 stream encryption and either 40-bit, 56-bit, or 128-bit secret keys. MPPE keys are generated from the MS-CHAP and EAP-TLS user-authentication process. The remote access server can be configured to require data encryption. If the remote access client cannot perform the required encryption, the connection attempt is rejected and the following error message (#742) appears: IPSEC is negotiated before PPP starts; MPPE is negotiated after PPP starts. PPP runs over L2TP using IPSec. During the PPP authentication phase, a user name is sent to the Remote Access Server (RAS) component of the virtual private network (VPN) server by using the configured authentication protocol (MS-CHAP, for example). The RAS server then matches the user name and other call properties to a Remote Access Policy. Each policy has a profile, and RAS compares the conditions of the incoming call with the profile to determine whether to accept the connection request.

Considerations

• If the VPN client is behind any network device that is performing network address translation (NAT), the L2TP session fails because encrypted IPSec Encapsulating Security Payload (ESP) packets become corrupted. If the VPN client is on the same computer as Windows Internet Connection Sharing/Network Address Translation, the client can probably establish an L2TP session because NAT does not perform any IP address or port translation on packets that originate from its own node.
• You need a computer certificate with a private key, which you can find in the local computer personal certificate store. You can verify this by starting the Microsoft Management Console, MMC.exe, and adding the Certificates snap-in for the local computer. The user must have the local administrator rights.
  
  If no computer certificate is found, L2TP issues a warning that you do not have a certificate, but L2TP does not know whether the certificate has a properly installed and associated private key for the existing certificate. Internet Key Exchange (IKE) determines this during negotiation. Start the Local Computer Certificates snap-in, double-click Certificate, and verify that General indicates that "You have a private key that corresponds with this certificate." Also verify that the certificate path is complete, and that the certificate is valid.
  
  The client must have a computer certificate whose root certificate authority is the same as the certificate on the gateway certificate. The reason for the certificate failure is noted by IKE in the Security log event entry. Both sides must be able to process the certificate validation successfully. If certificate authentication is successful, an entry in the Security log notes the successful event of an IPSec main mode SA establish (logon/logoff).
• IKE failure to negotiate: You can verify whether IPSec is succeeding by starting Microsoft Management Console (MMC.exe) and adding the IP Security Monitor (as local administrator). Set the options to refresh at one-second intervals. If you see the IPSec SA appear, IPSec succeeded. You can conclude that L2TP is the source of the problem. With the Windows XP support tools installed, use the netdiag /test:ipsec /v /debug command to see the details of IPSec policy. (You cannot see the whole policy if a domain administrator has set policy on your local computer.)
  
  Also note the following issues:
  • IKE timeout: IKE may time out during the initial negotiation request if routers in front of the VPN server do not allow UDP port 500 through. IKE also times out if the VPN server does not have appropriate IPSec policy configured, which usually means either that the RRAS server does not have L2TP ports enabled or that a manual IPSec policy setting is not correctly configured. When IKE times out, the audit log shows that peer failed to reply and that a network capture trace shows ISAKMP UDP packets initiating only from your client. If configured specifically for L2TP, the VPN client responds with the following error message:
    The security negotiation timed out.
    If configured with Automatic, it attempts the task again using the next protocol on its list, namely PPTP.
  • IKE Failure: Failure to negotiate IKE and the reasons for the failure are recorded in the security log if you enable Security Settings, Logon/Logoff failure audits. IKE can fail because certificate credentials do not work, or because there is an IPSec policy configuration problem if they set manual IPSec policy.
    
    Success of IKE negotiation is noted in the Audit log. For the entire IPSec security negotiation to be successful, you need both a main mode SA establish and a quick mode SA establish for UDP port 1701.
• If one of the following symptoms occurs, IPSec is not causing the problem:
  • The Audit log shows successful main mode SA establish and successful quick mode SA establish.
  • The network capture trace shows ESP traffic originating from the client or server.
  • Ipsecmon.exe shows an IPSec SA.
  Note that there are always two IPSec SAs established: one for each direction, each with its own security parameter Index (SPI); however, Ipsecmon.exe shows only the outbound SA.
• ESP blocked: When NAT is in front of the client or the routers are in front of the VPN, the server may not allow protocol 50 (ESP) through. Outbound ESP traffic with the SPI number appears, but inbound ESP packets from the gateway, with a different SPI number, do not appear.
• ESP modified: If NAT, or perhaps a faulty switch or other network node, is modifying or corrupting packets anywhere on the path, the packets are dropped by the IPSec driver, and Event 4285 "Failed to authenticate hash" appears in the System log of the receiving system. Packets can also be corrupted by a network interface that has IPSec offload capabilities. To determine whether an interface has this capability, use the netsh support tool. Type following command:
  netsh int ip show offload
• If the IPSec offload capability of a NIC is the suspected cause, start a Network Monitor capture and use Ipsecmon.exe to analyze each connection attempt. Examine the "Confidential Bytes Received" counter in Ipsecmon to determine whether packets are being lost on receive. You can also set the HKLM\System\CurrentControlSet\Services\IPSEC\EnableOffload DWORD registry value to 0. If the connection then succeeds, the issue is offload-related. Another troubleshooting alternative is to turn off the IPSec automatic policy.
• If the IPSec Policy Agent was stopped by using the Services snap-in or the net stop policyagent command, the L2TP automatic IPSec policy configuration is lost. For VPN clients, the policy is automatically plumbed when the client connectoid is started. Make sure that the IPSec policy agent service is started and running before you start the client connectoid. After you click Connect and the connection attempt is in progress, you can use the netdiag /test:ipsec /v /debug command to see IPSec statistics and active filters. Note that you cannot use the /debug option if you do not have domain administrative permissions.

For more information about VPN, see "Microsoft Privacy Protected Network Access: Virtual Private Networking and Intranet Security" at the following location:

For additional information, click the following article numbers to view the articles in the Microsoft Knowledge Base: