Basic L2TP/IPSec troubleshooting in Windows 2000

This article contains information that helps troubleshoot Layer Two Tunneling Protocol (L2TP) and Internet Protocol security (IPSec) in Windows.

L2TP is a standard that enables the transfer of Point-to-Point Protocol (PPP) traffic between different networks, as described in Request for Comments (RFC) 2661, "Layer Two Tunneling Protocol L2TP." L2TP is combined with IPSec to provide both tunneling and security for Internet Protocol (IP), Internetwork Packet Exchange (IPX), and other protocol packets across any IP network.

L2TP encapsulates original packets inside a PPP frame and performs compression when it is possible. Additionally, L2TP encapsulates inside a User Datagram Protocol (UDP)-type packet that is assigned to port 1701. Because the UDP packet is a part of the IP transport protocol, L2TP automatically uses IPSec to help secure the tunnel. L2TP does this based on the security settings in the user configuration of the L2TP tunnel. By default, the IPSec Internet Key Exchange (IKE) protocol negotiates security for the L2TP tunnel by using certificate-based authentication. This authentication process uses computer certificates, not user certificates, to verify that source computers and destination computers trust each other. If IPSec transport security is successfully established, L2TP negotiates the tunnel and performs access control based on the user's identity. When L2TP negotiates the tunnel, it negotiates compression and user-authentication options.

The L2TP/IPSec packet structure looks similar to the following example.

Note The PPP payload contains the original IP datagram. Italic text represents what is encrypted with IPSec. Microsoft Point-to-Point Encryption Protocol (MPPE) is negotiated by Windows when the L2TP peer (client or server) requests it. MPPE can be used to help secure the PPP payload when Extensible Authentication Protocol Transport Layer Security (EAP-TLS) or Microsoft Challenge Handshake Authentication Protocol (MS-CHAP) is used.

MPPE uses the Rivest-Shamir-Adleman (RSA) RC4 stream cipher and 40-bit, 56-bit, or 128-bit secret keys. MPPE keys are generated from the MS-CHAP and EAP-TLS user-authentication processes. The remote access server can be configured to require data encryption. If the remote access client cannot perform the required encryption, the connection attempt is rejected, and you may receive the following error message: IPSec is negotiated before PPP starts; MPPE is negotiated after PPP starts. PPP runs over L2TP; it uses IPSec to do this. During the PPP authentication phase, a user name is sent to the remote access server component of the virtual private network (VPN) server by using the configured authentication protocol, such as MS-CHAP. The remote access server then matches the user name and other call properties to a remote access policy. Each policy has a profile, and the remote access server compares the conditions of the incoming call with the profile to determine whether to accept the connection request.

Considerations

• If your Microsoft Windows 2000 Professional with Service Pack 2 or earlier-based VPN client is behind any network device performing network address translation (NAT), the L2TP session fails because encrypted IPSec Encapsulating Security Payload (ESP) packets become corrupted. If the VPN client is on the same computer as Windows Internet Connection Sharing/Network Address Translation, the client is probably able to establish an L2TP session because NAT does not perform any IP address or Port translation when packets originate from its own node. For additional information about improvements to IPSec that enable communication for VPN clients that are behind NAT devices, click the following article number to view the article in the Microsoft Knowledge Base:
  818043  (http://support.microsoft.com/kb/818043/ ) L2TP/IPSec NAT-T update for Windows XP and Windows 2000
• You need a machine certificate with a private key. The private key can be found in the Local Computer Personal Certificate store.
  
  For additional information about installing a certificate, click the following article number to view the article in the Microsoft Knowledge Base:
  253498  (http://support.microsoft.com/kb/253498/ ) How to install a certificate for use with IP Security
  If a machine certificate is not found, L2TP issues a warning that you do not have a certificate, but it does not know whether the certificate has a correctly installed and associated private key for the existing certificate. IKE determines this during negotiation. Start the Local Computer Certificates snap-in, double-click Certificate, and verify that General indicates "You have a private key that corresponds with this certificate." Also, verify that the certificate path is complete and that the certificate is valid.
  
  The client must have a machine certificate whose root certification authority is the same as the certificate on the gateway certificate. The reason for the certificate failure is noted by IKE in the Security log entry. For additional information, click the following article number to view the article in the Microsoft Knowledge Base:
  257225  (http://support.microsoft.com/kb/257225/ ) Basic IPSec troubleshooting in Microsoft Windows 2000 Server
  For additional information about what kind of computer certificate works correctly, click the following article number to view the article in the Microsoft Knowledge Base:
  249125  (http://support.microsoft.com/kb/249125/ ) Using certificates for Windows 2000 and Cisco IOS VPN interoperation
  Both sides must be able to process the certificate validation successfully. If certificate authentication is successful, an entry in the Security log notes the successful event of an IPSec main-mode security association (SA) establish (logon/logoff).
• IKE failure to negotiate: You can verify whether IPSec is succeeding by running Ipsecmon.exe when you have local administrator permissions and when options are set to update at one-second intervals. If you see the IPSec SA appear, it indicates that IPSec succeeded, and you may conclude that L2TP is the source of the problem. Use the netdiag /test:ipsec /v /debug command to see the details of IPSec policy.
  
  Note You cannot see the whole policy if a domain administrator has set policy on your local computer. For additional information about Ipsecmon.exe and the netdiag command, click the following article number to view the article in the Microsoft Knowledge Base:
  257225  (http://support.microsoft.com/kb/257225/ ) Basic IPSec troubleshooting in Microsoft Windows 2000 Server
  Also note the following:
  • IKE timeout: IKE may time out during the initial negotiation request if routers in front of the VPN server do not enable UDP port 500 through. IKE also times out if the VPN server does not have appropriate IPSec policy configured. This typically means that the Routing and Remote Access server does not have L2TP ports enabled or that a manual IPSec policy setting is misconfigured. When IKE times out, the audit trail shows that peer did not reply, and a network capture trace shows ISAKMP UDP packets initiating only from your client. If the VPN client is configured specifically for L2TP, you receive the following error message:
    The security negotiation timed out.
    If the VPN client is configured with Automatic, it tries the task again by using the next protocol on its list, namely PPTP.
  • IKE failure: Failure to negotiate IKE and the reasons for the failure are recorded in the Security log if you enable Security Settings, Logon/Logoff failure audits. IKE can fail because certificate credentials do not work or because there is an IPSec policy configuration problem if they set manual IPSec policy. For additional information about automatic policy, click the following article number to view the article in the Microsoft Knowledge Base:
    248750  (http://support.microsoft.com/kb/248750/ ) Description of the IPSec policy created for L2TP/IPSec
    Success of IKE negotiation is noted in the audit trail. For the whole IPSec security negotiation to be successful, you need both a main-mode SA establish and a quick-mode SA establish for UDP port 1701.
• If one of the following symptoms occurs, IPSec is not causing the problem:
  • The audit trail shows successful main-mode SA establish and successful quick-mode SA establish.
  • The network capture trace shows ESP traffic originating from the client or server.
  • Ipsecmon.exe shows an IPSec SA.
  Note There are always two IPSec SAs established: one for each direction, each with its own Security Parameter Index (SPI). However, Ipsecmon.exe shows only the outgoing SA.
• ESP blocked: When NAT is in front of the client, or when the routers are in front of the VPN, the server may not enable protocol 50, ESP, through. Outgoing ESP traffic with the SPI number appears, but incoming ESP packets from the gateway with a different SPI number do not appear.
• ESP modified: If NAT or perhaps a faulty switch or other network node is modifying or corrupting packets anywhere on the path, the packets are dropped by the IPSec driver, and the following event appears in the System log of the receiving system:

  Event ID: 4285
  Description:
  Failed to authenticate hash.

  Packets may also be corrupted by a Network Interface with IPSec Offload capabilities. To determine whether an interface has this capability, use the following command:
  netsh int ip show offload
• If the IPSec offload capability of a network adaptor is the suspected cause, start a Network Monitor capture and Ipsecmon.exe to analyze each connection attempt and to verify the "Confidential Bytes Received" counter in Ipsecmon.exe to determine whether packets are being lost on receive. You can also set the DWORD value of the following registry subkey to zero:
  HKLM\System\CurrentControlSet\Services\IPSEC\EnableOffload
  If the connection then succeeds, the issue is offload related. Another troubleshooting alternative is to disable the automatic IPSec policy. For additional information about disabling the automatic IPSec policy, click the following article number to view the article in the Microsoft Knowledge Base:
  240262  (http://support.microsoft.com/kb/240262/ ) How to configure an L2TP/IPSec connection using preshared key authentication
• If the IPSec Policy Agent was stopped by using the Services snap-in or the net stop policyagent command, the L2TP automatic IPSec policy configuration is lost. For VPN clients, the policy is automatically plumbed when the client connectoid is started. Make sure that the IPSec policy agent service is started and running before you start the client connectoid. After you click Connect and the connection attempt is in progress, you can use the netdiag /test:ipsec /v /debug command to see IPSec statistics and active filters.
  
  Note If you do not have domain administrator permissions, you cannot use the /debug switch. For additional information about other possible issues, click the following article number to view the article in the Microsoft Knowledge Base:
  247231  (http://support.microsoft.com/kb/247231/ ) Event ID 20111, Error 792, or Error 781 when establishing an L2TP/IPSec connection

For more information about VPN, visit the following Microsoft Web site: For additional information about IPSec troubleshooting, click the following article number to view the article in the Microsoft Knowledge Base: