When you try to use a virtual private network (VPN) connection through a Microsoft Windows 2000 Terminal Services client session, you may receive the following error message:
Error: 789 "The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer"
However, you can successfully establish the VPN connection from the Terminal server console.
This issue may occur if both of the following conditions are true:
You establish a client session by using Windows 2000 Terminal Services.
You try to establish a VPN connection by using Layer Two Tunneling Protocol (L2TP) from the Terminal server to connect to a Windows 2000 Server-based computer that is configured as a VPN server.
To resolve this issue, you can use a preshared key on both ends of the VPN connection. The L2TP/IPSec feature supports gateway-to-gateway VPN implementations by using a preshared key for Internet Key Exchange (IKE) authentication.
Note Microsoft does not support or recommend the use of a preshared key for IKE authentication on remote access L2TP/IPSec client connections. However, Windows 2000 is compliant with IKE Request for Comments (RFC) 2409.
To implement the preshared key authentication method for use with an L2TP/IPSec connection, follow these steps:
Add the ProhibitIpSec registry value to both Windows 2000 Server-based endpoint computers, and then restart both computers.
Manually configure an IPSec policy on both the Windows 2000 Server-based computers before you try to establish an L2TP/IPSec connection between them through a Terminal Services client session.
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section of this article.
For additional information about configuring L2TP, click the following article numbers to view the articles in the Microsoft Knowledge Base:
240262 How to configure a L2TP/IPSec connection using pre-shared key authentication
248711 Mutual authentication methods supported for L2TP/IPSec