How an IIS Web server and a Secure Socket Tunneling Protocol (SSTP)-based Routing and Remote Access server can co-exist on a Windows Server 2008-based server

This article describes the following topics:

• How an IIS Web server and a Secure Socket Tunneling Protocol (SSTP)-based Routing and Remote Access server can co-exist on a Windows Server 2008-based server
• The different scenarios in which both IIS and SSTP can co-exist on the same Windows Server 2008-based server
• How to modify the certificate hash value of the SSTP-based Routing and Remote Access server certificate so that VPN clients can use the IIS Web server certificate to establish VPN connections

Secure Socket Tunneling Protocol (SSTP) is a VPN tunnel that is added to the Routing and Remote Access service in Windows Server 2008 and in Windows Vista Service Pack 1 (SP1). SSTP enables Point-to-Point Protocol (PPP) packets to be encapsulated over HTTP. When encapsulated PPP packets over SSTP are enabled, VPN connections can be established through Web proxies. Additionally, VPN connections can be established more easily through firewalls and through Network Address Translation (NAT) devices.

In a scenario where you have configured an IIS Web server and a SSTP-based Routing and Remote Access server on the same Windows Server 2008-based server, you may notice that VPN connections to the SSTP-based Routing and Remote Access server cannot be established. The reason is that both the IIS Web server and the SSTP-based Routing and Remote Access server use different certificates to establish HTTPS connections. The following two important procedures must occur to enable both an IIS Web server and a SSTP-based Routing and Remote Access server to co-exist on the same Windows Server 2008-based server:

• Session demultiplexing
• Certificate selection

The following four important components are required where you have configured an IIS Web server and a SSTP-based Routing and Remote Access server on the same Windows Server 2008-based server:

• A computer certificate that is installed for the computer account in the local computer certificate store.
• The HTTPS Listener (HTTP.sys) component that closes HTTPS connections and that uses a certificate to establish HTTPS connections.
• The Routing and Remote Access server that depends on the HTTPS Listener component. The Routing and Remote Access server acts as an endpoint for VPN connections. The Routing and Remote Access server also uses the certificate hash of the computer certificate for its crypto-binding validation phase.
• The IIS Web server that depends on the HTTPS listener component. The IIS Web server acts as an endpoint for all the HTTPS connections for all Uniform Resource Identifiers (URI) except for the SSTP based VPN connections that are made to the Routing and Remote Access server.

You must make sure that all four components are working correctly before you can configure both the IIS Web server and the SSTP-based Routing and Remote Access server to co-exist on the same Windows Server 2008-based server.

Session demultiplexing

Session demultiplexing is the process where it is decided which HTTPS connection request is made to the Routing and Remote Access server and which HTTPS connection request is made to the IIS Web server. The SSTP- based Routing and Remote Access server listens for the HTTPS request that is made by a VPN client on a fixed URI. If you have configured IIS on the same server, IIS communicates to the HTTPS Listener, and IIS retrieves requests from rest of the URIs.

Certification selection

When a certificate selection occurs, the selection is based on the following scenarios:

• Scenario 1
  
  You have configured IIS first, and you have issued a certificate to the HTTPS Listener component for the 0.0.0.0:443 IPv4 listener. Later, you configure SSTP. In this scenario, the SSTP-based Routing and Remote Access server finds that a certificate has already been issued to HTTPS Listener for 0.0.0.0:443. Then, the SSTP-based Routing and Remote Access server issues the same certificate to the HTTPS Listener component for 0.0.0.0:443.
• Scenario 2
  
  You have configured IIS first, and you have issued a certificate to the HTTPS Listener component for a specific IPv4 address and port. For example, you have issued a certificate to an HTTPS site for the 1.1.1.1:443 listener. Later, you configure the Routing and Remote Access service.
  
  In Scenario 2, the SSTP based Routing and Remote Access server does not find that a certificate has already been issued to the HTTPS Listener component for the 0.0.0.0:443 and the [::]:443 listeners. Therefore, the SSTP- based Routing and Remote Access server searches all the available certificates in the local computer certificate store. Then, the SSTP- based Routing and Remote Access server issues a certificate that has the Server Authentication or All purpose Enhanced key usage (EKU) value and sends the value to the 0.0.0.0:443 and [::]:443 listeners.
• Scenario 3
  
  You have configured SSTP first, and you have issued a certificate to the HTTPS Listener component. Later, you configure an HTTPS site by using the IIS Web server on the same Windows Server 2008-based server. Then, you configure the IIS Web server to issue the same SSTP-based Routing and Remote Access server certificate to the HTTPS site for the 0.0.0.0:443 listener.
• Scenario 4
  
  You have configured SSTP first, and you have issued a certificate to the HTTPS Listener component. Later, you configure an HTTPS site by using the IIS Web server on the same Windows Server 2008-based server. Then, you configure the IIS Web server to issue a certificate that differs from the SSTP-based Routing and Remote Access server certificate to the HTTPS site for a specific or generic IPv4 address and port. For example, you have issued a certificate to an HTTPS site for the 1.1.1.1:443 or for the 0.0.0.0:443 listener.

Scenario 1

In Scenario 1, a VPN client tries to establish a connection to the STTP-based Routing and Remote Access server by using an IPv4 or and IPv6 address. However, the connection is not established. This is because the IIS Web server can only bind the certificate to the 0.0.0.0:443 listener. When you configure SSTP after you configure the IIS Web server, SSTP cannot set up the [::]:443 listener. Additionally, SSTP cannot configure the Sha256 hash in the registry. In this scenario, the following error message is logged in the System event log of the VPN client:

The Secure Socket Tunneling Protocol service either could not read the SHA256 certificate hash from the registry or the data is invalid. To be valid, the SHA256 certificate hash must be of type REG_BINARY and 32 bytes in length. SSTP might not be able to retrieve the value from the registry due to some other system failure. The detailed error message is provided below. SSTP connections will not be accepted on this server. Correct the problem and try again.

The system cannot find the file specified.

To work around this issue, follow these steps:

1. Issue the same certificate that is used by the IIS Web server to the [::]:443 listener.
2. Configure the Sha256 hash in the registry.
3. Restart the SSTP and Routing and Remote Access services.

For more information, see the "How to modify the certificate hash value" section.

Scenario 2

In Scenario 2, the listener is set up successfully with the Server Authentication or the All purpose certificate that is issued for 0.0.0.0:443 and for [::]:443. The Sha256 hash is also added in the registry for the SSTP-based Routing and Remote Access server when the IIS Web server is binding the certificate to a specific IP address and port combination. In Scenario 2, the following two sub-scenarios are possible:

• The certificate that is issued for the specific IP address and port combination by the IIS Web server is same as the certificate that is issued for 0.0.0.0:443 and for [::]:443 by the SSTP-based Routing and Remote Access server. In this case, all the SSTP-based VPN connections to this VPN server are successful.
• The certificate that is issued for the specific IP address and port combination by the IIS Web server differs from the certificate that is issued for 0.0.0.0:443 and for [::]:443 by the SSTP-based Routing and Remote Access server. In this case, a VPN connection is not successful if the connection is made by using this IPv4 address. The VPN connection is not successful because the SSTP-based VPN connection establishing process is supposed to use the hash value that is registered in the registry for the certificate that is issued to the 0.0.0.0:443 listener to establish the VPN connection. But the certificate that is used during the SSL phase is the same certificate that is issued for the specific IP address and port combination by the IIS Web server. Therefore, there is a mismatch in the hash values of the certificates that are used for the SSTP-based VPN connection process, and the VPN connection cannot be established. To work around this issue, you can replace the certificate hash value of the SSTP-based Routing and Remote Access server certificate with the certificate hash value of the IIS Web server certificate. For more information, see the "How to modify the certificate hash value" section.

Scenario 3

In Scenario 3, the certificate that is used by the SSTP-based Routing and Remote Access server and by the IIS Web server is same. In this scenario, a VPN client can successfully establish a connection with the SSTP-based Routing and Remote Access server. For example, if the SSTP-based Routing and Remote Access server uses a certificate named Cert 1, and the IIS Web server also uses the same certificate that is called Cert 1, VPN connections are successful.

Scenario 4

In Scenario 4, the certificate that the SSTP-based Routing and Remote Access server uses differs from the certificate that the IIS Web server uses. If a VPN client tries to connect from an IP address that is specified in the IIS Web server’s certificate, the client receives the certificate that the IIS Web server issued. This occurs during the SSL hand-shake phase of the SSTP connection process. Then, the client tries to perform crypto-binding validation for this certificate during the PPP authentication phase. But the SSTP-based Routing and Remote Access server expects the crypto-binding validation to be performed on the certificate that the SSTP- based Routing and Remote Access server uses. In this scenario, the VPN client cannot establish a VPN connection with the SSTP-based Routing and Remote Access server.

For example, assume that the following conditions are true:

• The SSTP-based Routing and Remote Access server uses a certificate that is named Cert 2.
• The IIS Web server uses a certificate that is named Cert 1.
• A VPN client uses the 1.1.1.1 or 3000::1 IP address to connect to the SSTP-based Routing and Remote Access server.
• The HTTP Listener uses Cert 1 for whichever IP address is used in the previous bullet point example.

In this case, the VPN client receives Cert 1 during the SSL hand-shake phase because the IP address of the VPN client has a match with the IP addresses that are specified for the IIS Web server certificate.

The client then computes the crypto-binding based on Cert 1 during the PPP authentication phase. But the SSTP-based Routing and Remote Access server is expecting crypto-binding that is based on Cert 2. At this point, the VPN client is unable to establish a VPN connection with the SSTP-based Routing and Remote Access server because of the differences in the certificates that are used.

Additionally, an error message that resembles the following is logged in the System event log of the VPN client:

The SSTP-based VPN connection to the remote access server was terminated because of a security check failure. Security settings on the remote access server do not match settings on this computer. Contact the system administrator of the remote access server and relay the following information:

SHA1 Certificate Hash: 0705ae2bac92bcbcbc54a11fc8c527dc0ecaf5ee

SHA256 Certificate Hash: d075f96f979fd4df20f3fdf7a5335807879ca627e5f3fc0bab7a7ac067c831c6

To work around the issue that is described in Scenario 4, you can replace the certificate hash value of the SSTP-based Routing and Remote Access server certificate with the certificate hash value of the IIS Web server certificate. For more information, see the "How to modify the certificate hash value" section.

How to modify the certificate hash value

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

To issue the IIS Web server certificate to the [::]:443 listener and to configure the Sha256 hash in the registry, follow these steps:

1. Obtain the Sha256 hash value for the IIS Web server certificate.
2. Open an elevated command prompt on the VPN server.
3. At the command prompt, type the following command, and then press ENTER to issue the IIS Web server certificate to the [::]:443 listener:
   netsh http add sslcert ipport=[::]:443 certhash= Sha256 hash value for the IIS Web server certificate appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY
4. At the command prompt, type the following command, and then press ENTER to configure the Sha256CertificateHash registry key value for the SSTP service:
   reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SstpSvc\Parameters /v SHA256CertificateHash /t REG_BINARY /d Sha256 hash value for the IIS Web server certificate /f
5. At the command prompt, type the following commands one at a time, and then press ENTER to restart the Routing and Remote Access service:
   net stop sstpsvc /y
   net start remoteaccess
6. At the command prompt, type exit, and then press ENTER to close the command prompt.

Now, VPN clients can establish VPN connections to the SSTP-based Routing and Remote Access server by using the same certificate.

To replace the certificate hash value of the SSTP-based Routing and Remote Access server certificate with the certificate hash value of the IIS Web server certificate, follow these steps:

1. Obtain the Sha256 hash value for the IIS Web server certificate. You can obtain the hash value from the error message that is logged in the System event log of the VPN client. For example, the Sha256 hash value may resemble the following hash value:
   d075f96f979fd4df20f3fdf7a5335807879ca627e5f3fc0bab7a7ac067c831c6
2. Open an elevated command prompt on the VPN server.
3. At the command prompt, type the following command, and then press ENTER to configure the Sha256CertificateHash registry key value for the SSTP service:
   reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SstpSvc\Parameters /v SHA256CertificateHash /t REG_BINARY /d d075f96f979fd4df20f3fdf7a5335807879ca627e5f3fc0bab7a7ac067c831c6 /f
4. At the command prompt, type the following commands one at a time, and then press ENTER to restart the Routing and Remote Access service:
   net stop sstpsvc /y
   net start remoteaccess
5. At the command prompt, type exit, and then press ENTER to close the command prompt.

Now, VPN clients can establish VPN connections to the SSTP-based Routing and Remote Access server by using the IIS Web server certificate.