How to configure a Windows Server 2003 terminal server to use TLS for server authentication

Article translations Article translations
Article ID: 895433 - View products that this article applies to.
Expand all | Collapse all

On This Page

SUMMARY

You can connect to a remote computer that is running Microsoft Windows Terminal Services by using a Remote Desktop Protocol connection. This kind of connection provides encryption for the data that is sent between the terminal server and the client computer. However, this kind of connection does not provide authentication for the terminal server. You may want to make sure that your terminal server is correctly authenticated before you connect to it. To do this, configure your terminal server to use Transport Layer Security (TLS) to authenticate the terminal server and to encrypt the data that is sent between the terminal server and the client computer.

To configure a TLS connection, you must configure both the terminal server and the client computer. To configure the terminal server, you must perform both the following steps:
  • You must install a valid certificate on the terminal server.
  • You must configure the authentication settings by using the Terminal Services Configuration tool.
To configure the client computer, you must perform both the following steps:
  • You must configure the client computer to trust the root Certification Authority that issued the terminal server's certificate.
  • You must configure the authentication settings for the remote connection by using the Remote Desktop Connection program or by modifying the registry.

INTRODUCTION

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 How to back up and restore the registry in Windows


If you use the Remote Desktop Protocol (RDP) to connect to a terminal server, RDP provides data encryption but it does not provide authentication. Therefore, you cannot verify the identity of the terminal server. You can use Microsoft Windows Server 2003 Service Pack 1 (SP1) together with Transport Layer Security (TLS) version 1.0 to help increase terminal server security by using TLS for server authentication and to encrypt terminal server communications.

This article describes how to configure Windows Server 2003 SP1 to use TLS 1.0 for server authentication to encrypt terminal server communications.

Prerequisites to configure server authentication

By default, Terminal Server uses native RDP encryption and does not authenticate the server. To use TLS for server authentication and to encrypt terminal server communications, you must configure both the server computer and the client computer correctly.

Server prerequisites

For TLS authentication to work correctly, your terminal server must meet both the following requirements:
  • Your terminal server must be running Windows Server 2003 SP1.
  • You must obtain a certificate for your terminal server. To obtain a certificate, use one of the following methods:
    • Visit the Web site for your certification authority. For example, visit http://servername/certsrv.
    • Run the Windows Server 2003 Certificate Request Wizard or the Windows 2000 Server Certificate Request Wizard.
    • Obtain a certificate from a third-party certification authority, and then manually install the certificate.
Note If you want to obtain a certificate by using the Microsoft Certificate Services Web page, or by using the Certificate Request Wizard, a public key infrastructure (PKI) must be configured correctly to issue SSL-compatible X.509 certificates to the terminal server. Each certificate must be configured as follows:
  • The certificate must be a computer certificate.
  • The intended purpose of the certificate must be for server authentication.
  • The certificate must have a corresponding private key.
  • The certificate must be stored in the computer account certificate store on the terminal server.

    Note You can view this store by using the Microsoft Management Console (MMC) Certificates snap-in.
  • The certificate must have a cryptographic service provider (CSP) that can be used for the TLS protocol. For example, the certificate must use a cryptographic service provider such as the Microsoft RSA SChannel Cryptographic Provider. For more information about Microsoft cryptographic service providers, visit the following Microsoft Web site:
    http://msdn2.microsoft.com/en-us/library/aa386983.aspx

Client prerequisites

For TLS authentication to work correctly, your Terminal Services client computer must meet the following requirements:
  • The client computer must be running Microsoft Windows 2000 or Microsoft Windows XP.
  • The client computer must be upgraded to use the RDP 5.2 client program. The RDP 5.2 client program is included with Windows Server 2003 SP1. You can install this client-side Remote Desktop Connection package by using the %SYSTEMROOT%\System32\Clients\Tsclient\Win32\Msrdpcli.msi file. The Msrdpcli.msi file is located on Windows Server 2003-based terminal servers. If you install this file from the terminal server, the RDP 5.2 version of Remote Desktop Connection is installed in the %SYSTEMDRIVE%\Program files\Remote Desktop folder on the destination computer. For more information about the Remote Desktop Connection for Windows Server 2003, visit the following Microsoft Web site:
    http://www.microsoft.com/downloads/details.aspx?familyid=CC148041-577F-4201-B62C-D71ADC98ADB1&displaylang=en
  • The client computer must trust the root Certification Authority of your terminal server's certificate. Therefore, the client computer must have the certificate of the Certification Authority in the Trusted Root Certificate Certification Authorities folder of the client computer. You can view this folder by using the Certificates snap-in.

To configure the terminal server

To configure your terminal server for TLS authentication, follow these steps:

Step 1: Request a computer certificate

If you do not already have a computer certificate that meets the requirements that are mentioned in the "Prerequisites to configure server authentication" section, obtain and install one. To do this, use one of the following methods.

Method 1: By using the Web site for your certification authority
The following steps describe how to obtain a certificate from a Windows Server 2003 stand-alone Certification Authority. You can also request a certificate from a Windows 2000 Certification Authority. Additionally, you must have Read permissions and Enroll permissions on the certificate template file to successfully request a certificate. Use this method if one or more of the following conditions are true:
  • You want to obtain a certificate from a stand-alone certification authority.
  • You want to obtain a certificate that is based on a certificate template that is configured to obtain the subject name from the subject.
  • You want to obtain a certificate that requires administrator approval before the certificate is issued.
To obtain a certificate, follow these steps:
  1. Start Microsoft Internet Explorer, and then visit http://servername/certsrv, where servername is the name of your server that is running Microsoft Certificate Services.
  2. Under Select a task, click Request a certificate.
  3. Click advanced certificate request, and then click Create and submit a request to this CA.
  4. Type your identifying information in the boxes under Identifying Information, and then click Server Authentication Certificate in the Type of Certificate Needed list.
  5. Leave the Create new key set option selected, and then click Microsoft RSA SChannnel Cryptographic Provider in the CSP list.

    Note This cryptographic service provider supports key derivation for the SSL2, PCT1, SSL3, and TLS1 protocols.
  6. Leave the Exchange option selected next to Key Usage. This option indicates that the private key can be used to enable the exchange of sensitive information.
  7. Click to select the Mark keys as exportable check box. When you do this, you can save the public key and the private key to a PKCS#12 file. Therefore, you can copy this certificate to another computer.
  8. Click to select the Store certificate in the local computer certificate store check box, and then click Submit.

    Important For TLS authentication to function, you must store the certificate in the local computer certificate store.
  9. If you receive a Certificate Issued Web page, click Install this certificate. If you receive a Certificate Pending Web page, you must wait until an administrator approves the certificate request. In this scenario, you must again visit the Certificate Services Web site to obtain and install this certificate.
Method 2: By using the Certificate Request Wizard
The following steps describe how to obtain a certificate from a Windows Server 2003 Certification Authority. You can also request a certificate from a Windows 2000 Certification Authority. Additionally, you must have Read permissions and Enroll permissions on the certificate template file to successfully request a certificate. Use this method if one or more of the following conditions are true:
  • You want to request a certificate from an Enterprise Certification Authority.
  • You want to request a certificate that is based on a template where the subject name is generated by Windows.
  • You want to obtain a certificate that does not require administrator approval before the certificate is issued.
To obtain a certificate, follow these steps:
  1. Click Start, click Run, type mmc, and then click OK.
  2. On the File menu, click Add/Remove Snap-in.
  3. Click Add, click Certificates, and then click Add.
  4. Click Computer account, and then click Next.
  5. If you want to add a certificate to the local computer, click Local computer. If you want to add a certificate to a remote computer, click Another computer, and then type the name of that remote computer in the Another computer box.
  6. Click Finish.
  7. In the Add Standalone Snap-in dialog box, click Close, and then click OK in the Add/Remove Snap-in dialog box.
  8. Under Console Root, click Certificates (Local Computer).

    Note If you configured the Certificates MMC snap-in to manage a remote computer, click Certificates (servername) instead of Certificates (Local Computer).
  9. On the View menu, click Options.
  10. In the View Options dialog box, click Certificate purpose, and then click OK.
  11. In the right pane, right-click Server Authentication, point to All Tasks, and then click Request New Certificate.
  12. In the Certificate Request Wizard that starts, click Next.
  13. In the Certificate types list, click Server Authentication, click to select the Advanced check box, and then click Next.
  14. In the Cryptographic Service Providers list, click Microsoft RSA SChannel Cryptographic Provider.

    Note This cryptographic service provider supports key derivation for the SSL2, PCT1, SSL3, and TLS1 protocols.
  15. In the Key Length list, leave the default option of 1024 selected or click the key length that you want to use.
  16. Click to select the Mark this key as exportable check box. When you do this, you can save the public key and the private key to a PKCS#12 file. Therefore, you can copy this certificate to another computer.
  17. If you want to enable "strong private key protection," click to select the Enable strong private key protection check box.
  18. Click Next, type the name of your Certification Authority in the CA box, click Next, type a name for this certificate in the Friendly name box, click Next, and then click Finish.
Method 3: By using a third-party certification authority
Obtain and install a certificate from a third-party certification authority.

Step 2: Configure TLS authentication and encryption

You can configure encryption settings on the terminal server by using Group Policy. However, you cannot use Group Policy to configure authentication settings on the terminal server. Therefore, this section describes how to configure authentication and encryption by using the Terminal Services Configuration tool. For TLS to function correctly on a terminal server, you must configure all the following items on the General tab of the RDP-Tcp Properties dialog box:
  • You must select a certificate that meets the requirements that are mentioned in the "Server prerequisites" section.
  • You must set the Security layer value to Negotiate or to SSL.
  • You must set the Encryption level value to High, or you must enable Federal Information Processing Standard (FIPS)-compliant encryption.

    Note You can also enable FIPS-compliant encryption by using Group Policy. However, you cannot enable TLS by using Group Policy.
Note If you enable TLS authentication in a session directory farm, you must configure one of the following settings on each one of the servers that are members of the session directory farm:
  • Set the Security layer value to SSL.
  • Set the Security layer value to Negotiate. If you set the Security layer to Negotiate, TLS authentication is only enabled if the client computer supports TLS authentication.
To configure TLS authentication and encryption on the server, follow these steps:
  1. Start the Terminal Services Configuration tool. To do this, click Start, point to Administrative Tools, and then click Terminal Services Configuration.
  2. In the left pane, click Connections.
  3. In the right pane, right-click the connection that you want to configure, and then click Properties.
  4. On the General tab, click Edit next to Certificate.
  5. In the Select Certificate dialog box, click the certificate that you want to use.

    NoteServer Authentication must appear in the Intended Purpose column for this certificate. Additionally, this certificate must be an X.509 certificate with a corresponding private key. To determine whether the certificate has a private key, click View Certificate. The following message text appears at the bottom of the certificate information:
    You have a private key that corresponds to this certificate.
    Click OK.
  6. Click OK.
  7. In the Security layer list, click one of the following options:
    • Negotiate: This security method uses TLS 1.0 to authenticate the server if TLS is supported. If TLS is not supported, the server is not authenticated.
    • RDP Security Layer: This security method uses Remote Desktop Protocol encryption to help secure communications between the client computer and the server. If you select this setting, the server is not authenticated.
    • SSL: This security method requires TLS 1.0 to authenticate the server. If TLS is not supported, you cannot establish a connection to the server. This method is only available if you select a valid certificate.
    Note If you click Negotiate or SSL in the Security layer list, you must also configure one of the following:
    • Set the encryption level to High.
    • Configure FIPS-compliant encryption.
  8. In the Encryption level list, click one of the following options:
    • FIPS Compliant: If you use this setting, or if you set the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing option by using Group Policy, data is encrypted and decrypted between the client computer and the server that has FIPS 140-1 encryption algorithms by using Microsoft cryptographic modules.
    • High If you use this setting, data that is sent between the client computer and the server is encrypted by using 128-bit encryption.
    • Client Compatible If you use this setting, data that is sent between the client computer and then server is encrypted by using the maximum key strength that is supported by the client computer.
    • Low If you use this setting, data that is sent between the client computer and the server is encrypted by using 56-bit encryption.

      Note This option is not available when you click SSL in the Security layer list.
  9. Click to select the Use standard Windows logon interface check box to specify that users log on to the terminal server by typing their credentials in the default Windows logon dialog box.
  10. Click OK.
Note
  • To configure these options, you must be a member of the Administrators group on the local computer or you must be delegated the appropriate rights. If the computer is joined to a domain, members of the Domain Admins security group have sufficient permissions to follow these steps.
  • Encryption levels that you configure by using Group Policy override the configuration options that you set by using the Terminal Services Configuration tool. Additionally, if you enable the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing policy, the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing policy overrides the Set client connection encryption level Group Policy setting.
  • When you change the encryption level, the new encryption level that you configure takes effect the next time a user logs on. If you require multiple encryption levels, install multiple network adaptors, and then configure each network adaptor with a different encryption level.

To configure the client computer

To configure the client computer for TLS authentication, follow these steps:

Step 1: Request a computer certificate

  1. Start Internet Explorer, and then visit http://servername/certsrv.
  2. Click Download a CA certificate, certificate chain, or CRL.
  3. Click install this CA certificate chain to configure your client computer to trust all the certificates that are issued by this certification authority.
  4. Click Yes if you are prompted to add the certificates from the Certification Authority Web site.
  5. After you receive the following message, quit Internet Explorer:
    The CA certificate chain has been successfully installed.
Note
  • You do not have to be logged on to the computer that has administrative privileges to perform this operation.
  • You install the CA certificate chain to make sure that the client computers trust the root of the terminal server's certificate. This means that the certificate of the root CA that issued the terminal server's certificate is stored in the client computer's local computer Trusted Root Certification Authorities certificate store. This is required for TLS to be used for server authentication when client computers connect to the terminal server.
  • You can use the install this CA certificate chain option to establish trust in a subordinate certification authority if you do not currently have the certificate of the root CA in your certificate store.

Step 2: Configure authentication on the client computer

To configure authentication on the client computer, use one of the following methods.
Method 1: By using Remote Desktop Connection
  1. Start Remote Desktop Connection.
  2. Click Options, and then click the Security tab.

    Note The Security tab appears if you install the Windows Server 2003 SP1 version of Remote Desktop Connection.
  3. In the Authentication list, click one of the following options:
    • No authentication: This is the default option. If you select this option, the terminal server is not authenticated.
    • Attempt authentication: If you select this option, and if TLS is supported and correctly configured, TLS 1.0 is used to authenticate the terminal server.

      If you click Attempt authentication, you can choose to continue your Terminal Services connection without TLS authentication if one of the following authentication errors occur:
      • The server certificate is expired.
      • The server certificate is not issued by a trusted root Certification Authority.
      • The name in the certificate does not match the name of the client computer.
      Other authentication errors cause the Terminal Services connection to fail.
  4. Require authentication: If you click this option, TLS is required to authenticate the terminal server. If TLS is not supported, or if TLS is not correctly configured, the connection attempt is not successful. This option is only available for client computers that connect to terminal servers that are running Windows Server 2003 SP1.
Note You do not have to have administrative rights to configure Remote Desktop Connection. Additionally, after you configure this connection, you can save your changes as a Remote Desktop file (.rdp). To configure other client computers to use the security settings that you have configured, distribute the .rdp file to those computers.

An .rdp file contains all the information for the connection to the terminal server. This includes the security settings that you configure on the Security tab. You can customize your connections to a particular terminal server by creating different .rdp files that correspond to the settings that you want to use when you connect to that terminal server. Additionally, you can change the .rdp file by using any text editor, such as Notepad. To modify the security settings of an .rdp file by using Notepad, follow these steps:
  1. Locate the .rdp file that you want to modify, and then open it by using Notepad.
  2. Locate the authentication level line in the RDP file.
  3. Set the authentication level value to one of the following values:
    • 0 This value corresponds to "No authentication."
    • 1 This value corresponds to "Require authentication."
    • 2 This value corresponds to "Attempt authentication."
    For example, to configure the Remote Desktop Connection to require authentication, type authentication level:i:1.
  4. Save the changes to the file, and then quit Notepad.
Method 2: By using Registry Editor
  1. Click Start, click Run, type regedit, and then click OK.
  2. Use one of the following methods:
    • To modify the registry settings for all the users who log on to the computer, locate and then click the following registry subkey:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Terminal Server Client
    • To modify the registry settings for only the currently logged on user, locate and then click the following registry subkey:
      HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client
  3. On the Edit menu, point to New, and then click DWORD Value.
  4. In the New Value #1 box, type AuthenticationLevelOverride, and then press ENTER.
  5. Right-click AuthenticationLevelOverride, and then click Modify.
  6. In the Value data box, type one of the following values, and then click OK:
    • 0 Type this value to configure an authentication level of "No authentication."
    • 1 Type this value to configure an authentication level of "Require authentication."
    • 2 Type this value to configure an authentication level of "Attempt authentication."
For additional information about these authentication levels, see the "Method 1: By using Remote Desktop Connection" section.

Note
  • If you configure the authentication level by using the registry, users who are logged on to the client computer cannot modify the authentication settings.
  • The authentication level that you set by using the HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client registry subkey overrides an authentication level that might be configured in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Terminal Server Client registry subkey.

REFERENCES

For more information about related topics, visit the following Microsoft Web sites:
http://www.microsoft.com/windowsserver2003/technologies/terminalservices/default.mspx

http://technet2.microsoft.com/WindowsServer/en/Library/9d47b6a2-3216-45fc-9bb8-41a7d89e42d11033.mspx

http://technet2.microsoft.com/windowsserver/en/library/590FCC3E-C54F-48B7-95F2-45EE2255FC111033.mspx

http://technet2.microsoft.com/windowsserver/en/library/32AACFE8-83AF-4676-A45C-75483545A9781033.mspx

Properties

Article ID: 895433 - Last Review: January 12, 2007 - Revision: 5.4
APPLIES TO
  • Microsoft Windows Server 2003, Enterprise Edition (32-bit x86)
  • Microsoft Windows Server 2003, Standard Edition (32-bit x86)
  • Microsoft Windows Server 2003, Web Edition
  • Microsoft Windows Server 2003, Datacenter Edition (32-bit x86)
  • Microsoft Windows Server 2003, Datacenter Edition for Itanium-Based Systems
  • Microsoft Windows Server 2003, Enterprise Edition for Itanium-based Systems
Keywords: 
kbtermserv kbhowtomaster KB895433

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com