Select the product you need help with

How to help protect SMTP communication by using the Transport Layer Security protocol in Exchange Server

Article ID: 829721 - View products that this article applies to.

SUMMARY

This article contains information about how to enhance the security of Simple Mail Transfer Protocol (SMTP) communication in Microsoft Exchange Server 2003 and in Microsoft Exchange 2000 Server by using the Transport Layer Security (TLS) protocol.

The use of the Transport Layer Security (TLS) protocol over SMTP offers certificate-based authentication and helps provide security-enhanced data transfers by using symmetric encryption keys. In symmetric-key encryption (also known as shared secret), the same key is used to encrypt and to decrypt the message. TLS applies a Hash-based Message Authentication Code (HMAC). HMAC uses a hash algorithm in combination with a shared secret key to help make sure that the data has not been modified during transmission. The shared secret key is appended to the data to be hashed. This helps enhance the security of the hash because both parties must have the same shared secret key to verify that the data is authentic.

An X.509 server certificate is a digital form of identification that is typically issued by a certification authority (CA) and contains identification information, a validity period, a public key, a serial number, and the digital signature of the issuer. You can help protect communication by increasing the encryption level of the key pair from 40 bits (the default) to 128 bits. The greater the number of bits, the more difficult the item is to decrypt. Because of export restrictions, the 128-bit key strength encryption feature is available only in the United States and Canada.

For more detailed information, visit the following Internet Engineering Task Force (IETF) Web sites and view the following Requests for Comments (RFC):

RFC 2246: "The TLS Protocol Version 1.0"
http://www.ietf.org/rfc/rfc2246.txt
(http://www.ietf.org/rfc/rfc2246.txt)
RFC 2104: "HMAC: Keyed-Hashing for Message Authentication"
http://www.ietf.org/rfc/rfc2104.txt
(http://www.ietf.org/rfc/rfc2104.txt)

When you configure your virtual servers to require basic authentication, it is strongly recommended that you also use TLS encryption. Without encryption, user names and passwords can be easily intercepted. Users who try to obtain access must use the same encryption level that you set; otherwise, messages are returned and a non-delivery report (NDR) is generated.

TLS is designed to help protect outgoing messages, but TLS does not help protect traffic that travels from clients to the server. These clients include Microsoft Outlook Web Access (OWA), POP3, and IMAP4 in particular. To fix this problem, you can enable the use of Secure Sockets Layer (SSL) with Outlook Web Access. You can also suggest that POP3 or IMAP4 users use a client that supports the use of SSL with POP3 and IMAP4 (for example, Microsoft Outlook Express).

How to Require Transport Layer Security Encryption for Clients

To require TLS encryption for clients, follow these steps:

Create and manage key certificates. To do so, follow these steps:
1. Install an X.509 server certificate on the server. For more information about X.509 certificates, click the following article number to view the article in the Microsoft Knowledge Base:
  319574
  (http://support.microsoft.com/kb/319574/ )
  How to use certificates with virtual servers in Exchange 2000 Server
2. Start Exchange System Manager.
3. Expand the Exchange Server, click Protocols, click SMTP, right-click the SMTP Virtual Server, and then click Properties.
4. Click the Access tab, and then click Certificate to set up new key certificates and to manage key certificates that are installed for the SMTP virtual server.
Set TLS encryption levels for the server. To do so, follow these steps:
1. Start Exchange System Manager.
2. Right-click the SMTP Virtual Server, and then click Properties.
3. Click the Access tab, and then click Authentication.
4. Click to select the Basic Authentication check box, click to select the Requires TLS encryption check box, and then click OK.

Enable Transport Layer Security Encryption for a Specific Remote Domain in an Exchange Organization

To enable TLS encryption for a specific remote domain in Exchange Server, follow these steps:

Install an X.509 server certificate on the server. For more information about X.509 certificates, click the following article number to view the article in the Microsoft Knowledge Base:
319574
(http://support.microsoft.com/kb/319574/ )
How to use certificates with virtual servers in Exchange 2000 Server
Create a new SMTP Connector. For more information about how to create a new SMTP Connector, click the following article number to view the article in the Microsoft Knowledge Base:
314961
(http://support.microsoft.com/kb/314961/ )
How to install and to configure SMTP Connectors in Exchange 2000 Server
To enable TLS encryption, right-click the SMTP connector, and then click Properties. Click the Advanced tab, click Outbound Security, and then click to select the TLS Encryption check box.

Note If the remote domain does not support TLS encryption, all messages are returned and an NDR is generated. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

329061

(http://support.microsoft.com/kb/329061/ )

Exchange Server cannot communicate with non-TLS domains

Enable Transport Layer Security Encryption for All Outgoing SMTP Connections in Exchange Server

To enable TLS encryption for all outgoing SMTP connections, follow these steps:

Install an X.509 Server Certificate on the server. For more information about X.509 certificates, click the following article number to view the article in the Microsoft Knowledge Base:
319574
(http://support.microsoft.com/kb/319574/ )
How to use certificates with virtual servers in Exchange 2000 Server
Start Exchange System Manager.
Right-click the SMTP Virtual Server, and then click Properties.
On the Delivery tab of the SMTP virtual server, click Outbound Security, and then click to select the TLS Encryption check box.