IPSec support for client-to-domain controller traffic and domain controller-to-domain controller traffic

This article describes the supported configurations for using Internet Protocol security (IPSec) to encrypt network traffic from a client computer to a domain controller or from a domain controller to another domain controller.

Important The information in this section applies only to those products listed in the "Applies to" section.

We support the use of IPSec to encrypt network traffic in end-to-end client-to-client, client-to-server, and server-to-server implementations when you use either Kerberos computer authentication or when you use certificate-based computer authentication. Currently, we do not support the use of IPSec to encrypt network traffic from a domain client or member server to a domain controller when you apply the IPSec policies by using Group Policy or when you use the Kerberos version 5 protocol authentication method.

Additionally, we support using IPSec to encrypt both the following kinds of network traffic:

• Domain controller-to-domain controller replication traffic
• Global catalog-to-global catalog replication traffic

To encrypt this traffic by using IPSec, configure both the following:

• Create an IPSec policy filter to encrypt all Unicast traffic by using the All IP Traffic option.
• Configure this IPSec policy filter to encrypt this traffic between two IP addresses by using IPSec Transport mode. In this scenario, do not use IPSec Tunnel mode.

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

After you configure this IPSec policy, you may notice that when the computers are started, several packets may be sent over the network unencrypted. This issue occurs because some packets might be sent over the network before the IPSec driver has been initialized and before the IPSec policy has been processed. To resolve this issue, put the IPSec driver IPSec.sys into Block Mode during the computer startup process. When you do this, IPSec blocks outgoing network traffic from the computer until the PolicyAgent component starts and until the PolicyAgent component loads the IPSec policies. After the IPSec PolicyAgent component has started, and after the IPSec policies are loaded, the PolicyAgent changes the IPSec driver's operation mode to permit the passage of IPSec traffic. To put the IPSec driver into Block Mode, set the following registry value: A value of 1 puts the IPSec driver into Block Mode. A value of 0 (zero) bypasses the IPSec driver's block mode.

For more information, click the following article number to view the article in the Microsoft Knowledge Base: We support using IPSec to encrypt domain controller-to-domain controller traffic such as Server Message Block (SMB), Remote Procedure Call (RPC) replication, and other kinds of traffic. You can transport this traffic by using IPSec to let you easily pass these kinds of traffic through a firewall. In this scenario, you only have to permit IPSec traffic and Internet Key Exchange (IKE) traffic through your firewall. For more information, click the following article number to view the article in the Microsoft Knowledge Base: We recommend that you require certificate-based authentication when you configure domain controller-to-domain controller IPSec policy rules. For detailed information about how to create an IPSec policy, see the Active Directory in Networks Segmented by Firewalls document. To obtain this document, visit the following Microsoft Web site: The rule must require certificate authentication if the security requirements do not allow Kerberos traffic through the firewall. By default, IKE certificate revocation checking is off, and may have to be enabled through the firewall. This depends on the PKI infrastructure that is being used.

Build the IPSec rule on the domain controllers by using the following specifications:

• The filter list specifies traffic going from the IP address on DC1 to the IP address on DC2 (mirrored), subnet masks 255.255.255.255, all protocols, and all ports. You may want to add a rule to the IPSec policy to exempt ICMP traffic from IPSec security negotiation if Ping is used to verify network connectivity to the remote system through the firewall. Otherwise, connectivity can be verified by a network sniff that shows IKE traffic (ISAKMP, UDP port 500) being sent and received from the DC to the other DC IP address.
  
  A network address translator (NAT) must not be used to change addresses or modify packets between the domain controllers that require IPSec protection between them.
  
  
• Under Tunnel Setting, click This rule does not specify an IPSec tunnel so that it uses Transport mode.
• Select Use Certificate for the authentication method. You can use Kerberos, see the following note.
• Create a custom filter action by clearing the Accept Unsecured Communication and Allow Unsecured Communication check boxes and specifying the appropriate data encryption method by using the ESP format of IPSec. Network adaptors that perform IPSec per-packet encryption in hardware are needed in each domain controller so that IPSec encryption does not consume all the computer's CPU cycles.

Note The initial release (build 2195) of Windows 2000 does not protect IKE, Kerberos, or RSVP traffic using IPSec transport filters. If Kerberos is used as the IPSec rule authentication method to protect domain controller-to-domain controller traffic instead of certificates, the firewall also must allow Kerberos traffic to go through. This must be a default setting. Windows 2000 Service Pack 1 provides IPSec with the capability of protecting Kerberos and RSVP traffic.