We support the use of IPSec to encrypt network traffic in end-to-end client-to-client, client-to-server, and server-to-server implementations when you use either Kerberos computer authentication or when you use certificate-based computer authentication. Currently, we do not support the use of IPSec to encrypt network traffic from a domain client or member server to a domain controller when you apply the IPSec policies by using Group Policy or when you use the Kerberos version 5 protocol authentication method.
Additionally, we support using IPSec to encrypt both the following kinds of network traffic:
- Domain controller-to-domain controller replication traffic
- Global catalog-to-global catalog replication traffic
- Create an IPSec policy filter to encrypt all Unicast traffic by using the All IP Traffic option.
- Configure this IPSec policy filter to encrypt this traffic between two IP addresses by using IPSec Transport mode. In this scenario, do not use IPSec Tunnel mode.
After you configure this IPSec policy, you may notice that when the computers are started, several packets may be sent over the network unencrypted. This issue occurs because some packets might be sent over the network before the IPSec driver has been initialized and before the IPSec policy has been processed. To resolve this issue, put the IPSec driver IPSec.sys into Block Mode during the computer startup process. When you do this, IPSec blocks outgoing network traffic from the computer until the PolicyAgent component starts and until the PolicyAgent component loads the IPSec policies. After the IPSec PolicyAgent component has started, and after the IPSec policies are loaded, the PolicyAgent changes the IPSec driver's operation mode to permit the passage of IPSec traffic. To put the IPSec driver into Block Mode, set the following registry value:
Value name: OperationMode
Value type: REG_DWORD
Value data: 1
For more information, click the following article number to view the article in the Microsoft Knowledge Base:
We recommend that you require certificate-based authentication when you configure domain controller-to-domain controller IPSec policy rules.
The rule must require certificate authentication if the security requirements do not allow Kerberos traffic through the firewall. By default, IKE certificate revocation checking is off, and may have to be enabled through the firewall. This depends on the PKI infrastructure that is being used.
Build the IPSec rule on the domain controllers by using the following specifications:
- The filter list specifies traffic going from the IP address on DC1 to the IP address on DC2 (mirrored), subnet masks 255.255.255.255, all protocols, and all ports. You may want to add a rule to the IPSec policy to exempt ICMP traffic from IPSec security negotiation if Ping is used to verify network connectivity to the remote system through the firewall. Otherwise, connectivity can be verified by a network sniff that shows IKE traffic (ISAKMP, UDP port 500) being sent and received from the DC to the other DC IP address.
A network address translator (NAT) must not be used to change addresses or modify packets between the domain controllers that require IPSec protection between them.
- Under Tunnel Setting, click This rule does not specify an IPSec tunnel so that it uses Transport mode.
- Select Use Certificate for the authentication method. You can use Kerberos, see the following note.
- Create a custom filter action by clearing the Accept Unsecured Communication and Allow Unsecured Communication check boxes and specifying the appropriate data encryption method by using the ESP format of IPSec. Network adaptors that perform IPSec per-packet encryption in hardware are needed in each domain controller so that IPSec encryption does not consume all the computer's CPU cycles.