TLS client authentication fails between Unified Communications peers with a logged Schannel warning

The following applications fail at creating a TLS connection with its Unified Communications (UC) peer:

• Microsoft Exchange Server
• Microsoft Office Communications Server
• Microsoft Communicator client
• Microsoft Office Live Meeting 2007 client
• Microsoft Lync Server
• Microsoft Lync client 

Additionally, the following Warning event that describes the issue is displayed in the Windows Server event log:

Event Type: Warning
Event Source: Schannel
Event Category: None
Event ID: 36885
Date: date
Time: time
User:
Computer: COMPUTERNAME

Description: When asking for client authentication, this server sends a list of trusted certification authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certification authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certification authorities trusted for client authentication and remove those that do not really need to be trusted.

The UC server does not pass the correct certification authority information back to the UC client during the negotiation of the TLS connection.

• The UC server passes a list of installed certification authority information to the UC client that requests the secure TLS connection.
• This list is truncated as per the design limitations of the Windows Server Schannel component.
• The UC client that requested the secure TLS connection does not receive certification authority information that matches the entries in its installed certification authority list.
• The TLS connection attempt fails with the error that is listed in the "Symptoms" section.

This issue occurs because of the design of the Schannel component of the Windows Server operating system that is hosting the UC applications.

The methods that are listed here provide a workaround for the issue that is described in the "Symptoms" section.

Method 1: Remove some trusted root certificates

If some trusted root certificates are not used in your environment, you should remove them from the server that is hosting the UC application. To do this, follow these steps.

Note The steps that are listed here can be performed in Windows Server 2003 and in Windows Server 2008.

1. Click Start, click Run, type mmc, and then click OK.
2. On the File menu, click Add/Remove Snap-in, and then click Add.
3. In the Add Standalone Snap-in dialog box, click Certificates, and then click Add.
4. Click Computer account, click Next, and then click Finish.
5. Click Close, and then click OK.
6. Under Console Root in the Microsoft Management Console (MMC) snap-in, expand Certificates (Local Computer), expand Trusted Root Certification Authorities, and then click Certificates.
7. Remove trusted root certificates that you do not have to have. To do this, right-click a certificate, click Delete, and then click Yes to confirm the removal of the certificate.

Note There are some root certificates that are required by Windows. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

Method 2: Configure Group Policy to ignore the list of trusted certification authorities on the computer that hosts the UC client

If the server that hosts the UC application is a member of a domain, you can create a policy that causes the server to ignore the list of trusted certification authorities on the computer that hosts the UC client. When you apply this policy, affected servers and clients trust only certificates that are in the Enterprise Root Certification Authorities store. Therefore, you do not have to modify individual computers.

Note The information that is listed in steps 1 and 2 is available only for the Windows Server 2003 Active Directory directory service.

To create this policy, follow these steps:

Step 1: Create a Group Policy object. To do this, log on to a domain controller, and then start the Active Directory Users and Computers tool.

1. Click Start, click Run, type dsa.msc, and then click OK.
2. Right-click the container in which you want to configure the Group Policy object, and then click Properties. For example, right-click the domain container, or right-click an organizational unit container.
3. Click the Group Policy tab, and then click New.
4. Type a descriptive name for the policy, and then press Enter.
5. Click Edit to start the Group Policy Object Editor.
6. Expand Computer Configuration, expand Windows Settings, expand Security Settings, and then click Public Key Policies.
7. Right-click Trusted Root Certification Authorities, and then click Properties.
8. Click Enterprise Root Certification Authorities, and then click OK.
9. Exit the Group Policy Object Editor.
10. Click OK to close the ObjectName Properties dialog box.

Step 2: Add root certificates to the Trusted Root Certification Authorities certificate store. Export any needed root certificates from the local computer store of the appropriate server. This includes root certificates for internal certification authorities (CAs) and root certificates for public certification authorities that your organization requires.

1. Log on to a domain controller, and then start the Active Directory Users and Computers tool.
2. Right-click the container that contains the Group Policy object that you created in the "Step 1: Create a Group Policy object" section, and then click Properties.
3. Click the Group Policy tab, click the Group Policy object, and then click Edit.
4. Expand Computer Configuration, expand Windows Settings, expand Security Settings, and then click Public Key Policies.
5. Right-click Trusted Root Certification Authorities, and then click Import.
6. Follow the steps in the Certificate Import Wizard to import the root certificate or the certificates that you exported in step 2A.
7. Exit the Group Policy Object Editor.
8. Click OK to close the ObjectName Properties dialog box.
   

Note There are some root certificates that are required by Windows. You must add these certificates to the policy that you created. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

Method 3:Configure Schannel to no longer send the list of trusted root certification authorities during the TLS/SSL handshake process

Note The steps that are listed here can be performed in Windows Server 2003 and in Windows Server 2008.

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require you to reinstall the operating system. Microsoft cannot guarantee that these problems can be resolved. Modify the registry at your own risk.

On the server that is running the UC application on which you experience this problem, set the following registry entry to false:



By default, this entry is not listed in the registry. By default, this value is 1 (True). This registry entry controls the flag that controls whether the server sends a list of trusted certification authorities to the client. When you set this registry entry to False, the server does not send a list of trusted certification authorities to the client. This behavior may affect how the client responds to a request for a certificate. For example, if Internet Explorer receives a request for client authentication, Internet Explorer displays only the client certificates that appear in the chain of one of the certification authorities that are in the list from the server. However, if the server does not send a list of trusted certification authorities, Internet Explorer displays all the client certificates that are installed on the client computer.

To set this registry entry, follow these steps:

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following registry subkey:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
3. On the Edit menu, point to New, and then click DWORD Value.
4. Type SendTrustedIssuerList, and then press Enter to name the registry entry.
5. Right-click SendTrustedIssuerList, and then click Modify.
6. In the Value data box, type 0 if that value is not already displayed, and then click OK.
7. Exit Registry Editor.

Note There are some root certificates that are required by Windows. You must add these certificates to the policy that you created. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

Although this fact is not noted in the "Symptoms" section, this issue can occur on all computers that are running Windows Server 2003 or Windows Server 2008 and are hosting Microsoft Exchange Unified Communications components such as the following roles:

• Client Access Server
• Mailbox
• Unified Communications Server
  

The Windows Server operating systems use the automatic root update mechanism to download certification authority updates from the Microsoft Windows Update website when a client requires a secure TLS connection. This automated certificate update process causes the server to accumulate the additional certification authority information that is needed to make sure of secure UC client TLS connection requests. The Windows Server Schannel component marshals the root certification authority information to the UC client that requires the secure TLS connection. The UC client will compare the content of the Schannel root certification authority list with its own list of certification authority information. This process makes sure that matching root certificate information is present at both endpoints of the TCP connection for the continuance of the TLS handshake process. However, the Windows Server Schannel component can marshal only a limited amount of the Windows Server installed certification authority information back to the UC client that requests the secure TLS connection. In some scenarios, this causes the UC client's secure TLS connection request to fail.

See the following sections for more information about the design differences for the Windows Server 2003 and Windows Server 2008 automatic root update configurations and about Schannel certification authority list limitations.

Windows Server 2003

In Windows Server 2003, the issuer list cannot be larger than 12,288 or 0x3000 bytes. The installation of Windows Server 2003 SP1 or of the Windows Server 2003 SP2 hotfix KB933940 that is listed in the "More Information" section provides an enlarged issuer list capacity of 16,384 or 0x4000 bytes.

The automatic root update mechanism is not enabled in Windows Server 2003. Windows Server 2003 supports the automatic root update mechanism. For more information about automatic root updates for Server 2003, visit the following Microsoft TechNet website:

Windows Server 2008

In Windows Server 2008, the issuer list cannot be larger than 16,384 or 0x4000 bytes.

By default, the automatic root update mechanism is enabled in Windows Server 2008 and in later versions of Windows. For more information about how to manage the Windows Server 2008 automatic root update mechanism, visit the following Microsoft TechNet website:


For more information about the issue that is described in the "Symptoms" section, click the following article numbers to view the articles in the Microsoft Knowledge Base: