Bug: MSDTC communication is not working on a Windows 2008 and R2 Cluster using Incoming Caller Authentication

Applies to: Windows Server 2008 EnterpriseWindows Server 2008 DatacenterWindows Server 2008 R2 Enterprise

Symptoms


When you run an application that tries to connect to a SQL server cluster on Windows 2008 or Windows 2008 R2, you may get the following error when the DTC security is configured for "Incoming Caller Authentication Required": 

"New transaction cannot enlist in the specified transaction coordinator"

In most cases the failure doesn’t occur if the MS DTC security on the cluster is configured for "No Authentication Required" instead. The client operating system could be either Windows 2003 or Windows 2008. 

Cause


If you enable MS DTC tracing (TRACE_CM) on the cluster, you may see the messages similar to these:

12-21-2009 17:36:25:700 : [33a8. e34] 0x00000000 [    TRACE_CM] CRpcIOManagerClient::SetRpcSecurity (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@636): We are a cluster resource - Impersonate the virtual server name DTC01
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [TRACE_CLUSTER] MtxCluGetVirtualServerToken (d:\rtm\com\complus\dtc\shared\mtxclu\src\clusterutil.cpp@5159): OpenClusterResource failed for virtual server name DTC01
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [    TRACE_CM] CRpcIOManagerClient::SetRpcSecurity (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@641): Error from MtxCluGetVirtualServerToken
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [    TRACE_CM] CRpcIOManagerClient::SecureBuildContextWrapper (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@760): [Remote:APPS01 07adda78] Call to SetRpcSecurity failed
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [ TRACE_CM] CRpcIOManagerClient::Bind (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@324): [Remote:APPS01 07adda78] SecureBuildContextWrapper call failed. This is usually due to security/network configuration issues.
12-21-2009 17:36:25:709 : [33a8. e34] 0x80070005 [    TRACE_CM] CRpcIOManagerClient::BuildContextWrapper (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@847): [Remote:APPS01 07adda78] Exception raised in the rpc call C_BuildContextW.

This is a known issue with the Distributed Transaction Coordinator. The OpenClusterResource fails because it could not find the Name resource with the name "DTC01" in the DTC cluster resource group. "DTC01" here is the DNS name or called private name of the DTC network Name resource. The error 0x8007138f means ERROR_RESOURCE_NOT_FOUND. You encounter this problem if the network name that the DTC resource depends on is not the same as its DNS name. In this example the MS DTC resource is in its own group whose network name is "MSDTC Network Name" and the corresponding DNS name is DTC01". Note we are talking about the network Name resource that the DTC resource depends on but not the DTC resource name itself.

Resolution


To workaround the issue use "Incoming Caller Authentication Required" for the MS DTC security on a Windows 2008 or Windows 2008 R2 cluster, the DTC network name must be the same as its DNS name. In this example we can use the following command to change the DTC network name from "MSDTC Network Name" to "DTC01":

Cluster.exe res “MSDTC Network Name” /ren:DTC01

Windows 2008/Windows 2008 R2 can have multiple DTC instances on a cluster. Each DTC instance can be configured in any resource group. This change is not limited to one DTC resource group or one SQL Server resource group that has a MSDTC resource. The same change needs to be made in any SQL resource group that has a MSDTC resource or any other group that has a MSDTC resource where the network name for MSDTC is different from its DNS name. 

A common configuration is to have a SQL cluster resource group and then to add a DTC resource to that same group. Both the SQL and the DTC resources depend on the same network name "SQL Network Name". In this case the resource group "SQL Network Name" will need to be changed to its DNS name.

More Information


For more info on configuring MS DTC instances on a Windows 2008 cluster, please read this article: How to Configure Multiple Instances of Distributed Transaction Coordinator (DTC) on a Windows Server Failover Cluster 2008