MSDTC fails to mutually authenticate when computers do not run in the same domain

View products that this article applies to.

SYMPTOMS

When two computers that are not running in the same domain use Microsoft Distributed Transaction Coordinator (MSDTC) for communication and for distributed transactions, the mutual authentication may fail.

When you run a distributed transaction against a linked server in Microsoft SQL Server 2000 on a computer that is running Microsoft Windows Server 2003 Enterprise Edition, you may receive the following error message:

Server: Msg 7391, Level 16, State 1, Line 2
The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction. OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator.

Back to the top

CAUSE

The MSDTC proxy may not correctly authenticate MSDTC when the communicating computers are not in the same domain.

Back to the top

WORKAROUND

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

322756 (http://support.microsoft.com/kb/322756/ ) How to back up and restore the registry in Windows

For information about how to edit the registry, view the "Change Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Microsoft recommends that you back up the registry before you edit it.

To work around this problem, turn off the RPC security on your Windows Server 2003 computers. RPC security is a new DTC feature in Windows Server 2003. When you disable RPC security, the DTC authentication security level for RPC calls goes back to a level that is available in Microsoft Windows 2000 Server. To do this, follow these steps to set the DWORD value for the TurnOffRpcSecurity registry value to 1:

Start Registry Editor (Regedt32.exe).
Locate the following key in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC
On the Edit menu, click Add Value, and then add the following registry value:
Collapse this tableExpand this table
Value name Data type Value
TurnOffRpcSecurity REG_DWORD 1
Quit Registry Editor.

Note After you install Windows Server 2003 Service Pack 1, the registry entry TurnOffRpcSecurity is set to 0. You have to restore the value to 1 by using Registry Editor before you restart the computer.

Back to the top

STATUS

This behavior is by design.

Back to the top

MORE INFORMATION

Steps to reproduce the behavior

Make sure that both computers are running Windows Server 2003.
Install SQL Server 2000 with Service Pack 3 (SP3) on both the computers.
Make sure that MSDTC is started on both the computers.
On the first computer, start the SQL Query Analyzer (Isqlw.exe) utility, and then connect to the local SQL Server.
Add the second computer as the linked server. To do so, run the following Transact-SQL statement in SQL Query Analyzer:
EXEC sp_addlinkedserver 'remote_server', N'SQL SERVER' GO
Note Replace remote_server with the name of the second computer.
Run the following Transact-SQL statement in SQL Query Analyzer:
SET xact_abort ON GO USE pubs GO BEGIN DISTRIBUTED TRANSACTION SELECT * FROM remote_server.pubs.dbo.authors COMMIT TRAN GO
You may receive the error message that is mentioned in the "Symptoms" section of this article.