When two computers that are not running in the same domain use Microsoft Distributed Transaction Coordinator (MSDTC) for communication and for distributed transactions, the mutual authentication may fail.
When you run a distributed transaction against a linked server in Microsoft SQL Server 2000 on a computer that is running Microsoft Windows Server 2003 Enterprise Edition, you may receive the following error message:
Server: Msg 7391, Level 16, State 1, Line 2 The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction. OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator.
The MSDTC proxy may not correctly authenticate MSDTC when the communicating computers are not in the same domain.
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 How to back up and restore the registry in Windows
For information about how to edit the registry, view the "Change Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Microsoft recommends that you back up the registry before you edit it.
To work around this problem, turn off the RPC security on your Windows Server 2003 computers. RPC security is a new DTC feature in Windows Server 2003. When you disable RPC security, the DTC authentication security level for RPC calls goes back to a level that is available in Microsoft Windows 2000 Server. To do this, follow these steps to set the DWORD value for the TurnOffRpcSecurity registry value to 1:
Start Registry Editor (Regedt32.exe).
Locate the following key in the registry:
On the Edit menu, click Add Value, and then add the following registry value:
Quit Registry Editor.
Note After you install Windows Server 2003 Service Pack 1, the registry entry TurnOffRpcSecurity is set to 0. You have to restore the value to 1 by using Registry Editor before you restart the computer.
This behavior is by design.
Steps to reproduce the behavior
Make sure that both computers are running Windows Server 2003.
Install SQL Server 2000 with Service Pack 3 (SP3) on both the computers.
Make sure that MSDTC is started on both the computers.
On the first computer, start the SQL Query Analyzer (Isqlw.exe) utility, and then connect to the local SQL Server.
Add the second computer as the linked server. To do so, run the following Transact-SQL statement in SQL Query Analyzer: