"OnlyUseLatestCLR" breaks Exchange on SBS 2011 Standard


Symptoms


Symptom 1:

The Microsoft Exchange RPC Service continually crashes, logging the following errors:

Log Name: System
Source: Service Control Manager
Date: 9/7/2011 11:48:29 AM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: SBS2011.contoso.local

Description:

The Microsoft Exchange RPC Client Access service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name: Application
Source: MSExchange Common
Date: 9/7/2011 11:44:40 AM
Event ID: 4999
Task Category: General
Level: Error
Keywords: Classic
User: N/A
Computer: SBS2011.contoso.local

Description:

Watson report about to be sent for process id: 4448, with parameters: E12, c-RTL-AMD64, 14.01.0323.006, M.E.RpcClientAccess.Service, unknown, M.E.R.C.SyncInstance.ExecuteDataSourceOperation, System.InvalidOperationException, 50a0, unknown.

Log Name: Application
Source: MSExchangeRPC
Date: 9/7/2011 11:44:39 AM
Event ID: 1008
Task Category: General
Level: Error
Keywords: Classic
User: N/A
Computer: SBS2011.contoso.local

Description:

Encountered unexpected error when starting MSExchangeRPC service. Error details: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.

at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
at Microsoft.Exchange.RpcClientAccess.ConfigurationManager.SyncInstance.ExecuteDataSourceOperation(IEnumerable`1 dataSources, Action`1 operationDelegate)
at Microsoft.Exchange.RpcClientAccess.ConfigurationManager.ForEach(Action`1 action)
at Microsoft.Exchange.RpcClientAccess.Service.RpcClientAccessService.<DeferredServiceStartInitialization>b__4()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate)

Symptom 2:

OWA does not open, "page cannot be displayed".  The application pool terminates with the following error:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/7/2011 11:48:22 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: SBS2011.contoso.local

Description:

A process serving application pool 'MSExchangeOWAAppPool' terminated unexpectedly. The process id was '9868'. The process exit code was '0xff'.

Symptom 3:

The Microsoft Exchange Transport Service crashes when routing email destined for local submission.  These email messages are considered poison by the service and placed in the poison queue accordingly. The following event will be logged:

If you view the queue you will see the poisoned messages; run Get-Queue from Exchange PowerShell:

Log Name: Application
Source: MSExchangeTransport
Date: 9/7/2011 1:12:42 PM
Event ID: 10001
Task Category: PoisonMessage
Level: Warning
Keywords: Classic
User: N/A
Computer: SBS2011.contoso.local

Description:

1 messages have reached or exceeded the configured poison threshold of 2. After the Microsoft Exchange Transport service restarted, these messages were moved to the poison message queue.

 

Cause


The following registry key is used to globally override the version of the CLR that framework code will execute in.  This is used in compatibility testing scenarios when developing managed code applications and should never be created on a production server, much less a Small Business Server:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework]

"OnlyUseLatestCLR"=dword:00000001

The .Net Framework 4.0 is installed on SBS 2011 out of the box.  The creation of this key effectively forces Exchange 2010 managed code to execute in the 4.0 CLR, which it was not designed to do.

 

Resolution


Remove this key and restart all Microsoft Exchange Services, no reboot is necessary.  To resubmit the poisoned messages run the following commands from Exchange PowerShell:

$colmsg= Get-Message -Queue "Poison"
$colmsg | ForEach {Resume-Message $_.Identity}