Symptoms
After you install and upgrade to Microsoft Exchange Server 2016 Cumulative Update 6 (CU6), you can't access Outlook Web App (OWA) or Exchange Control Panel (ECP), and you receive the following error message:
:-( Something went wrong We can't get that information right now. Please try again later. X-ClientId: ClientID X-FEServer: Exch1
In addition, the following events information is recorded in the Application log of the Exchange server that hosts the mailbox database:
Log Name: Application Source: MSExchange OAuth Event ID: 2004 Task Category: Configuration Level: Warning Keywords: Classic User: N/A Computer: mail.contoso.com Description: Unable to find the certificate with thumbprint CertificateValue in the current computer or the certificate is missing private key. The certificate is needed to sign the outgoing token.
Log Name: Application Source: ASP.NET 4.0.30319.0 Event ID: 1309 Task Category: Web Event Level: Warning Keywords: Classic User: N/A Description: Event code: 3005 Event message: An unhandled exception has occurred. Event ID: EventID Event sequence: 2 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/2/ROOT/owa-2-DomainID Trust level: Full Application Virtual Path: /owa Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\ Machine name: Exch1 Process information: Process ID: 21508 Process name: w3wp.exe Account name: NT AUTHORITY\SYSTEM Exception information: Exception type: TargetInvocationException Exception message: Exception has been thrown by the target of an invocation. at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder) at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder) at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup) at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup) at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint() at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory) at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context) at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Encryption certificate is absent at Microsoft.Exchange.Security.Authentication.Utility.GetCertificates() at Microsoft.Exchange.Clients.Owa2.Server.Core.notifications.SignalR.SignalRStartup.Configuration(IAppBuilder app)
Request information: Request URL: https://mail.contoso.com:URLID/owa/?bO=1 Request path: /owa/ User host address: UserHostAddressC:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\ User: Is authenticated: False Authentication Type: Thread account name: NT AUTHORITY\SYSTEM Thread information: Thread ID: 24 Thread account name: NT AUTHORITY\SYSTEM Is impersonating: False Stack trace: at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder) at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder) at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup) at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup) at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint() at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory) at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context) at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Cause
This issue occurs if the Exchange server Auth certificate that's used for OAuth signing is missing from the Exchange server. You can run the following command to check whether the certificate is missing:
Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint
If the certificate is missing, you will receive the following error message:
A special Rpc error occurs on server Exch1: The certificate with thumbprint CertificateValue was not found. + CategoryInfo : NotSpecified: (:) [Get-ExchangeCertificate], InvalidOperationException + FullyQualifiedErrorId : [Server=Exch1,RequestId=RequestID] [FailureCategory=Cmdlet-InvalidOperationException] CEA009BC,Microsoft.Exchange.Management.SystemConfigurationTasks.GetExchangeCertificate
Resolution
To fix this issue, install Cumulative Update 7 for Exchange Server 2016 or a later cumulative update for Exchange Server 2016.
Workaround
If your organization has multiple Exchange servers, run the following command in the Exchange Management Shell to confirm if the OAuth certificate is present on other Exchange servers:
Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint
If the certificate is present on other Exchange servers, export the certificate and then import it to the Exchange server that has the issue.
If the certificate isn't present on all Exchange servers in the organization, follow these steps to create and deploy a new OAuth certificate to the Exchange server:
-
Create a new OAuth certificate by running the following command:
New-ExchangeCertificate -KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName "contoso.com"
Note Change the value of the DomainName parameter in the example (contoso.com) to the SMTP domain that's used in your organization.
-
Set the created certificate to be used for server authentication by running the following commands:
Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate (Get-Date)Set-AuthConfig –PublishCertificate Set-AuthConfig -ClearPreviousCertificate
-
Restart the Microsoft Exchange Service Host Service.
-
Either run the IISReset command to restart IIS or run the following commands (in elevated mode) to recycle OWA and ECP APP pools:
Restart-WebAppPool MSExchangeOWAAppPool Restart-WebAppPool MSExchangeECPAppPool
Note In some environments, it may take an hour for the OAuth certificate to be published.
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
References
Learn about the terminology that Microsoft uses to describe software updates.
