Content Index status of all or most of the mailbox databases in the environment shows "Failed"


Symptom 1

The state of Microsoft Exchange Server 2013 content indexes is displayed as "Failed" by Get-MailboxDatabaseCopyStatus, and the associated error message indicates an internal error that occurs with the database or its index (Refer the following screen shot).

Screenshot for the status of content indexes

The Content Index status of all or most of the mailbox databases in the environment may show "Failed."

Symptom 2

The Application log may display the following events:

Log Name: Application
Source: MSExchangeFastSearch
Event ID: 1009
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
The indexing of mailbox database DB6 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Fast.FastConnectionException: Connection to the Content Submission Service has failed. ---> Microsoft.Ceres.External.ContentApi.ConnectionException: Given CSS specification failed:
Could not connect to CSS node at net.tcp://localhost:17028/ContentSubmissionServices/content with flow: Microsoft.Exchange.Search.Writer.12.14. Error: Object reference not set to an instance of an object. Inner exception:---
at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SetCssNodes(IEnumerable`1 newCssNodes)
at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder..ctor(DocumentFeederOptions options)
at Microsoft.Exchange.Search.Fast.FastFeeder.Initialize()
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Fast.FastFeeder.Initialize()
at Microsoft.Exchange.Search.Fast.Factory.InternalCreateFastFeeder(ISearchServiceConfig config, String indexSystemFlow, String indexSystemName, String instanceName, Int32 numberOfSessions)
at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)

Log Name: Application
Source: MSExchangeFastSearch
Event ID: 1010
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.FaultException: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs.

Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()
at Microsoft.Exchange.Search.Fast.IndexManager.<GetFlows>b__16()
at Microsoft.Exchange.Search.Fast.IndexManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
--- End of inner exception stack trace ---

This issue may occur if the search platform tries to check its membership in a security group that is named "ContentSubmitters." This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail.
To resolve the issue, do the following:
  1. Create a new Active Directory group that is named "ContentSubmitters" and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.
  2. Force or wait for Active Directory replication.
  3. Restart the following services:
    • Microsoft Exchange Search
    • Microsoft Exchange Search Host Controller

Article ID: 2807668 - Last Review: 03/07/2014 01:51:00 - Revision: 4.0

Microsoft Exchange Server 2013 Standard, Microsoft Exchange Server 2013 Enterprise

  • kbsurveynew KB2807668