Content Index status of all or most of the mailbox databases in the environment shows "Failed"

Symptom 1 

The state of Microsoft Exchange Server 2013 content indexes is displayed as "Failed" by Get-MailboxDatabaseCopyStatus, and the associated error message indicates an internal error that occurs with the database or its index.



The Content Index status of all or most of the mailbox databases in the environment may show "Failed."

Symptom 2 

The Application log may display the following events:

Log Name:      Application
Source:        MSExchangeFastSearch
Event ID:      1009
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Description:
The indexing of mailbox database DB6 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Fast.FastConnectionException: Connection to the Content Submission Service has failed. ---> Microsoft.Ceres.External.ContentApi.ConnectionException: Given CSS specification failed:
Could not connect to CSS node at net.tcp://localhost:17028/ContentSubmissionServices/content with flow: Microsoft.Exchange.Search.Writer.12.14. Error: Object reference not set to an instance of an object. Inner exception:---
   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SetCssNodes(IEnumerable`1 newCssNodes)
   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder..ctor(DocumentFeederOptions options)
   at Microsoft.Exchange.Search.Fast.FastFeeder.Initialize()
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Fast.FastFeeder.Initialize()
   at Microsoft.Exchange.Search.Fast.Factory.InternalCreateFastFeeder(ISearchServiceConfig config, String indexSystemFlow, String indexSystemName, String instanceName, Int32 numberOfSessions)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
   at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)

Log Name:      Application
Source:        MSExchangeFastSearch
Event ID:      1010
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Description:
An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs.

Server stack trace:
   at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()
   at Microsoft.Exchange.Search.Fast.IndexManager.<GetFlows>b__16()
   at Microsoft.Exchange.Search.Fast.IndexManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
   --- End of inner exception stack trace ---

This issue may occur if the search platform tries to check its membership in a security group that is named "ContentSubmitters." This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail.

To resolve the issue, use one of the following methods.

Method 1

1. Create a new Active Directory group that is named "ContentSubmitters," and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed. 
2. Force or wait for Active Directory replication.
3. Restart the following services:
   • Microsoft Exchange Search 
   • Microsoft Exchange Search Host Controller
     

Method 2

You can also resolve this problem by removing a configuration entry that refers to the ContentSubmitters group. This entry is present in four files. Each file is called WcfConfigurator.xml, and you can find each file in the following folder:

Here, "%ExchangeInstallPath%" is the Exchange installation root path (available in the environment variable of that name), and the placeholder NODENAME represents one of the following names. (There is one name for each of the four files that are to be edited.) 

• AdminNode1
• ContentEngineNode1
• IndexNode1
• InteractionEngineNode1

These are text files. You can use Notepad or any other convenient plain text editor to change them. Make sure that you make only the change that is described here.

Each file contains at most one line that contains the following:

    <AuthorizedRole>ContentSubmitters</AuthorizedRole>

Remove this line, and then save the file. Repeat this operation for each of the four files, and then restart the Microsoft Exchange Search Host Controller service.