Event ID 1009 and high CPU or memory usage in noderunner.exe in Exchange Server 2013

Symptoms
After you apply Cumulative Update 10 or Cumulative Update 11 for Microsoft Exchange Server 2013, you may experience some or all of the following symptoms:
  • High CPU usage in the noderunner.exe process
  • High memory usage in noderunner.exe
  • High handle count in noderunner.exe
  • Failed content indexes
  • Database failovers
Memory usage may exceed 10 GB for a single noderunner.exe process, and handle counts may exceed 20,000.

Additionally, the following event may be logged in the Application log:

Source: MSExchangeFastSearch
Category: General
Description:
The indexing of mailbox database <DB name> encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Fast.FastDocumentTimeoutException: The Content Submission Service returned failure for the document: Document timeout expired
. ---> System.TimeoutException: Document timeout expired

--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Mdb.RetryFeeder.DocumentCompleteCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)



If you review the Search Service logs, you may see several events that contain the following text:

Failure: Microsoft.Exchange.Search.Fast.FastDocumentTimeoutException: The Content Submission Service returned failure for the document: Document timeout expired. ---> System.TimeoutException: Document timeout expired

Also, the following event is logged in the System log:

Product: Windows Operating System
ID: 7031
Source: Service Control Manager
Version: 6.X
Symbolic Name: EVENT_SERVICE_CRASH
Message: The Microsoft Exchange Search Host Controller service terminated unexpectedly.  It has done this x time(s).  The following corrective action will be taken in 30 milliseconds: Restart the service.
and
Message: The Microsoft Exchange Search service terminated unexpectedly.  It has done this x time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.


Finally, the Unified Logging Service (ULS) logs may contain several events with the following text:
System.TimeOutException Parsing

Operation timed out

Note
 By default, the Search Service logs are located at %ProgramDir%\Microsoft\Exchange Server\V15\Logging\Search, and the ULS logs are located at %ProgramData%\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\Logs.

Database failovers occur when content indexes are marked "Failed" or "Failed and Suspended." These failovers are typically indicated in the Application log through event ID 122.
Cause
This issue occurs because the Exchange Search and Microsoft Exchange Search Host Controller services are stopped by the Exchange Server 2013 Managed Availability diagnostic when parsing certain types of embedded OLE objects. Therefore, the content index is changed to a Failed and Suspended state. This behavior was introduced in Cumulative Update 10 and shouldn't be triggered by any earlier cumulative updates.
Resolution
To fix this issue, install Cumulative Update 12 or a later cumulative update for Exchange Server 2013.
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
Properties

Article ID: 3121815 - Last Review: 05/11/2016 17:52:00 - Revision: 1.0

Microsoft Exchange Server 2013 Enterprise, Microsoft Exchange Server 2013 Standard

  • kbqfe kbsurveynew kbfix kbexpertiseinter KB3121815
Feedback