Search doesn't work in Exchange Server 2013 when the NodeRunner.exe process stops

This article describes an issue that's caused by the NodeRunner.exe process. To fix this issue, follow the steps in the "Resolution" section.
Symptoms
When this issue occurs, you may experience the following symptoms.

Symptom 1

The Content Index State on all mailbox databases is FailedAndSuspended.

Symptom 2

The following message is logged in the Unified Logging System (ULS) logs:

NodeRunnerIndexNode1-XXXX(0x59D4) 0x6390 Search Search Component
IndexRouter[IndexRouter]: Caught exception when preparing generation GID[GID]: (IndexNode1-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.Single: bad allocation): Microsoft.Ceres.SearchCore.FastServer.FastServerException:
IndexNode1-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.Single: bad allocation
at Microsoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()
at Microsoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractProtectedGenerationBuilder.FlushGenerationInternal()
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.PrepareCloseGeneration()


Note
 By default, Exchange ULS log files are saved in the following location:
C:\Program Files\Microsoft\Exchange Server\V15\Logging\Search

Symptom 3

The NodeRunner.exe process throws an exception when it tries to allocate more memory than the available memory. In this situation, the following errors are logged:

ID:1026
Level:Error
Source:.NETRuntime
Message: Application: NodeRunner.exe
Framework Version:Framework Version
Description: The process was terminated due to an unhandled exception
Exception Info: System.OutOfMemoryException
Stack:
at Microsoft.Ceres.CoreServices.Framework.Container.ComponentManager.FailingInvoker()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()


ID:1000
Source: Application
Error Message:
Faulting application name: NodeRunner.exe,
version: Version,
time stamp: Time Stamp
Faulting module name: ntdll.dll,
version: Version,
time stamp: Time Stamp
Exception code: Exception code
Fault offset:offset
Faulting process id:Process id
Faulting application start time:Time
Faulting application path: C:\Program Files\Microsoft\ExchangeServer\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: Report Id
Faulting package full name: Faulting package-relative application ID:


ID: 1026
Level:Error
Source: .NET Runtime
Application: NodeRunner.exe
Framework Version: Framework Version
Description: The process was terminated due to an unhandled exception.
Exception Info:
System.AccessViolationException
Stack:
at Microsoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()
at Microsoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()
at Microsoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.PublishGeneration()
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.CloseGeneration()
at Microsoft.Ceres.SearchCore.Journal.RegistrationHandler.CloseGenerationToUsers(Microsoft.Ceres.SearchCore.Journal.RecoveryState)
at Microsoft.Ceres.SearchCore.Journal.Journal.RecoveryEndGeneration(Microsoft.Ceres.SearchCore.Services.GenerationId)
at Microsoft.Ceres.SearchCore.Journal.RegistrationHandler.RedoJournalEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16,Microsoft.Ceres.SearchCore.Journal.RecoveryState)
at Microsoft.Ceres.SearchCore.Journal.Journal.RecoverEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16)
at Microsoft.Ceres.SearchCore.Journal.DoubleBuffered.DoubleBufferJournal.RecoveryScan(Microsoft.Ceres.SearchCore.Services.Journal.CheckpointId, Microsoft.Ceres.SearchCore.Journal.JournalEntryRecovered, Microsoft.Ceres.SearchCore.Journal.ResumeCheckpointing, System.Collections.Generic.IDictionary`2<UInt16,Microsoft.Ceres.Common.Utils.IFactory`1<Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry>>) at Microsoft.Ceres.SearchCore.Journal.Journal.DoLogicalRecovery()
at Microsoft.Ceres.SearchCore.Journal.Journal.EnsureFullRecovery()
at Microsoft.Ceres.SearchCore.Journal.JournalComponent.DoReadying()
at System.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()


ID: 1000
Level: Error
Message:
Faulting application name: NodeRunner.exe, version: Version, time stamp: Time stamp
Faulting module name: MSVCR110.dll, version: Version, time stamp: Time stamp
Exception code: Code Fault offset: offset
Faulting process id: Process ID
Faulting application start time: Time
Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR110.dll Report Id: Report Id
Faulting package full name: Faulting package-relative application ID:

Cause
This issue occurs when the NodeRunners.exe process stops because of an OutOfMemory exception. Either the server is out of memory or the .NET Framework common language runtime (CLR) is forcing a memory allocation limitation on the NodeRunners.exe process. In this situation, NodeRunner.exe tries unsuccessfully to allocate more memory, generates an exception, and then stops.

Resolution
To fix this issue, follow these steps:
  1. Locate the Noderunner.exe.config file. By default, this file is located along the following path:
    C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config
  2. Edit the file, and then locate the following key:
    <nodeRunnerSettings memoryLimitMegabytes="<value>" />
  3. If the value of the key is set to any value other than 0, change it to 0, as follows:
    <nodeRunnerSettings memoryLimitMegabytes="0" />
  4. Restart the Microsoft Exchange Host Controller Service.
About the NodeRunner.exe process
Microsoft Exchange Host Controller Service starts four worker processes, and each is named NodeRunner.exe. NodeRunner.exe is part of the Exchange search component. The individual functionality of each NodeRunner.exe process is set through configuration. The NodeRunner.exeprocess that starts single Admin node is a process of its own. Because NodeRunner.exe is a stand-alone process, it derives some of its operating properties from the NodeRunner.exe.config application configuration file during the start of the Admin node.

The following screen shot shows the four nodes of the NodeRunner.exe process: Admin, Content, Query, and Index.

Screen shot of the four nodes

If you're familiar with SharePoint Server, you probably know how to limit the memory that's allocated to the NodeRunner.exe process by using the application configuration file. However, it's unsupported in Exchange Server 2013 to limit memory allocation for NodeRunner.exe by this method.

The default setting for the minimum memory requirement for NodeRunner.exe is 0. The NodeRunner.exe process can consume and change its memory requirements dynamically, based on current requirements and available memory. However, you can set the upper limit and restrict the volume of memory that Node Runner can access by using the memoryLimitMegabytes parameter in the NodeRunner.exe.config file. If you limit memory usage by NodeRunner.exe, and if the Exchange server can't allocate memory for the NodeRunner.exe operation, the operation may fail with an OutOfMemoryException exception.
Properties

Article ID: 3094698 - Last Review: 10/16/2015 16:39:00 - Revision: 1.0

Microsoft Exchange Server 2013 Enterprise, Microsoft Exchange Server 2013 Standard

  • kbtshoot kbprb kbsurveynew kbexpertiseadvanced KB3094698
Feedback