How to troubleshoot Event ID 2115-related performance problems in Operations Manager

Applies to: System Center 2012 R2 Operations ManagerMicrosoft System Center 2012 Operations Manager Service Pack 1Microsoft System Center 2012 Operations Manager More

Symptoms


The issue of Operations Manager Database and Data Warehouse insertion times is one of the more common performance concerns in the following program versions:

  • Microsoft System Center 2012 R2 Operations Manager (OpsMgr 2012 R2)
  • Microsoft System Center 2012 Operations Manager (OpsMgr 2012)
  • Microsoft System Center 2007 Operations Manager (OpsMgr 2007)

This article provides some tips to help you identify and troubleshoot problems that affect Database and Data Warehouse data insertion.

A typical sign of Operations Manager database performance issues is the presence of Event ID 2115 events in the Operations Manager event log. These events typically indicate that performance issues exist on the Management server or on the server that is running Microsoft SQL Server and that is hosting the OperationsManager or OperationsManager Data Warehouse databases.

Background

Database and Data Warehouse write action workflows run on the Management Servers. These workflows first retain the data that they receive from the agents and gateway servers in an internal buffer. Then, they gather this data from the internal buffer and insert it into the database and data warehouse. When the first data insertion is completed, the workflows create another batch.

The size of each batch of data depends on how much data is available in the buffer when the batch is created. However, there is a maximum limit of 5,000 data items in a batch. If the rate of incoming data items increases, or if the data item insertion throughput to the Operation Manager and Data Warehouse databases throughput is reduced, the buffer accumulates more data and the batch size increases.

There are several write action workflows that run on a Management server. For example, the following workflows handle data insertion to the Operations Manager and Data Warehouse databases for different data types:

  • Microsoft.SystemCenter.DataWarehouse.CollectEntityHealthStateChange
  • Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
  • Microsoft.SystemCenter.DataWarehouse.CollectEventData
  • Microsoft.SystemCenter.CollectAlerts
  • Microsoft.SystemCenter.CollectEntityState
  • Microsoft.SystemCenter.CollectPublishedEntityState
  • Microsoft.SystemCenter.CollectDiscoveryData
  • Microsoft.SystemCenter.CollectSignatureData
  • Microsoft.SystemCenter.CollectEventData
When a Database or Data Warehouse write action workflow on a Management server experiences slow data batch insertion (for example, in excess of 60 seconds), the workflow begins to log Event ID 2115 to the Operations Manager event log. This event is logged every minute until the data batch is inserted into the database or data warehouse, or the data is dropped by the write action workflow module. Therefore, Event ID 2115 is logged because of the latency that occurs during the insertion of data into the database or data warehouse. The following is an example of an event that is logged because data is dropped by the write action workflow module:

Event Type: Error
Event Source: HealthService
Event Category: None
Event ID: 4506
Computer: <RMS NAME>
Description: Data was dropped due to too much outstanding data in rule "Microsoft.SystemCenter.OperationalDataReporting.SubmitOperationalDataFailed.Alert" running for instance <RMS NAME> with id:"{F56EB161-4ABE-5BC7-610F-4365524F294E}" in management group <MANAGEMENT GROUP NAME>.
Taking a deeper look
Event ID 2115 contains two significant pieces of information:

  • The name of the workflow that is experiencing the issue
  • The elapsed time since the workflow began to insert the last batch of data
For example:

Log Name: Operations Manager
Source: HealthService
Event ID: 2115
Level: Warning
Computer: <RMS NAME>
Description:
A Bind Data Source in Management Group <MANAGEMENT GROUP NAME> has posted items to the workflow, but has not received a response in 300 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectPublishedEntityState
Instance : <RMS NAME>
Instance Id : {88676CDF-E284-7838-AC70-E898DA1720CB}
The Microsoft.SystemCenter.CollectPublishedEntityState workflow writes Entity State data to the Operations Manager database. The Event ID 2115 message indicates that the Microsoft.SystemCenter.CollectPublishedEntityState workflow is trying to insert a batch of Entity State data, and that it started 300 seconds ago. In this example, the insertion of the Entity State data is not finished. Usually, the insertion of a batch of data should finish within 60 seconds.

If the Workflow ID contains the term "DataWarehouse," the problem involves the Operations Manager Data Warehouse database. Otherwise, the problem involves the insertion of data into the Operations Manager database.

Cause


Any of the following issues can cause these kinds of problems.

Resolution


The first step to troubleshoot the performance-related Event ID 2115 messages is to identify which data items are returned within the event. For example, the Workflow ID indicates which type of data items (such as Discovery, Alerts, Event, Perf) and which database are involved. If the Workflow ID contains the term "DataWarehouse," the troubleshooting focus should be on the Operations Manager Data Warehouse database. In other cases, the focus should be on the Operations Manager database.