Select the product you need help with
- Internet Explorer
- Windows Phone
- More products
How to detect and troubleshoot frequent configuration changes in Operations Manager
Article ID: 2603913 - View products that this article applies to.
The System Center Management Configuration service is responsible for calculating the configuration of every health service in the Operations Manager Management Group. The configuration of a health service consists of the rules, monitors, discoveries, and tasks for the health service and for all the instances that the health service monitors.
To calculate all the required configurations for each health service, the Management Configuration service must have a list of the following items:
Objects in a management group will be defined as instances of monitored classes based on discovery data that is submitted by discovery workflows. If a key property of an object changes, that object may be added as a new instance of a monitored class. Otherwise, that object is no longer considered an instance of that class.
As the list changes for the classes that the object is a member of, the configuration also changes for the health service that monitors that object. These changes occur as rules, monitors, discoveries, tasks, and overrides are added or removed from the previous configuration.
Agents may be unable to receive a stable configuration in the following scenarios:
Discovery data is submitted by a health service when a discovery workflow runs. Introduction of a new Management Pack to a Management Group can cause several discovery workflows to run on each agent. And, as new instances are discovered, additional discoveries may be run on some agents. Changes to groups, overrides, and other workflows can cause discovery workflows to run on agents. And, introduction of new agents can also cause the Management Configuration service to update the instance space by using the new agent's configuration.
The Configuration Management service is forced to recalculate the health service configuration frequently in the following scenarios:
Identifying configuration churn by using the MS event log
An event that resembles the following in the Operations Manager event log on Management Server indicates that the Management Group configuration has changed because of new discovery data.
Log Name: Operations Manager
An event that resembles the following indicates that the Management Configuration service has finished processing the new discovery data and calculated any changes that are required to the Management Group configuration, based on the new data.
Log Name: Operations Manager
Source: OpsMgr Connector
Event ID: 21025
Computer: <MS Name>
OpsMgr has received new configuration for management group <ManagementGroupName> from the Configuration Service. The new state cookie is "34 FA 11 61 4D B8 03 59 3D 1D 66 B7 83 F3 C0 AA 7A 6F 1A 3B "
In a typical environment, every 21024 should be followed by a 21025. If the discovery data did not cause any configuration data to change, the event ID will be 21026 instead. In a large Management Group, pairs of 21024 and 21025 or 21026 events should be expected to occur several times per hour. Long strings of 21024 events without a corresponding 21025 or 21026 event is a sign of configuration churn. In addition, the event log may show the following event that indicates that churn was detected.
Log Name: Operations ManagerThe Data Access Layer must read multiple tables when the Data Access Layer queries for changes. If one of the tables is modified after it is read but before all tables were read, the Data Access Layer logs the previous event ID 29202 and retry. If an entity or relationship instance was read during this time, information about these instances is included in the event fields. Otherwise, these fields are left empty.
Source: OpsMgr Config Service
Event ID: 29202
Computer: <MS Name>
OpsMgr Config Service could not retrieve a consistent state from the OpsMgr database due to too frequent database changes.
This could be due to a normal and temporary increase of discovery data; however check the most recent changes to determine if this increase is unexpected.
Most recent monitoring object change:
Instance = %1
Class = %2
Modified time = %3
Most recent monitoring relationship change:
Relationship instance = %4
Source instance = %5
Target instance = %6
RelationshipClass = %7
Modified time = %8
Identifying potential causes of configuration churn by using the Operations Manager Datawarehouse
In management groups in which the Operations Manager Reporting component was installed, several SQL queries can be used to identify workflows that are submitting frequent changes. These queries should be run in SQL Management Studio against the Datawarehouse instance.
Total changes submitted by discovery workflows in last 24 hours:
This query creates three columns. The first column is the class of object at which the workflow is targeted. The second column indicates the internal name of the discovery workflow. The third column indicates the total number of property changes for all instances of this class that were submitted by the workflow in the last 24 hours. The total number of changes, for all classes, represents the number of times the Configuration Management service must recalculate the configuration for an agent health service.
select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes' from (select distinct MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName, PropertySystemName, D.DiscoverySystemName, D.DiscoveryDefaultName, MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName', ME.Path, ME.Name, C.OldValue, C.NewValue, C.ChangeDateTime from dbo.vManagedEntityPropertyChange C inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%' left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId where ChangeDateTime > dateadd(hh,-24,getutcdate()) ) As #T group by ManagedEntityTypeSystemName, DiscoverySystemName order by count(*) DESC
The number of changes for some classes of objects, even in a stable environment, may not ever reach zero. Any change, such as adding or removing a property, agents that are added or decommissioned, server roles that are added or changed, and so on, are reflected in the numbers that are returned. In environments in which configuration churn is experienced, one or several workflows will likely show a significantly larger value than other workflows.
Properties changed in the last 24 Hours:
This query can identify which properties have changed in the last 24 hours. Combined with the previous query, this query can show what the old and new values were for the property, which agents submitted the change, the workflow that conducted the discovery, and the management pack in which it was contained.
select distinct MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName, PropertySystemName, D.DiscoverySystemName, D.DiscoveryDefaultName, MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName', ME.Path, ME.Name, C.OldValue, C.NewValue, C.ChangeDateTime from dbo.vManagedEntityPropertyChange C inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%' left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId where ChangeDateTime > dateadd(hh,-24,getutcdate()) ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName
How to reduce configuration churn
Older management packs introduced discovery workflows that submitted property changes too frequently. The current version of most management packs have modified these discovery workflows to submit data less frequently, or the management packs do not query volatile properties that frequently change. We recommend that you upgrade management packs that contain workflows that frequently occur in the previous query. New versions of the management pack can be downloaded from the management pack catalog:
If a new version of the management pack is not available, or the new version cannot be deployed now, the discovery interval can be adjusted by using override to run less frequently. Sometimes, the discovery that is responsible for the configuration churn can be completely disabled by override. If the discovery is disabled for several weeks, the objects that are discovered by the workflow may be groomed out of the database. However, disabling the discovery can provide a short-term workaround to eliminate configuration churn, as long as a permanent solution can be implemented before any objects are groomed from the database. The workflow can also be enabled for short intervals to rediscover the objects before they are groomed.
Some workflows in these older management packs are discussed in the following blog:
http://blogs.technet.com/b/kevinholman/archive/2009/10/05/what-is-config-churn.aspxIf the workflow is from a custom discovery that targets a volatile property, such as free disk space, the discovery should be rewritten so that it does not target a property that frequently changes. Discovery workflows should not target instances that have a short lifetime (several weeks or less). Discovery workflows should not collect properties of those instances that frequently change (one or more times a month). Volatile data is not considered in calculating a configuration. Therefore, volatile data should be collected by performance rules and not by discovery workflows.
Additional performance tuning
In large management groups (greater than 1,000 agents), the RMS may become very busy with operations that typically do not cause a problem in smaller management groups. In this situation, even a small rate of property changes could cause frequent churn because of the length of time that is required to process the changes. Several configuration changes can be used to reduce the operational overhead of the RMS and enable it to process a typical rate of property changes quickly enough to avoid configuration churn. These configuration changes are discussed in the following blog:
Forcing configuration change for the management group
If configuration churn for the management group occurs constantly, any changes to reduce the frequency of the problem workflows or to disable the problem workflows will never be propagated to agents. In this case, the flow of incoming discovery data must be blocked to allow the System Center Configuration Management service to calculate a current configuration in which the workflow that produces this data is disabled or runs less frequently.
Discovery data is submitted to the OperationsManager database through the System Center Data Access Service (DAS). The data is first submitted to the DAS by the System Center Management service on the RMS. The RMS obtains this data from agents or from other management servers. You can use Windows firewall or some other networking means to block incoming connections to the RMS on port 5723. This blocking procedure prevents discovery data from being submitted to the OperationsManager database just long enough for the Configuration Management service to calculate the current configuration for the agents that are submitting the data.
The System Center Management service and the System Center Data Access Service on the RMS should not be stopped or disabled while the Configuration Management service is calculating the current configuration. The System Center Configuration Management service requires the following in order to complete the calculation of the management group configuration:
Identifying potential causes of configuration churn by using Operations Manager reporting
New reports were introduced with version 6.1.7599.0 of the Operations Manager 2007 R2 Management Pack. These reports provide insight into the overall volume of data that the management group processes. These reports can be used to establish a standard baseline and to identify opportunities for tuning object discovery workflows. As soon as configuration churn is identified and addressed, these reports can be used for long-term planning to prevent recurrences of churn.
To download the management pack, visit the following Microsoft website:
Article ID: 2603913 - Last Review: August 2, 2012 - Revision: 8.0