When using connectors to forward alerts in System Center Operations Manager 2007 (OpsMgr 2007) or System Center 2012 Operations Manager (OpsMgr 2012, SP1 and OpsMgr 2012 R2), in certain situations such as an alert storm (defined as a large number of alerts being generated in a very short period of time) there may be alerts that are not forwarded via a connector. When this occurs, these alerts will never be forwarded and will remain in a "New" state.
This can occur if there is an alert storm and the connector processes alerts in batches (e.g. 100 at a time). If there are too many alerts to be processed within 5 seconds (the default value) from the time they were inserted into the database, they will be skipped and never reprocessed.
To resolve this issue, increase the amount of time allowed for the alerts to be processed. This can be done using the registry value named AlertSubscriptionWatermarkLatencyMilliseconds. This value can be set to increase the latency value so that when the query executes to collect the alerts for forwarding, the alerts that have a timestamp less than or equal to the current time minus the latency value (the current default is 5000 milliseconds).
This registry value does not exist by default but it can be created on the Root Management Server (for OpsMgr 2007) or on all of the Management Servers (for SCOM 2012).
Note: When this key is added, the Health Service on the Root Management Server (or on Management Servers in OpsMgr 2012) must be restarted to implement the change.
The value should only be increased, never decreased. It should be incremented (for example, adding 5000 milliseconds at a time) and tested to make sure the value is sufficient to not skip alerts. Setting it too high can cause performance issues by causing the connector to look at too many alerts at the same time unnecessarily.