The SCVMM Virtual Machine Manager service may consume high memory or CPU utilization

Symptoms

In System Center Virtual Machine Manager 2008, if the Job Count exceeds the recommended upper limit then the Virtual Machine Manager service may consume a large amount of memory or exhibit high CPU utilization.

Cause

This issue may occur if the Job Count exceeds the recommended limits.

The maximum number of running jobs that an SCVMM system should be able to handle is around 10,000 and hitting this many running jobs at one time would be an exceptionally rare occurrence. Should it occur, however, it can have a severe impact on the performance of the SCVMM engine.

The number of retained Jobs count refers to the number of running tasks tracked in the TaskTrail database table in SQL. The size of this database can have as much of an impact on performance as the actual running jobs. If the number of running jobs nowhere reaches 10,000 then it may be that the tombstone limit on entries tracked in the TaskTrail database is too conservative

By default the TaskGC entry in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings\SQL is set to tombstone entries at 90 day intervals. If a user is running a number of jobs on a daily basis that would cause the TaskTrail database to exceed this tolerance then this number can be reduced to tombstone entries more often.

By default, a job runs every 20 hours that will read this value and start the process to remove the jobs above the upper recommended limit, however it may take several cycles to bring the total job count down to acceptable limits.

Resolution

  Add the TaskGC registry value to limit the number of days that VMM retains job historical information.
  1.  Navigate to the following registry entry on your VMM server

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings\Sql

    2.    Create a DWORD value named “TaskGC” and set its value to a number less than 90 (decimal).

(TaskGC value controls the number of days that the events are stored in the DB)

    3.    Restart the VMM server several times to remove jobs until the total drops below 10,000

This registry key runs a stored procedure on the SQL Server once every 20h. In one run, a maximum of 50,000 objects in the jobs table is deleted. As there are multiple objects per job, this might not show a significant reduction immediately, but will succeed over several days to cleanup the DB.

Alternatively, you may run SQL Manager directly and run the following commands multiple times, until the execution does not find any more objects.

DECLARE @CutoffDateTime datetime
SELECT @CutoffDateTime = '4/1/2009'
EXECUTE [VirtualManagerDB].[dbo].prc_TR_GarbageCollect @CutoffDateTime

Note: You need to adjust CutoffDateTime to a date for your desired history length.

More Information

The VMMCA (Virtual Machine Manager Configuration Analyzer) will report a warning that the total number of jobs exceeds the recommended number.

Microsoft Internal Support Information

For immediate cleanup, run the following SQL commands. 

delete from [VirtualManagerDB].[dbo].[tbl_TR_DeletedObject]

delete from [VirtualManagerDB].[dbo].[tbl_TR_AuditTrail]

delete from [VirtualManagerDB].[dbo].[tbl_TR_SubtaskTrail]

delete from [VirtualManagerDB].[dbo].[tbl_TR_TaskTrail]

 
Steps to reproduce.

Product Bug Number:
Author ID (email alias):  mbriggs
Writer ID(email alias):  mbriggs
Tech Review ID (email alias):  jonjor, virtsee
Confirm Article has been Tech Reviewed: Yes
Confirm Article released for Publishing: Yes
Properties

Article ID: 2009348 - Last Review: Sep 27, 2010 - Revision: 1

Feedback