MSDPM.EXE crashes during Garbage Collection causing DPMDB growth in Data Protection Manager 2007

Symptoms

Every day at midnight, System Center Data Protection Manager 2007 (DPM) starts a couple of maintenance tasks which are referred to as Garbage Collection. During Garbage Collection, a couple things happen such as:
  • Expired Recovery Points are removed
  • Entries from the DPMDB database that are older than 33 days are removed
A problem that can occur is that during Garbage Collection, the DPM service (MSDPM.EXE) crashes and the Garbage Collection job never completes. As result, stale data within DPMDB is never completely removed and you end up with a large and ever growing DPMDB database.

Cause

This can occur if SQL encounters a deadlock and in order to resolve it, one of the store procedures involved in the deadlock is killed by the SQL Engine. 

You can find something similar to the following in MSDPMCurr.errlog if this is occurring:

NOTE If DPM was installed in its default location, this file will be in C:\Program Files\Microsoft DPM\DPM

Attempt 1 failed with exception Microsoft.Internal.EnterpriseStorage.Dls.DB.NonFatalDbException: exception ---> System.Data.SqlClient.SqlException: Transaction (Process ID 63) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
     at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
     at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
     at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
     at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
     at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
     at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
     at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
     at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
     at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ExecuteNonQuery()
     --- End of inner exception stack trace ---
  *** Mojito error was: DatabaseNonFatalError; 0; None
  --- SqlException details -----------------
  System.Data.SqlClient.SqlException: Transaction (Process ID 63) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
     at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
     at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
     at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
     at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
     at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
     at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
     at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
     at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
     at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ExecuteNonQuery()
  Error = 1205
  Index #0
  Source: .Net SqlClient Data Provider
  Number: 1205
  State: 52
  Class: 13
  Server: <Server_Name>
  Message: Transaction (Process ID 63) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
  Procedure: prc_PRM_GarbageCollect
  Line: 714
  --- Original Command -----------------
  dbo.prc_DLS_GarbageCollect
  --- Caller StackTrace -----------------
  FileName:; Method:ExecuteNonQuery(); lineNo:0; ilOffset:8.
  FileName:; Method:CleanupDlsDatabase(); lineNo:0; ilOffset:119.
  FileName:; Method:OnStart(); lineNo:0; ilOffset:260.
  FileName:; Method:Start(); lineNo:0; ilOffset:2.
  FileName:; Method:Execute(); lineNo:0; ilOffset:8.
  FileName:; Method:ChangeState(); lineNo:0; ilOffset:62.
  FileName:; Method:Process(); lineNo:0; ilOffset:338.
  FileName:; Method:Function(); lineNo:0; ilOffset:16.
  FileName:; Method:Run(); lineNo:0; ilOffset:95.
  FileName:; Method:PerformWaitCallbackInternal(); lineNo:0; ilOffset:28.
FileName:; Method:PerformWaitCallback(); lineNo:0; ilOffset:40.


Resolution

This is a known issue in System Center Data Protection Manager 2007. To have this issue addressed, please contact Microsoft Support (http://support.microsoft.com/contactus/).

More Information

This issue is addressed in System Center Data Protection Manager 2010 and later.
Properties

Article ID: 2758637 - Last Review: Oct 10, 2012 - Revision: 1

Feedback