How to debug the Virtual Server service on a computer that is running Virtual Server 2005 or Virtual PC 2004, part 2 of 2

This article describes how to debug a virtual machine on a host computer that is running Microsoft Virtual Server 2005 or Microsoft Virtual PC 2004. This article is not a complete description about how to debug a computer. Instead, this article covers only the basic steps and information that you must use to obtain debug output in a virtual machine environment. This article is part 2 of 2 articles. For additional information about the other article that is part 1, click the following article number to view the article in the Microsoft Knowledge Base:

Warning If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor at your own risk.

Configure a user mode debug of the Virtual Server service

It is best to configure a user mode debug of the Virtual Server service (Vssrvc.exe) on the host computer even if you only need a dump file of the Virtual Server service. However, if you debug the Virtual Server service and the Virtual Server service has stopped responding, the host computer may stop responding. Therefore, two different methods are used to configure a user mode debug of the Virtual Server service. Although one of these methods is easier to implement, the other method provides the more reliable method to obtain the debugging information that you want.

Method 1: Use NTSD

We recommend this method. In this method you attach the Microsoft NT Symbolic Debugger (NTSD) to the Virtual Server service during startup and export the debug output through a kernel debugger that is attached to the host computer. In this scenario, if the host computer stops responding, you can continue to gather data. However, in this scenario you are performing a local debugging session. Therefore, you may have to install symbol files together with the debugging tools on the local computer. You may also have to install other programs such as Breakin.exe on the local host computer.

Note If you do not want to install these tools on the host computer, install them on a different computer and copy the installation folder to the host computer.

To configure the NTSD to obtain debug output from the Vssrvc.exe process, follow these steps:

1. Configure a memory dump file. For additional information about how to do this, click the following article number to view the article in the Microsoft Knowledge Base:
   303021  (http://support.microsoft.com/kb/303021/ ) How to generate a memory dump file when a server stops responding (hangs)
   Important Microsoft Knowledge Base article 303021 contains information about how to cause the computer to crash. The instructions in this article are used to obtain a Kernel mode dump file. However, we describe steps to obtain a user mode dump file. Therefore, do not follow the instructions to crash the virtual machine.
2. Download a kernel debugger from Microsoft. For information about how to do this, visit one of the following Web sites:
   http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx (http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx)
   
   http://www.microsoft.com/whdc/devtools/debugging/default.mspx (http://www.microsoft.com/whdc/devtools/debugging/default.mspx)
   We recommend that you use the latest version of the kernel debugger that is available. Additionally, we recommend that you perform a custom installation of the debugger. In this scenario, specify the installation path of \debuggers. This article assumes that the kernel debugger is installed in the \debuggers folder.
   
   Note the following commands and their usage:
   • g - You can obtain a memory dump file if the Virtual Server service stops or if you break in to the Vssrvc.exe process. In this scenario, use the g command to go to the service. If the service exits you are successful. If the service does not exit, you must wait for the next occurrence of a break.
   • gN - If you receive a break from the kernel, the host computer will stop responding. In this scenario, your kernel debug prompt changes to indicate that you are in the kernel debugger. In this scenario, type gN to restore your connection to the Virtual Server service.
     
     To restore the user mode debugger session, follow these steps:
     1. Log on the computer locally, and then run the Tlist command to obtain a list of process IDs (PIDs).
     2. Run the breakin.exe PID of the service that you want to break in to command.
     Note Break in to the Vssrvc.exe process and then type g to go to the service. This verifies that you can obtain a memory dump file of the Virtual Server service.
3. Configure the Virtual Service startup options. To do this, configure the following registry values:
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\vssrvc.exe
   
   Value name: GlobalFlag
   Value type: REG_DWORD
   Value data: 0x00000000
   
   Value name: Debugger
   Value type: String
   Value data: "C:\\debuggers\\ntsd -d -g"
   
   Note This registry value uses "escaped" backslash (\) characters.
4. Configure a local symbol tree.
   
   Note You can skip this step if you only want to obtain a memory dump file.
   
   Because you are breaking in to a process during the debug operation, you must configure a symbol tree on the local computer. To do this, obtain a memory dump file, open the memory dump file on the local computer, and then build a symbol tree. You can open the dump file in WinDbg and use the following path to build a symbol tree:
   SRV*c:\localstore*http://msdl.microsoft.com/download/symbols
   This remotely references the symbols. You can also build your own complete symbol tree offline by using the Symchk command. You can run this command over night to create a symbol tree. To create a symbol tree by using the Symchk command, follow these steps:
   1. Run the following command:
      symchk /r c:\Program Files\Microsoft Virtual Server /s SRV*c:\localstore*http://msdl.microsoft.com/download/symbols
   2. Run the following command:
      symchk /r c:\Windows\system32 /s SRV*c:\localstore*http://msdl.microsoft.com/download/symbols
      Note Each of these commands is one line.
5. Debug the virtual machine. Connect to the external debugger that you have configured to your debug computer. You will receive a different prompt when the Virtual Server service starts, and you will see information about the Virtual Server service in the debugger. If the service stops responding, you can debug or obtain a simple memory dump file by running the following command:
   .dump /ma /u c:\debuggers\vssrvc.dmp
   After you run this command, run the g command.
   
   Note If this is a scenario where the service stops responding, run the dump command two more times.
6. Restart the host computer. After the host computer has restarted, transfer the event logs or the current version of the MPS reports output together with the dump file to Microsoft. Because the dump file may be large, compress it by using an archive program such as WinZIP.

Method 2: Use CDB

Use Microsoft CDB debugger to obtain debug output from the vssrvc.exe process. To do this, follow these steps:

1. Download a kernel debugger from Microsoft. To do this, visit one of the following Web sites:
   http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx (http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx)
   
   http://www.microsoft.com/whdc/devtools/debugging/default.mspx (http://www.microsoft.com/whdc/devtools/debugging/default.mspx)
   
   http://www.microsoft.com/whdc/devtools/debugging/default.mspx (http://www.microsoft.com/whdc/devtools/debugging/default.mspx)
   We recommend that you use the latest version of the kernel debugger that is available. Additionally, we recommend that you perform a custom installation of the debugger. In this scenario, specify the installation path of \debuggers. This article assumes that the kernel debugger is installed in the \debuggers folder.
2. Configure the Virtual Service startup options. To do this, configure the following registry values.

   Registry entry 1

   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\vssrvc.exe
   
   Value name: GlobalFlag
   Value type: REG_DWORD
   Value data: 0x00000000
   
   Value name: Debugger
   Value type: String
   Value data: "C:\\debuggers\\ntsd -d -g"
   
   Note This registry value uses "escaped" backslash (\) characters.

   Registry entry 2

   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
   
   Value name: Auto
   Value type: REG_DWORD
   Value data: 1
   
   Value name: UserDebuggerHotKey
   Value type: REG_DWORD
   Value data: 0x00000000
   
   Value name: Debugger
   Value type: String
   Value data: "c:\\debuggers\\cdb.exe -p %ld -e %ld -g -c \".dump /ma /u C:\\WINDOWS\\user.dmp;q\""
   
   Note This registry value uses escaped backslash characters. Additionally, this registry value contains two sets of quotation marks at the end of the path.
3. Debug the Virtual Server service, restart the Virtual Server computer, and then send in the debug output together with the event logs or with the current version of MPS reports.
   
   Important Obtain the MPS report data or the event log information after the dumps are obtained.