How to troubleshoot the “Needs Attention”, “Not Responding”, and “Access Denied” hosts in System Center Virtual Machine Manager

Applies to: System Center 2012 Virtual Machine ManagerMicrosoft System Center 2012 Virtual Machine Manager Service Pack 1Microsoft System Center 2012 R2 Virtual Machine Manager More

Home users: This article is only intended for technical support agents and IT professionals. If you're looking for help with a problem, please ask the Microsoft Community.

Summary


This article discusses how to troubleshoot the “Needs Attention”, “Not Responding”, and “Access Denied” host status in System Center 2012 and later versions of Virtual Machine Manager. Note that any referenced articles also apply to System Center 2012 and later versions of Virtual Machine Manager.
The “Needs Attention”,“Not Responding”, and “Access Denied” hosts in the VMM console occurs because the VMM Server is unable to communicate with the host machine or components (WMI, WinRM, etc.) on the host machine that are used to communicate with the VMM Server are not functioning correctly.

The following are common errors that are logged in the Jobs views in the VMM console when the host status is “Needs Attention”, “Not Responding”, or "Access Denied".

Error (2911)
Insufficient resources are available to complete this operation on the servername.domainname server. (Not enough storage is available to complete this operation (0x8007000E))

Error (2912)
An internal error has occurred trying to contact an agent on the servername.domainname server. (No more threads can be created in the system (0x800700A4))

Warning (2915)
The Windows Remote Management (WS-Management) service cannot process the request. The object was not found on the server (servername.domain.com). Unknown error (0x80041002) or Unknown error (0x80338000)

Error (2916)
VMM is unable to complete the request. The connection to the agent servername.domain.com was lost. Unknown error (0x80338126) or Unknown error (0x80338012)

Error (2927)
A Hardware Management error has occurred trying to contact server servername.domain.com. Unknown error (0x803381a6)

Warning (12710)
VMM does not have appropriate permissions to access the Windows Remote Management resources on the server (servername.domain.com). Unknown error (0x80338104)

Warning (13926)
Host cluster servername.domain.com was not fully refreshed because not all of the nodes could be contacted. Highly available storage and virtual network information reported for this cluster might be inaccurate.

Error (20506)
Virtual Machine Manager cannot complete the Windows Remote Management (WinRM) request on the computer servername.domain.com.

Warning (13926)
Host cluster servername.domainname was not fully refreshed because not all of the nodes could be contacted. Highly available storage and virtual network information reported for this cluster might be inaccurate.

Error (406)
Access has been denied while contacting the computer servername.domain.com.

Perform the steps documented in the More Information section to identify the cause of the “Needs Attention”, “Not Responding”, or "Access Denied" host status.

More Information


Step 1: Check the Health status of the Host

To check the Health status of a host, perform the following steps:
  1. Open the VMM Console.
  2. Select the Fabric view, right-click the host that’s experiencing issues and chose properties.
  3. Within the host properties, select Status.
  4. Select the category that has the Red exclamation to view the error details.
For more information on the host health check feature, please reference the following blog: http://blogs.technet.com/b/scvmm/archive/2011/12/19/host-properties-new-in-vmm-2012-expanded-health-checks.aspx

Step 2: Verify the VMM service account is a member of the local administrators group on the host

  • If the VMM service is running under a domain account, verify the domain account is a member of the local administrators group on the host.
  • If the VMM is running under the local system account, verify the computer account is a member of the local administrators group on the host.

If the VMM service account is removed from the local administrators group on the host, this issue could be caused by a “Restricted Groups” group policy.

To resolve this issue, perform one of the following steps:
  • Add the VMM service account to the Administrators "restricted groups" group policy setting.
  • Create a new organizational unit in the domain, move the host computer object to the new OU and then configure the new organizational unit to block policy inheritance.

Step 3: Check for corrupted performance counters

Check the Application event log on the host to see if the following event is logged:
Log Name:      Application
Source:        Microsoft-Windows-LoadPerf
Event ID:      3012
Description:
The performance strings in the Performance registry value is corrupted when process Performance extension counter provider. The BaseIndex value from the Performance registry is the first DWORD in the Data section, LastCounter value is the second DWORD in the Data section, and LastHelp value is the third DWORD in the Data section.

If the Event ID 3012 is logged on the host machine, perform the steps documented in the following knowledge base article to rebuild the performance counters:
2554336 How to manually rebuild Performance Counters for Windows Server 2008 64bit or Windows Server 2008 R2 systems

Step 4: Check the Svchost.exe process of the Windows Remote Management service

VMM depends on the Windows Remote Management service for host communication. Therefore, the "Not Responding" status is very likely to occur because of an error in the underlying Windows Remote Management communication between the VMM server and the host computer. In this scenario, the host status is "OK" shortly after you restart the host computer. However, the status changes to "Not Responding" after three to four hours, and jobs on the VMM server fail and return an error that resembles the following:
Error (2927)
A Hardware Management error has occurred trying to contact server servername.domain.com.
Unknown error (0x803381a6)
Additionally, if you stop the Windows Remote Management service at a command prompt, this process takes much longer than usual to be completed. Sometimes, it can take up to five minutes to stop.

This problem can occur if the shared Svchost.exe process that hosts the Windows Remote Management service is experiencing issues.

To resolve this problem, configure the Windows Remote Management service to run in a separate Svchost.exe process. To do this, open an elevated command prompt, type the following command and then press Enter.
sc config winrm type= own
Note Make sure that you type the command exactly as it appears here. Notice the space after the equal sign (=) symbol.

If the command is completed successfully, you should see the following output:
[SC] ChangeServiceConfig SUCCESS

Step 5: Increase the default values for WinRM

As a best practice, run the following command lines on the SCVMM host and all the Hyper-V hosts that are being managed by SCVMM (and press Enter after each line).
Admin CMD

Winrm quickconfig
winrm set winrm/config @{MaxTimeoutms="1800000"}
winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="1500"}winrm set winrm/config/winrs @{MaxConcurrentUsers="100"}
winrm set winrm/config/winrs @{MaxProcessesPerShell="100"}
winrm set winrm/config/winrs @{MaxShellsPerUser="100"}

PowerShell

set-item "WSMan:\localhost\Plugin\WMI Provider\Quotas\MaxConcurrentOperationsPerUser" 400

*You must restart WINRM (for the WINRM changes) and restart WMI (for the SC config setting), or reboot the server for the changes to take effect.

When you add Untrusted Hosts or Perimeter/Workgroup hosts, make sure that the LocalAccountTokenFilterPolicy registry is set to 1. For information about how to make this setting, see Description of User Account Control and remote restrictions in Windows Vista.