Cluster Storage Validation May Fail After Power Outage

Applies to: Hyper-V Server 2008Microsoft Hyper-V Server 2008 R2Windows Server 2008 Datacenter More


The Cluster Validation is a suite of tests which verify that the hardware meets Windows Server Failover Clustering requirements and that it is properly configured.  The Validate Disk Arbitration test verifies the shared storage properly supports the persistent reservation SCSI commands and conducts a test that verifies disk arbitration functionality.  A random persistent reservation key is generated during the test for each disk.  Under certain conditions the test may generate duplicate persistent reservation keys on two or more different nodes, which may cause the test to incorrectly fail.

More Information

If two or more cluster nodes are simultaneously powered on or rebooted prior to the test, it may result in the hardware TickCount to be the same.  An identical TickCount across multiple machines may cause the random number generator used to create the key during the test to generate the same persistent reservation key on different nodes.  A TickCount on one node within 16 milliseconds of the TickCount on another node could result in a duplicate key being generated.  This issue is most likely to occur after a power outage to a datacenter and all servers simultaneously regain power and boot.  This issue is unique to the Cluster Validation test and does not impact production clustered disks, as the Physical Disk resource uses a different mechanism to generate persistent reservation keys.

To resolve this issue, reboot all of the nodes in the cluster in a staggered fashion, waiting at least 1 second between POST on each cluster node. (A 1 second delay between POST on the cluster nodes will avoid the 16 millisecond window that can lead to duplicate keys).  Then run the Cluster Validation tool again. If this Validate Disk Arbitration test continues to fail after a staggered power on or reboot, please contact your storage vendor, as the failure indicates that the storage is not handling arbitration properly. This will be addressed in a future version of Windows.