Symptoms
Assume that one or more clustered generic services are configured on a Windows Server 2008-based two-node cluster. The Cluster service cannot bring the clustered generic services online if one of the following operations is performed:
-
A restart of both nodes in the same timeframe.
-
A restart of the cluster service on both nodes in the same timeframe.
-
A restart of one node and a restart of the cluster service on the other node in the same timeframe.
-
A restart of one node when the other node is offline.
-
A restart of the cluster service on one node when the other node is offline.
Note Any of these operations will cause the cluster to be reformed (The term "form" in the context of a cluster is defined in the "Cause" section).
A manual attempt to bring the generic service resource online from the Failover Cluster Management administrative console will fail, and the following error message is reported in a message box:
The action ‘Bring this resource online’ did not complete. An error occurred while attempting to bring the resource ‘service name’ online. The object already exists.
Additionally, the following error messages are logged in the cluster log:
[RES] Generic Service <Simple Service>: Service 'SimpleService' is controlled by another resource. Error: 5010.
[RHS] RhsCall::Perform_NativeEH: ERROR_OBJECT_ALREADY_EXISTS(5010) because of Resource service name: Open call failed.
Notes
-
This issue can be triggered by a nonclustered service that is set to start automatically on one or more nodes of the cluster. The nonclustered service interacts with the cluster service by using cluster API.
-
This issue also affects clusters that have multiple nodes. The following generic operations will cause this issue to occur when the cluster has multiple nodes (an n-node cluster):
-
A restart of all nodes in the same timeframe.
-
A restart of the cluster service on all nodes in the same timeframe.
-
A restart of i nodes and a restart of the cluster service on the other j nodes, possibly when k nodes are offline, in the same timeframe (where i, j and k are any positive, or zero for k, and n =i+j+k).
-
Cause
Initialization of Microsoft Cluster Service defines a special phase called the form phase. This phase is executed by the first node that starts in the cluster, to create the cluster entity. Other nodes that restart and find that the cluster is available do not have to form a new cluster entity. Instead, they will join the cluster that was previously formed by another node.
Note It is irrelevant whether the cluster has restarted because of a node restart (a restart of the operating system) or a cluster service restart.
The form phase has to briefly start all the resource monitor processes (Rhs.exe) for configured resources. These resource monitor processes should be stopped by the cluster service as soon as the cluster is formed, and immediately restarted for regular operations. Because of an issue in the cluster service code, the first set of resource monitor processes remains in the running state. However, these resource monitor processes do not perform any actions. In rare circumstances, the first set of resource monitor processes prematurely access certain system objects, such as semaphores. Their inability to close correctly causes a failure of the second set of resource monitor processes if the objects they access were designed to be uniquely accessed or created.
In this situation, the following error message is logged in the in the cluster log:
ERROR_OBJECT_ALREADY_EXISTS(5010) because of Resource service name: Open call failed.
Notes
-
This issue is more likely to occur when there is a nonclustered service that is set to start automatically, and this nonclustered service interacts with the cluster service by using the cluster API. This behavior was observed to be a trigger that causes the first set of resource monitor processes to prematurely access system objects.
-
When this issue occurs, the number of Rhs.exe processes on the forming node is double the usual amount.
Resolution
Hotfix information
A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.
If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.
Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft Web site:
http://support.microsoft.com/contactus/?ws=supportNote The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.
Prerequisites
To apply this hotfix, you must be running Windows Server 2008 Service Pack 2 (SP2). Additionally, you must enable the Cluster service.
For more information about how to obtain a Windows Server 2008 service pack, click the following article number to view the article in the Microsoft Knowledge Base:
968849 How to obtain the latest service pack for Windows Server 2008
Registry information
To use the hotfix in this package, you do not have to make any changes to the registry.
Restart requirement
You must restart the computer after you apply this hotfix.
Hotfix replacement information
This hotfix does not replace a previously released hotfix.
File information
The global version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.
Windows Server 2008 file information notes
Important Windows Vista hotfixes and Windows Server 2008 hotfixes are included in the same packages. However, only "Windows Vista" is listed on the Hotfix Request page. To request the hotfix package that applies to one or both operating systems, select the hotfix that is listed under "Windows Vista" on the page. Always refer to the "Applies To" section in articles to determine the actual operating system that each hotfix applies to.
-
The files that apply to a specific product, SR_Level (RTM, SPn), and service branch (LDR, GDR) can be identified by examining the file version numbers as shown in the following table.
Version
Product
SR_Level
Service branch
6.0.600
2.
22xxxWindows Server 2008
SP2
LDR
-
The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed separately in the "Additional file information for Windows Server 2008 " section. MUM files and MANIFEST files, and the associated security catalog (.cat) files, are extremely important to maintain the state of the updated components. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.
For all supported x64-based versions of Windows Server 2008
|
File name |
File version |
File size |
Date |
Time |
Platform |
|---|---|---|---|---|---|
|
Clussvc.exe |
6.0.6002.22735 |
3,905,024 |
31-Oct-2011 |
18:36 |
x64 |
|
Cluswmi.dll |
6.0.6002.22735 |
281,088 |
31-Oct-2011 |
20:28 |
x64 |
For all supported IA-64-based versions of Windows Server 2008
|
File name |
File version |
File size |
Date |
Time |
Platform |
|---|---|---|---|---|---|
|
Clussvc.exe |
6.0.6002.22735 |
6,544,896 |
31-Oct-2011 |
22:07 |
IA-64 |
|
Cluswmi.dll |
6.0.6002.22735 |
500,224 |
31-Oct-2011 |
23:41 |
IA-64 |
Workaround
Workaround 1
This workaround describes a permanent configuration that is not affected by this issue. However, this workaround has drawbacks. Therefore, it is better to install the hotfix that is proposed by this article.
To implement this configuration, follow these steps:
-
Open Failover Cluster Manager.
-
Enter the clustered application that owns the affected generic service.
-
Select the affected generic service resource and open its properties.
-
Select the Advanced Policies tab.
-
Clear Run this resource in a separate Resource Monitor.
-
Click OK to confirm the new settings.
Notes
-
You may have to restart the cluster service on the forming node again if you are experiencing this issue.
-
The common resource monitor is the only resource monitor that is stopped correctly when the form phase is completed.
-
Configuring a resource to run in the common resource monitor can affect reliability of the whole cluster solution. Additionally, a fault in the resource DLL for the clustered generic service could automatically trigger the resource to be isolated to run again in its own separate resource monitor. Therefore, this workaround will no longer take effect. This workaround should be considered an emergency solution.
Workaround 2
To temporarily resolve the issue after it has occurred, follow these steps:
-
Identify the forming node that caused the issue. You can do this by examining Task Manager and looking for the node that has approximately two times as many Rhs.exe processes as the other nodes.
-
On the forming node, restart the cluster service from the Services administrative console. Or, restart the forming node.
Note These steps remove the error condition after it has occurred. These steps do not prevent the issue when another forming phase is executed (a restart of the whole cluster, a restart of the cluster services on all nodes, or other conditions that are described in the "Symptom" section).
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
More Information
For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684 Description of the standard terminology that is used to describe Microsoft software updates
Additional file information
Additional file information for Windows Server 2008
Additional files for all supported x64-based versions of Windows Server 2008
|
File name |
Amd64_0f288b636f949ca7b5871c574947fbea_31bf3856ad364e35_6.0.6002.22735_none_4af4063dda22f866.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_1d35928fb881debe8754e32ec0e543a0_31bf3856ad364e35_6.0.6002.22735_none_9a318061f73157df.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_22737939f1f03df96c058b1302086fc8_31bf3856ad364e35_6.0.6002.22735_none_32064e3621901794.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_266c6d6ef4d6ad687a3ea6598c382e1f_31bf3856ad364e35_6.0.6002.22735_none_a6584ad0883d1482.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_3d16792d6ae87f0c063e8c7c0fa1b828_31bf3856ad364e35_6.0.6002.22735_none_3dcaffb0dc6e189b.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_43bdaaed096bcb811745b292e6f9ebf0_31bf3856ad364e35_6.0.6002.22735_none_0e3d3cf07acc37a2.manifest |
|
File version |
Not Applicable |
|
File size |
715 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_4ae91ceda92f65a75197af835b8d8314_31bf3856ad364e35_6.0.6002.22735_none_d1508cb060981498.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_50b8beb2edd2c178527a9367577f2ca3_31bf3856ad364e35_6.0.6002.22735_none_bf83d8fcff6f075e.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_53041006b6c5f2efba8857a09acc2851_31bf3856ad364e35_6.0.6002.22735_none_596351b546fce6f2.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_7076af66afc932fb48c4812efe8be586_31bf3856ad364e35_6.0.6002.22735_none_47a225cd7e68fb0d.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_798087d1e3942201e9f2399ab1eaafed_31bf3856ad364e35_6.0.6002.22735_none_bccb9bd8fa56ad5d.manifest |
|
File version |
Not Applicable |
|
File size |
1,072 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_79e2db9f39f85a76ed9e285270c4aee8_31bf3856ad364e35_6.0.6002.22735_none_6e993694ac2a8450.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_7d36038f3da4d27aa02a6e2af79970fa_31bf3856ad364e35_6.0.6002.22735_none_3b3fd508f25711ac.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_8782a8ee76d79f894dc26567b88432c9_31bf3856ad364e35_6.0.6002.22735_none_7cfc9a3bab2dbf3a.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_8d4cec6a67367576eed93683ced8381c_31bf3856ad364e35_6.0.6002.22735_none_60f986a844f75f2d.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_975d63f07c5851612b586650d6162905_31bf3856ad364e35_6.0.6002.22735_none_ed51fc46959b0153.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_a872b3fad37097750710a0a19aebaadf_31bf3856ad364e35_6.0.6002.22735_none_d9b89d0345245f10.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_afc4576c35fdd9cb69ce745fe5a46e7b_31bf3856ad364e35_6.0.6002.22735_none_20bd3424be07c8c9.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_b1cb1a0d4d6a8796f72b9a1ffbefbc6c_31bf3856ad364e35_6.0.6002.22735_none_43df17495d9820f8.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_b4149a94ba11d1a736023b472b14f80e_31bf3856ad364e35_6.0.6002.22735_none_9fe0d5d9cc9ea077.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_c6af30f46be6812eddc938d60bb43582_31bf3856ad364e35_6.0.6002.22735_none_c98390dfa81a2f57.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_cf53933e6a4231d4d96563d80f5d93c0_31bf3856ad364e35_6.0.6002.22735_none_358ba51e62ea13b3.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_d047a8cc6398cbafac27a4d9e92f6922_31bf3856ad364e35_6.0.6002.22735_none_769864f59614aee0.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_eaa55585d31d2afc4e74e5584b75c03e_31bf3856ad364e35_6.0.6002.22735_none_77db0bb0197069f3.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_f075cb5cfee9b30edc4f33cb3222cdbf_31bf3856ad364e35_6.0.6002.22735_none_b38900ddfbef4290.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_fe7461f6c616e62d09eecfc3fdf4f236_31bf3856ad364e35_6.0.6002.22735_none_d8811601d9a3944b.manifest |
|
File version |
Not Applicable |
|
File size |
723 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Amd64_microsoft-windows-f..overcluster-clussvc_31bf3856ad364e35_6.0.6002.22735_none_1834f6487040eca9.manifest |
|
File version |
Not Applicable |
|
File size |
15,155 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
20:53 |
|
File name |
Amd64_microsoft-windows-f..overcluster-cluswmi_31bf3856ad364e35_6.0.6002.22735_none_1ae2ef8c6e868686.manifest |
|
File version |
Not Applicable |
|
File size |
11,257 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
20:51 |
|
File name |
Wow64_microsoft-windows-f..overcluster-clussvc_31bf3856ad364e35_6.0.6002.22735_none_2289a09aa4a1aea4.manifest |
|
File version |
Not Applicable |
|
File size |
4,510 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
17:25 |
Additional files for all supported IA-64-based versions of Windows Server 2008
|
File name |
Ia64_132978c842133764c8a357e18f83caee_31bf3856ad364e35_6.0.6002.22735_none_c68d895becbf1f4d.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_46495fcad8377594209112a5330fa5c6_31bf3856ad364e35_6.0.6002.22735_none_21c5ba2727bbc47b.manifest |
|
File version |
Not Applicable |
|
File size |
713 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_5c1c8865e300177682e88fb4b67c65b1_31bf3856ad364e35_6.0.6002.22735_none_63d730836d03b199.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_c228ceae91f089ad7c67c408f156aa0f_31bf3856ad364e35_6.0.6002.22735_none_c702c6503d39376b.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_c72d15776fe3bff043c5502c3792c8dd_31bf3856ad364e35_6.0.6002.22735_none_57d0ea1295a94e23.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_dd9779fc0e8634b6d02fdd852f8e6481_31bf3856ad364e35_6.0.6002.22735_none_e5f7e948d11d8e88.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_e79dd28131232d26fc551b43a42e2dce_31bf3856ad364e35_6.0.6002.22735_none_ac692374bdeccd94.manifest |
|
File version |
Not Applicable |
|
File size |
1,070 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_e9ef9067baeabfeef6cb7033a54fff21_31bf3856ad364e35_6.0.6002.22735_none_0b7dc6dc55732f41.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_f87d0d55cd9bc8bfaccebf54a70261f0_31bf3856ad364e35_6.0.6002.22735_none_dfd0dad80e64a7b2.manifest |
|
File version |
Not Applicable |
|
File size |
721 |
|
Date (UTC) |
01-Nov-2011 |
|
Time (UTC) |
02:05 |
|
File name |
Ia64_microsoft-windows-f..overcluster-clussvc_31bf3856ad364e35_6.0.6002.22735_none_bc17febab7e1846f.manifest |
|
File version |
Not Applicable |
|
File size |
15,136 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
23:59 |
|
File name |
Ia64_microsoft-windows-f..overcluster-cluswmi_31bf3856ad364e35_6.0.6002.22735_none_bec5f7feb6271e4c.manifest |
|
File version |
Not Applicable |
|
File size |
11,245 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
23:58 |
|
File name |
Wow64_microsoft-windows-f..overcluster-clussvc_31bf3856ad364e35_6.0.6002.22735_none_2289a09aa4a1aea4.manifest |
|
File version |
Not Applicable |
|
File size |
4,510 |
|
Date (UTC) |
31-Oct-2011 |
|
Time (UTC) |
17:25 |
