Multicast support enabled for the cluster heartbeat

Windows Server 2003 cluster nodes exchange multicast heartbeats rather than unicast heartbeats. In Microsoft Windows NT 4.0 and Microsoft Windows 2000, all heartbeat messages were unicast.

Multicast networking refers to transmitting network data to a group of interfaces instead of a single destination interface. Only the other interfaces in the same multicast group can use the transmitted data. Using multicast technology permits several unicast messages to be replaced with a single multicast message. Unlike broadcast, multicast traffic does not have to be flooded throughout the network, reducing the chance that unnecessary CPU cycles are expended filtering traffic on nodes outside the cluster. As the number of nodes increases, the ability to replace several unicast messages with a single multicast message improves node performance and decreases network bandwidth consumption.

Clusters that are comprised of fewer than three nodes will not send multicast heartbeats. The Cluster service also does not send multicast heartbeats in mixed version clusters. For instance, a four-node Windows Datacenter Server cluster that has three nodes running Windows Server 2003 Datacenter Server and the remaining node running Windows 2000 Datacenter Server reverts to unicast heartbeats. Otherwise, the Cluster service automatically configures and uses multicast for the cluster heartbeats by default.

The cluster Network Driver (Clusnet.sys) is the core cluster driver that has been rewritten to support multicast. Multicast is not used for all intracluster communications. Multicast messages replace unicast messages in two components of Windows Clustering:

• Heartbeats: The Windows Clustering failure detection engine is based on a scheme whereby nodes send heartbeat messages to other nodes. Specifically, for each network interface, a node sends a heartbeat message to all other nodes with interfaces on that network. Heartbeat messages are sent every 1.2 seconds. In the common case where each node has an interface on each cluster network, there are N * (N - 1) unicast heartbeats sent per network every 1.2 seconds in an N-node cluster. With multicast heartbeats, the message count drops to N multicast heartbeats per network every 1.2 seconds, because each node sends 1 message instead of N - 1. This represents a reduction in processing cycles on the sending node and a reduction in network bandwidth consumed.
• Regroup: The Windows Clustering membership engine executes a regroup protocol during a membership view change. The regroup protocol algorithm assumes the ability to broadcast messages to all cluster nodes. To avoid unnecessary network flooding and to properly authenticate messages, the broadcast primitive is implemented by a sequence of unicast messages. Converting the unicast messages to a single multicast message will conserve processing power on the sending node, reduce network bandwidth consumption, and more accurately reflect the intentions of the algorithm.

Configuration

Multicast configuration primarily involves selecting and agreeing upon a multicast group address for each cluster network. A multicast group address can be any IPv4 class D address (224.0.0.0 to 239.255.255.255). The multicast group address can be configured in one of three ways:

• From a Multicast Address Dynamic Client Allocation Protocol (MADCAP) server
• Automatically by self-selecting a multicast address
• Manual configuration by the administrator

From a MADCAP Server

If a cluster administrator did not manually configure a multicast address, the cluster tries to obtain a multicast address from a MADCAP server. MADCAP servers lease multicast addresses much like DHCP servers lease unicast addresses.

The following is the procedure that the Cluster service uses to obtain a multicast address when there is a MADCAP server on the network:

1. Send a MADCAP request to obtain a list of multicast scopes.
   
   A scope is a range of multicast addresses that are configured so that traffic that is sent to these addresses can be limited to a subset of the network. A MADCAP server may offer addresses in several scopes. When you request a MADCAP address, the cluster must specify the scope from which the address should be selected.
2. From the scopes advertised by MADCAP servers, determine those that are valid for the cluster network.
   
   A scope is valid for the cluster network only if it meets both of the following criteria:
   • The scope is available on the local interface for this network (as opposed to a different network interface on this node).
   • The scope is not in the range of multicast addresses reserved for single-source traffic (232.*.*.*).
3. Rank the valid scopes that remain according to the following criteria:
   • Scope falls in the administrative local scope (239.255.*.*)
   • Proximity of MADCAP server with regard to the cluster node (same subnet as cluster node)
   • Scope has the largest range of addresses of those enumerated
   • Scope has the lowest time-to-live attribute (TTL) of those enumerated
   
   
4. Request an address lease from each scope in ranked order until a lease is obtained. Much like a DHCP assigned address, the MADCAP assigned address has an expiration time by which the lease must be renewed.
   
   If no MADCAP server responds or if there are no valid scopes with available leases, the Cluster service automatically selects an address.

Automatically by Self-Selecting a Multicast Address

If an administrator did not specify an address and if MADCAP requests fail, the Cluster service automatically selects an address. The Cluster service randomly selects an address from a subset of all IPv4 class D addresses. The range of multicast addresses to use for self-selection is determined as follows:

1. Check the cluster registry to see if administrator specified a multicast address range. If no range was specified, the administrative local scope (239.0.0.0 to 239.255.255.255) [RFC 2365] is chosen as the default range.
2. Apply the following exclusions to the address range:
   • Upper 256 addresses in both the administrative local scope and the administrative organizational scope, which are reserved for relative allocations as per RFC 2365 [RFC 2365]
   • The single-source scope (232.*.*.*)
   • Any valid scopes recently obtained from a MADCAP server
   If the range of addresses for self-selection is not empty after the exclusions are applied, the Cluster randomly selects an address. The Cluster configures self-selected addresses to expire in 24 hours in case a MADCAP server is deployed on the network (or a lease becomes available).

Manual Configuration by the Administrator

An administrator can manually configure a multicast address for a cluster network. If a class D IPv4 address is manually configured, that address is used without question. A manually configured address persists, even if the cluster service is stopped and restarted, until the administrator manually configures a different address. If an administrator specifies an invalid multicast address (for example, not an IPv4 class D address), the cluster service tries to automatically configure a valid multicast address.

An administrator can also disable multicast either on the whole cluster or for individual networks so that heartbeat traffic reverts to unicast. If multicast is disabled for a network, there will be no attempt to obtain a multicast address and the Cluster service uses unicast until the network is reconfigured.

An administrator configures multicast by modifying the private properties of individual networks and the cluster. To view the private properties for a network with regard to the multicast configuration; use the /privproperties parameter without any options. For example, to see the heartbeat multicasting properties for the cluster network PrivNet, type the following at the command prompt: The following table lists all of the configuration parameters for a particular cluster network with regard to multicasting. The writable column indicates whether or not the parameter can be set with Cluster.exe.




The MulticastClusterDisabled parameter is writable and indicates whether multicast should be used for all cluster networks. Each cluster network MulticastDisabled setting overrides the global cluster MulticastClusterDisabled setting. The Cluster Service on all nodes must be stopped completely before the cluster service is started on each node for the global disable to take effect. The following table describes whether heartbeat multicasting is disabled for a specific network as determined by the MulticastDisabled value for that network and the MulticastClusterDisabled value for the whole cluster:

* - "Not Set" implies that the parameter is absent

Examples

The following examples are sample uses of Cluster.exe to handle multicast settings:

To turn off heartbeat multicasting for the cluster network named PrivNet, type the following at the command prompt: To set the multicast address for network PrivNet to 224.1.2.3, type the following at the command prompt: To set the range of multicast addresses for network PrivNet from which an address should be automatically chosen when no MADCAP server is detected to 239.255.0.0-239.255.0.255, type the following at the command prompt: To disable multicast for the whole cluster, type the following at the command prompt:

Key for Signing and Verifying Messages

The final step for the multicast configuration is agreeing on a key for signing and verifying messages. The key must be secret, but it must be agreed upon by all cluster nodes. Periodically, for each cluster network where multicast is not disabled, one cluster node generates a large random number to serve as the new multicast key. That key is encrypted and distributed to all the other cluster nodes. When each node receives the new key, each node stops signing messages with the old key. Multicast keys are never recorded in the cluster database and are always encrypted before distribution across a network.

• If the multicast heartbeats ever fail (for example, because of misconfigured hardware), the cluster will automatically and silently revert to unicast heartbeats without user intervention
• If a cluster holds a MADCAP lease on a multicast address when an administrator manually configures an address or disables multicast, the MADCAP server is contacted to release the address. There is no attempt to release an address if the cluster is shut down (if intracluster communication breaks down, it can be impossible to determine if other nodes are alive and using the address)
• Enforcement of non-writable multicast private properties is only implemented on Windows Server 2003 nodes. Windows 2000 nodes will not prevent read-only properties from being set
• The multicast TTL for outgoing messages is set to 1. Windows Server 2003 Windows Clustering clusters cannot span subnets. Setting the multicast TTL to 1 instructs multicast-capable routers not to forward intracluster multicast messages outside the subnet. For geographically dispersed clusters, VLAN hardware must be configured to forward multicast traffic
• If automatic configuration is necessary and MADCAP requests time out, multicast configuration can be delayed by more than 60 seconds. Because MADCAP requests are invoked asynchronously, multicast configuration will not delay bringing any cluster resources online
• ClusNet uses the HMAC SHA-1 hashing algorithm to sign and verify multicast messages

For more information about the Private Network and multicast, see the following articles and documents: