Provide Feedback on this Broadcast

Note This document is based on the original spoken Support WebCast transcript. It has been edited for clarity.

Allen Mock: Welcome to Clustering Microsoft® Exchange Server 2003. Some of the things that you should learn from this Support WebCast today (slide 2) are some of the changes in Exchange Server 2003 clustering, and also gain a little bit of insight into different migration and deployment strategies from moving from Exchange 5.5 to an Exchange 2000 Server to Exchange Server 2003 clustering. Just a quick note for those tuning in, this WebCast does assume prior knowledge of Exchange 2000 Server clustering. There will be some links later on to some additional documents for that.

On our agenda today (slide 3), we're going to do a quick recap of some basic cluster terminology that most of you who are tuning in to this WebCast should already have. We're also going to discuss some changes coming from Exchange 2000 clustering and, once again, the different migration and deployment methods and strategies.

Diving right in to our cluster terminology (slide 4), a node is a physical server or just a physical computer that's going to be part of the cluster system. A shared disk is going to be an external disk that is accessible by all nodes, that's generally a SCSI or fiber channel, and it is not an IP-based solution.

A cluster (slide 5) is a group of nodes working together as single systems, to ensure high availability of applications. In this particular WebCast, we are dealing with Exchange Server 2003, and the cluster should ensure higher availability than on a stand-alone server. Now the key thing to keep in mind here is that this is primarily hardware redundancy and not data redundancy; there is no data replication. That would all depend on the physical hardware such as disk arrays. As you see here, we have two cluster nodes standing by the pink computers, and in between we have two shared disks, and those are your physical disks.

In this WebCast, we're also going to be referring to virtual servers (slide 6). A virtual server is nothing more than a group that is inside of a cluster. It needs a minimum of four cluster resources. We have an IP address, a network name, a physical disk, and an application. In this case, the application is Exchange. When we have Exchange as that application, we refer to it as an EVS or an Exchange Virtual Server.

Down below in the diagram there we see another computer, which is just a one-cluster node, which currently owns an Exchange Virtual Server. In the top white box we have a physical disk, an IP address, a network name, and the Exchange resource. Also, on the individual nodes, there's going to be a copy of the Exchange binaries on the local drive. You see with the arrow pointing that is the external disk or the shared disk between the two nodes. That's where all of our Exchange data is going to be, such as our databases and log files.

In clustering with Exchange, we have two different types of clustering (slide 7). We have Active/Active and Active/Passive. First, Active/Passive by definition is when you have fewer Exchange Virtual Servers than the total number of nodes in the cluster. So if you have a two-node cluster and you have one Exchange Virtual Server, you're in an Active/Passive configuration. Anything above two nodes, up to eight with Windows Server™ 2003, you'll be forced into an Active/Passive configuration, meaning you will have one passive node at all times unless the node fails. Active/Passive is the Microsoft recommendation for performance reasons, which we will address in a moment.

Active/Active refers to when the number of Exchange Virtual Servers is equal to or greater than the number of nodes inside of your cluster. So if you have a two-node cluster and you have two Exchange Virtual Servers, even if they're running on the same node and you have one node that's sitting idle, you're still in an Active/Active configuration because you can never guarantee that the EVSs will stay on a single node at one time, and that one node will always be passive.

Moving into why we recommend Active/Passive over Active/Active (slide 8). The main reason for this is for performance reasons and virtual memory fragmentation. There can only be one Store.exe process running per node in a cluster. Each Exchange Virtual Server is going to have its very own instance inside of that store process when they're running on the same machine. You can think of virtual memory fragmentation as being almost like hard disk drive fragmentation.

Essentially, we allocate and release various sizes of memory within a process, and the virtual address space can become fragmented over time, which can eventually cause an Exchange Virtual Server to fail. In order to come online on the other node, we need a 10-megabyte (MB) block of contiguous virtual memory in order for that EVS to successfully come online. If we do not have that on the other node because it also is experiencing virtual memory fragmentation, that EVS will not come online and will stay in a failed state. Therefore, this is why we recommend Active/Passive as opposed to Active/Active.

Moving along to some of the requirements for Exchange Server 2003 clustering (slide 9), we need to have, at a minimum, the operating system needs to be Windows® 2000 Advanced Server, which will give you up to two nodes for clustering. If you're running Datacenter for Windows 2000, you can have up to four nodes. Windows 2000 must have SP4 installed or SP3 with hotfix 329938. If you're running Windows Server 2003 Enterprise Edition, you can have up to eight nodes in a cluster; Datacenter will also allow you to have eight nodes in a cluster.

The hardware requirements in Windows Server 2003 are slightly different from Windows 2000 (slide 10). For Windows Server 2003, we have the new Windows Catalog online, which you can look at to verify that your hardware is on the Hardware Compatibility List. It's very important to note that the cluster Hardware Compatibility List is completely different than for stand-alone servers. You can't just take separate pieces of equipment and put them together and call them a cluster, and consider it on the HCL. Even if all the individual pieces are on the HCL, the entire solution must be certified in order to be on the cluster HCL.

We also need to have the Microsoft Distributed Transaction Coordinator or DTC installed on each cluster node, and you also get a cluster resource for that. This is due to workflow dependencies within Exchange.

We have some setup changes from Exchange 2000 to Exchange 2003 (slide 11). We now have block removal of a cluster node if there is currently an EVS running on that. The setup /disasterrecovery switch is now blocked. This was not blocked in Exchange 2000. One thing — this is a key point here — we prevent Exchange 2003 from being the first non-legacy server in an Exchange 5.5 site. This is because the site replication service or SRS database cannot be located on a cluster server. You must introduce a stand-alone server into your 5.5 site first. That can either be Exchange 2000 or Exchange 2003, so this can hold the SRS and talk to your 5.5 servers.

Also, in Exchange 2003 setup, we no longer create the POP3 and IMAP4 resources by default. This was part of our emphasis on security in Exchange 2003. But if you are doing an upgrade from Exchange 2000, these protocols will remain in place, and you can remove them later if you're not using them. If you're doing a fresh install, these protocols will not be created. Article 818480 will give you the steps you must use to create the resources in the cluster if you add those to your Exchange Server cluster.

Keeping on with the security changes in Exchange Server 2003 (slide 12), the Cluster service account no longer needs Exchange full administrator rights in order to install Exchange. Any user in Windows 2000 with Cluster service accounts could have potentially destroyed your Exchange 2000 environment, because they could have deleted the System Attendant resource out of the Cluster Administrator. The Cluster service account needed to be delegated full admin rights. This is separated now with Exchange 2003, so that a regular user that has Exchange full admin rights does not need Cluster service account rights; everything is run under the context of their security principle.

We now also support IPSec between front-end and back-end servers. Article 821839 will explain how to configure that. Running IPSec can slow down failover slightly, but there is a registry key you can use that will change the time-out values. This article describes how to change that value. This is due to the time it takes for IPSec to renegotiate security associations.

For authentication changes with Exchange 2003, we now require a Kerberos-enabled Network Name (slide 13). You can do this in Windows 2000 on a cluster through the command line by running the Cluster.exe utility, but in Windows Server 2003 this is available in the GUI with a couple of check boxes. Article 235529 will describe how to use Cluster.exe, and check if Kerberos is enabled on your Windows 2000 server. With Windows 2000 SP3, we also require the hotfix 329938; with Windows 2000 SP4, not hotfix is required.

One of the major changes that we have as well in Exchange Server 2003 for clustering is we have a flattened dependency tree, which will allow us to achieve a faster failover time (slide 14). As you see here with Exchange 2000, we have three resources that are dependent on the System Attendant. We have the routing engine, the information store, and the MTA. The resources that are dependent on the information store are essentially all the protocols and Microsoft Search.

With Exchange Server 2003, we shifted everything to be dependent on the System Attendant, because in Exchange 2000, before the information store could come offline, we would need to wait for IMAP4, SMTP, HTTP, POP3, and MS Search to all come offline before the information store could come offline. Now by making all of these dependent on the System Attendant, they can all come offline at the same time, and then the System Attendant can come offline gracefully and move over to another node or prepare for maintenance.

A new feature of Windows Server 2003 that Exchange Server 2003 can take advantage of is volume mount points (slide 15). We support this for data files only. You cannot put binary files on a mount point in a cluster. Article 318458 will describe the benefits of using volume mount points inside of a cluster and describe how to use those.

Our biggest benefit to using volume mount points in this situation is going to be if you're running a four-node or an eight-node cluster. What happens is in these four-node and eight-node cluster scenarios, each node automatically reserves three drive letters. You have A is generally the floppy, if a floppy exists. You have the C for the local system, and also possibly a D for CD-ROM drive or whatever drive letter has been allocated for that. So that would leave 23 drive letters to be shared amongst all the cluster nodes. If you follow the standard performance and disk allocations for Exchange, you would separate your log files and your database files accordingly for performance reasons and for redundancy. This could take up a significant number of drive letters when the number of EVSs grows inside your cluster. So by creating mount points, which is essentially nothing more than a folder or a directory underneath a physical drive letter, so if you had an R drive you could create a mount point underneath it and call it R\Data, to allow you to store all the data on one physical disk, but it's just represented by a folder or a directory inside of the file system structure.

You can tell the difference in Windows Explorer between a mount point and a regular folder, because it will actually have a different disk icon inside of Windows Explorer. This will definitely allow us to save drive letters and achieve four-node and eight-node clustering a little bit easier.

Moving on to the setup portion of today's Support WebCast (slide 16), we're going to first discuss installing a brand new Exchange Server 2003 cluster. The first thing that you would obviously need to do is install and configure Windows Cluster service, and we're also going to assume, at this point, that you've already created a group for an Exchange Virtual Server that already contains an IP address, a network name, and any physical disks and mount points that need to be added to this particular group. At that point, you can run Setup.exe on both nodes. You want to make sure not to do this at the same time; do not run it simultaneously.

At this particular time, there are no service packs available, but in the future, if service packs are available, you can install a service pack and bring the node up to that service pack level when it's installed. Once again, do not install the service packs simultaneously on both nodes.

After you've installed the binaries on both nodes, you're going to go ahead into the Cluster Administrator and create the System Attendant resource in the Exchange group. We're going to do a brief walk-through of this, and you would repeat these steps twice if you wanted to do an Active/Active configuration.

On our first slide here (slide 17), when you right-click inside of the group, inside of Cluster Administrator, or right-click on the group and select New Resource, you'll come up with this particular screen. You want to choose Microsoft Exchange System Attendant in the Resource type drop-down box, and in the Name box call it something logical such as the Exchange Virtual Server name, in this case, EVS01 System Attendant. You also have the option to select which group you would like to put this particular object or resource in, if there is more than one group.

When you click Next, you'll arrive at the Possible Owners page (slide 18). If you do not see a node in your cluster listed as an available node, it means that the Exchange binaries have not been installed (or have not been successfully installed) on that node. You should look into that and investigate why that is. Move any possible owners to this particular group over to the right-hand side by highlighting them on the left, clicking Add, and at that point you can click Next.

This is where we come to our dependencies (slide 19). You need to have all physical disks listed in this particular section here; this includes mount points. Otherwise, we could end up getting negative 1022 errors when we fail over. Also, we need to have the network name added in here, and notice we do not have the IP address resource in here. This is because the network name is dependent on the IP address resource. Then you can click Next after that is complete.

You have the choice of selecting an administrative group that this Exchange Virtual Server is going to be installed into (slide 20). You will always see this dialog box when the first EVS is created, but will not allow you to select an alternate group if there is only one administrative group. If you're creating more than one EVS as well, for your second time around, this dialog box should default and will default to the first administrative group, because you cannot have multiple EVSs in the same cluster in different administrative groups.

Then you click Next and you get the option to choose the routing group that this Exchange Server should belong to, only if there is only one routing group available (slide 21). Once again, if you're installing multiple EVSs in a cluster, this will pick it up and keep it with the first routing group that you had selected for the first EVS, so they all need to be in the same routing group.

If you click Next — we actually omitted a slide here, where we would have the option to select a data path (slide 22). If you have more than one drive selected, or more than one physical disk drive added into a particular cluster group and listed as a dependency, you'll get a drop-down box that allows you to select any of those physical disks and a path location to store your Exchange Server data.

After you click Next on the data path portion of that, you will come up to this Summary screen (slide 23) that essentially gives you a summary of all the options you selected. Make sure you review this page and that all the options are correct before proceeding and clicking Finish. If you need to make any modifications, just simply click Back and you can make the change, then click Next until you get back to this page with the Finish button. When you click Finish, you get this lovely dialog box (slide 24) that tells you that the System Attendant has been created successfully.

Inside of Cluster Administrator (slide 25), you'll see all of your resources inside of your EVS, everything located on the right-hand side. The top resource that is currently offline is the System Attendant, and at this point all you'll need to do is right-click on System Attendant and say Bring Online, and you should be up and running with your Exchange Server. If you click on the group itself on the left-hand side and say Bring Online, that will actually bring all the resources in that particular group online, as opposed to just individual resources active at that time.

We're going to briefly touch on a migration from Exchange 5.5 to Exchange Server 2003 (slide 26). In-place upgrades are not supported on clusters going from Exchange 5.5 to Exchange 2003. A lot of this is due to the SRS, but hardware limits, I would say, definitely come into play here if you're running a 5.5 cluster on, say, Windows NT® 4.0. Also, the clustering hardware is probably several years old and would probably need to be upgraded as well. As mentioned previously, you must install a stand-alone server into the same administrative group to run the Site Replication Service database, and this can either be Exchange Server 2000 or 2003.

At that point, once you have the SRS up and running in that same administrative group, you can install your Windows cluster, Windows 2000 or 2003, and install Exchange just like we did in the previous section. After you have your Exchange Server running online on your cluster, you can then proceed to move mailboxes from Exchange 5.5 to your Exchange 2003 cluster. At that point, you can choose to decommission your 5.5 server, the stand-alone SRS server, and be left with the cluster.

Migrating from Exchange 2000 to Exchange 2003 is a little bit different (slide 27). It's a rolling upgrade. For those of you who are familiar with Exchange 2000 clustering, this is essentially the same way that we would update service packs, except there are a couple of additional steps here that you have be aware of. If you're running an Active/Active configuration, you'd want to move all of your Exchange Virtual Servers to one particular node, and at that point upgrade the passive node by running Setup on that server, and reboot as necessary. I recommend rebooting just to make sure the version of Exres.dll got updated correctly. At this point, you can move the Exchange Virtual Server to that upgraded node after it has been rebooted.

At that point, you will notice that the Exchange resources do not come online. This is a little bit different than Exchange 2000, where the upgrade was automatic and we would try to bring the System Attendant online. What you'll need to do is right-click on the group or on the System Attendant object and select Upgrade Exchange Virtual Server. At that point, you see a dialog box that says that your server has been upgraded successfully, and you can bring the System Attendant online if you wish. On the node that has not been upgraded yet, once again, run the Setup.exe and restart the server as necessary, which we recommend for Exres, and then redistribute the EVSs if applicable, if you're running an Active/Active configuration.

This next slide here (slide 28) just shows us what the dialogs will look like when you right-click the System Attendant, the options. The second one down, this is Upgrade Exchange Virtual Server.

Now we'll discuss the removal of Exchange 2003 Virtual Servers (slide 29). It's a little bit different than the process in Exchange 2000, where we would just bring the System Attendant resource offline and then delete the System Attendant, which would delete our Active Directory® object at that point as well. With 2003, it's slightly different where we would bring the System Attendant offline, and then right-click and say Remove Exchange Virtual Server.

If you happen to use the Exchange 2000 method on an Exchange 2003 server and just delete the System Attendant object, we're going to log a 1027 event in the application log and the Active Directory objects are still going to remain in place. What you need to do at this point if this is the case, is to re-create the System Attendant resource inside of Cluster Administrator, and then remove it correctly. Because to the permissions changes that were mentioned earlier, you're going to need to log on with an account that has Exchange full administrator rights so it can access that portion of the Active Directory.

This next slide (30) shows the remove feature, the first item listed in the context menu. You can set this by either right-clicking on the System Attendant or on the Properties, or by right-clicking on the group itself.

Some additional resources (slide 31) to help you get on your way with Exchange Server 2003 clustering are the Exchange Server 2003 planning guides; one of our deployment guides, which not only covers clustering scenarios, but stand-alone servers and various migrations and strategies for that; and our Exchange Server 2003 Technical Library.

That's going to wrap up our WebCast for today on Exchange Server 2003 clustering, which is going to lead us into our question and answer portion of this WebCast. Today I'm joined by my colleague, Evan Dodds, who is a Technical Lead with Exchange Admin here; and Tim McMichael, who is on our third-tier support. They're going to assist me today with some of our Q&A session.

Otto Cate: Excellent. Thank you very much for the presentation, Allen. Before we jump into the Q&A, I'd like to share a couple real quick program notes with our listeners here. If you'd like to have a copy of the PowerPoint® slides for your reference, they are available for download, along with the on-demand streaming archive. Those will both be available on the site at http://support.microsoft.com/webcasts/, and simply click on today's session.

The Q&A portion of the Support WebCast is really intended to encourage further discussion of the topic that we addressed today. In addition, one-on-one product support issues are really outside the scope of what we're able to address during our time frame in this forum. So if you do find that you need some more complex technical assistance, feel free to contact a Support Professional either via Web or via phone. With that, let's go ahead and jump into the questions.

The first one is: Why is Active/Passive better than Active/Active?

Allen: This is due to the virtual memory fragmentation, which was mentioned previously in this WebCast. Since Store.exe can only have one process on each node, each EVS will then have its own instance that runs inside that process space. What happens is eventually the virtual address space for that Store.exe process can become so fragmented that it can fail over to the other node, and if memory is being fragmented on the second node, which is also running in Exchange Virtual Server, we might not have a large enough block of memory space to grab in order to bring the information store process online. We need a 10-MB contiguous chunk of virtual address space in order for that EVS to successfully come online on that second node. Therefore, you could end up having one of your EVSs down at that point in time, which is the reason we don't recommend running in an Active/Active configuration.

Otto: Maybe in a nutshell, can you describe in which scenarios Active/Active is recommended?

Evan Dodds: There really aren't any scenarios where we would recommend Active/Active. In pretty much every case Active/Passive is going to give you a higher amount of scalability, and a higher amount of reliability as well.

Otto: Which product will replace Exchange 2000 Instant Messaging?

Evan: That question is probably going to end up being outside of the scope of this session. There is a replacement in the works, but it's not tied in with the clustering portion of the product.

Otto: The next question here: My company is in the process of preplanning for a migration from Exchange 5.5 over to a 2003 cluster. Can you point me to a Microsoft project template that outlines everything that we need to do to be successful in that project?

Allen: The links at the end of the WebCast there, specifically the "Deployment Guide," should address some of those areas.

Evan: I'm not aware of any project documents that would cover the whole process. It would be difficult to create a document like that, because each and every installation is going to be very different. But overall, the planning guide and the deployment guide that are marked at the end of the presentation will cover a lot of different technical scenarios that may help you out.

Otto: Is there a sizer tool available from Microsoft?

Evan: Yes. There are some performance and sizing tools that are available from the http://www.microsoft.com/exchange/ portion of the Web site. I don't know that any of them are specific to clustering, but with regards to Exchange scaling overall, there are definitely some tools available there. Perhaps we can go back in and put into the transcript afterwards the exact paths to those tools.

Follow-up information: We have a wealth of information about performance tuning and troubleshooting Exchange 2000 in the "Exchange Performance Tuning" document at http://www.microsoft.com/exchange/techinfo/administration/FineTune.asp. The tuning and sizing information available at this link remains relevant to Exchange 2003. There is also a "Performance and Scalability Guide for Exchange 2003" that we expect to release in March 2004. Best practices and sample scenarios are included in the recently released planning, deploying, and administration guides for Exchange 2003. These are available in the Exchange Server 2003 Technical Documentation Library at http://www.microsoft.com/exchange/library/.

Otto: The next question here: Are there any known problems with upgrading both Exchange and the operating system to 2003? I'm assuming, essentially, they're asking like side-by-side or one after the other.

Evan: There aren't really any known problems with having them both be upgraded, but you would definitely want to be aware of the order in which you upgrade them. Of course, if you start with either Exchange 5.5 or Exchange 2000, for the sake of this example let's say you start with Exchange 2000; you would not be able to upgrade the operating system first to Windows Server 2003. So you'd need to be mindful that your Exchange would have to upgraded to Exchange 2003 first, and then follow that with the operating system upgrade.

Otto: The next question here: Is there any reason you keep the Exchange binaries on local drives as opposed to putting them over on shared disks?

Allen: Yes. This is, I guess, from some lessons learned from Exchange 5.5, where we would have the binaries on a shared disk. We were not able to do a rolling upgrade process similar to what we have now. Having the binaries on the local drives allows for higher availability, so that you can essentially have a minimal downtime when failing over in Exchange Virtual Server to perform an upgrade on those binaries, whereas in 5.5 you would have to bring down the entire Exchange server to update the binaries.

Otto: We've got several users that are asking about best practices and case studies and such. Are these available in these additional resources here like, for instance, in the planning or deployment guides?

Allen: I'm not aware that there are any case studies specific to Exchange 2003 clustering at this time, but there are definitely scenarios that are described inside the planning and deployment guides, and perhaps the scenarios would be sufficient to answer questions.

Otto: The next question here: What SAN configurations are supported at the time, and does it require certification from Microsoft to support the configurations?

Allen: Basically, your cluster configuration is going to be dependent on using a certified solution based on the Hardware Compatibility List for Windows Server 2003 and for Exchange 2003. So as long as the solution itself that you're implementing appears on that as a clusterable solution, you will be in a supported configuration from the Microsoft standpoint. I do want to reiterate that, again, it is not necessarily the components that you use to build the cluster are separately listed on the HCL as components that are supported, as much as the entire implemented solution is listed.

Otto: The next question here: I thought that I had heard that you could not remove any nodes if an EVS was running on any of the nodes in 2003. Is that correct? If so, how would you recommend replacing old machines in a cluster?

Evan: Essentially, you would just fail over or move the EVS to another node, and then at that point you can remove the node from the cluster itself. If you need to replace that node with another piece of hardware, you could evict it at that point in time.

Otto: Is it possible to move from two Exchange 2000 EVSs to one EVS on 2003?

Allen: Yes, absolutely. That's basically the process that we use to go from an Active/Active configuration to an Active/Passive configuration on a two-node cluster. So the process is similar to what we would do in Exchange 2000. You would move all of your users and connectors and whatnot off of the second EVS. You'd want to be mindful that you're not moving the users off of the first EVS. We need to make sure that it's the second EVS, the one that does not have the message transfer agent, the one that we're actually going to be removing. But once you've got that second EVS ready to be removed, you just go in and right-click on the group and you select the Remove Exchange Virtual Server option. That will clean up the Active Directory objects, and it will also clean up the cluster resource objects. It's essentially the equivalent of deleting the System Attendant in Exchange 2000.

Otto: In a mixed Exchange 5.5 and 2000 environment, is it necessary to upgrade the ADC before upgrading a cluster?

Allen: In terms of Exchange 2003, absolutely, and that's not necessarily a cluster limitation. In Exchange 2003, we have a prerequisite that blocks the installation of Exchange 2003 if you haven't already upgraded your ADC to the Exchange 2003 version. So the same thing would hold for a stand-alone Exchange 2003 installation as well. But yes, on a cluster, it will block as well.

Otto: The next question here: Has the VM fragmentation been resolved with servers with many stores? We're still seeing a lot of fragmentation on Exchange 2000 servers with many stores, for instance, eight plus.

Allen: The VM fragmentation issues haven't been resolved per se, but there have definitely been changes to the way that VM is allocated and the way that Exchange deals with VM fragmentation. Changes were included in Exchange 2003. If you're having VM fragmentation issues on systems with many stores in Exchange 2000 or on Exchange 2003, I suggest that you take a look, there is a tuning guide out there, and there are some documents that have information on tuning in a cluster as well. We can find some of those document links and provide them in the transcript. Take a look through that, and there are some settings that can be set to alleviate or eliminate VM fragmentation issues.

Follow-up answer: The issue is not "resolved" by Exchange 2003, but the impact should definitely be lessened. There are a number of tuning steps that you can take to aid in better scaling on servers with many stores. These changes are not really specific to clustering; they apply to highly scaled stand-alone servers as well. For Exchange 2003, take a look at Knowledge Base article 822180, "Exchange Server 2003 and Virtual Memory Fragmentation" for more information about how to best tune your Exchange 2003 server. Because you also mention Exchange 2000, please note that there is a subset of these tuning settings that you can also use on Exchange 2000 running on Windows 2000 servers. Take a look at Knowledge Base article 325044, "HOW TO: Troubleshoot Virtual Memory Fragmentation in Exchange 2003 and Exchange 2000" for more information about some additional settings you may want to implement on your Exchange 2000 server.

Evan: Slide 8 has a link to a Support WebCast that Michael Palermiti delivered, regarding Exchange performance tuning, which addresses virtual memory fragmentation as well.

Otto: The next question here: You mentioned earlier that hardware for cluster machines needs to be validated. I found hardware considerations for planning cluster systems on http://www.microsoft.com/windows2000/. We're kind of curious if that needs to be updated for Exchange 2003 or are those guidelines sufficient for Exchange 2003 clusters? The exact address is http://www.microsoft.com/windows2000/hpc/cluscomp3.asp.

Allen: I'm not terribly familiar with that particular link, but what I'll tell you is that, as was mentioned earlier in the slides as well, in order to be validated as a cluster configuration, it would need to pass from a Windows perspective rather than an Exchange perspective. In order to pass from a Windows perspective, in Windows 2000 there is the cluster HCL, and there is an equivalent hardware validation process for Windows Server 2003, and there is a link to that on slide 10 that you could reference for your particular configuration.

Otto: Are the some recommendations concerning SG in databases design for one Active/Passive cluster scenario within the "Planning and Deployment Guide" there?

Allen: That's probably going to depend very much on what the customer's actual design configuration is going to be. There are advantages to using multiple storage groups, but there are also disadvantages to multiple storage groups. Generally, the fewer storage groups that you can get away with using, the less likely you are to run into virtual memory fragmentation issues; but the specific design that you'll need for your cluster is probably going to be heavily dependent on how you design your Exchange servers overall.

Otto: The next question we've got here: I thought that the service dependency improvements for cluster with Exchange 2003 were supposed to address some of the issues with virtual memory fragmentation. Is this true or not true?

Allen: The changes that were made to the cluster service dependency and resource dependency in Exchange 2003 are actually in terms of failover performance rather than virtual memory fragmentation. It shouldn't actually affect virtual memory fragmentation at all, but what it does do is it allows the SMTP virtual server resource and the information store virtual server resource to go offline at the same time, whereas in Exchange 2000 the information store had to wait until the SMTP resource had gone offline before it could begin to go offline. These two resources are the two resources that take the greatest amount of time during a failover.

Otto: The next question here: Are there any major benefits to using a cluster versus using replication software?

Allen: I'm not entirely clear on what the question is referring to as replication software, but if the question is actually referring to like a software driver that replicates disks, I know there are some third-party products out there that can do that. The primary reason that we, Microsoft, would give as a benefit of using our cluster software solution versus that replication software is that the cluster solution, as it has been described today, is a tested solution that is supported Microsoft PSS. I can't speak to any of the specific replication software products, but the potential deficiency of using replication software is that it hasn't been tested, and it's very difficult to presume the outcome of using that solution. There are a lot of issues with bandwidth and latency that may get in the way with a replication solution but, again, I'm not entirely sure what the third-party solutions would provide, so I can't comment too much on that.

Otto: The next question here: If we're running Exchange 2000 and then install 2003 Server into Active Directory, can we do a simple mailbox move to the Exchange 2003 Server?

Evan: Yes. Using Active Directory Users and Computers, just as you'd move a mailbox between two Exchange 2000 servers, you can simply move the mailbox between Exchange 2000 and Exchange 2003. They're the same method. You can also do it in ESM, if you have the 2003 System Management totally installed. So you're not just limited to using Active Directory Users and Computers for your mailbox moves.

Otto: This question might be related to licensing or better handled via licensing, but the question is: Do you have to purchase Exchange Enterprise for each node?

Evan: Yes, you do. You have to have Exchange Enterprise Edition in order to install into a clustered solution.

Otto: Do you know approximately how many clients an Active/Passive one-to-one cluster can handle in Exchange 2003 over 2000?

Allen: The answer to that question probably depends a bit on how you have your typical user defined. Here at Microsoft, of course we have very heavy user profile; we're very active in e-mail. So on an Active/Passive cluster here, our user count would be dramatically less. But relatively, between Exchange 2000 and Exchange 2003, there have been some changes made that would allow a higher user count for an Active/Passive cluster on Exchange 2003. Some of those changes are in Exchange 2003, and some of those changes are in the updated version of the Outlook® client.

What you would need to do in your environment is do some testing to determine what your user profile is, and then test with some of the tools like Jetstress ESP or the new LoadSim 2003, and see what sort of scalability you can achieve on your hardware given the user profile you defined.

Otto: The next question here: What is the average time needed to do a failover in an Active/Passive scenario, and how can we improve that?

Allen: There are a great lot of things that go into doing a failover in an Active/Passive scenario. It's sort of like the last question. It's difficult to really give an average time, because it's going to very much depend on how many resources exist in your cluster and how active they are. The typical things that make the failover time slower in an Active/Passive scenario are the time it takes to flush the SMTP queues to disk. That's what controls how long the SMTP Virtual Server resource and cluster takes to go offline, and then additionally the time it takes to flush the log depth into the information store resource. Again, both of those things will depend on how many messages are queued up for SMTP, and on how many log files are outstanding for the information store.

I've seen a great variation in the time for a failover. I would say a failover of several minutes is probably good. A failover time of 8, 10, or 12 minutes is probably getting to be a bit long. But what's an acceptable failover time for you will probably depend what an acceptable failover time is for you. There are a number of things you can do to increase the performance during failover, and there are some registry changes you can make that will increase failover performance, possibly at the cost of slightly decreased performance while the service isn't running. Those should be covered in the "Exchange 2000 Deployment Guide," and the "Exchange 2003 Deployment Guide."

Otto: The next question: Do I still have to cycle the services on fail-back like in 2000 Active/Passive mode?

Allen: I'm guessing that this refers to the case in Exchange 2000 when we would have a virtual memory fragmentation problem. We would fail over in an Active/Passive scenario and fail to the passive node, and then the resources would come online. If memory became fragmented again, we would try and fail back over to that initial node, but the Store.exe process was still running and the virtual address space was still fragmented. So what we would need to do is restart that service through the Service Control Manager to ensure a clean address space. The answer to that question is that you no longer have to do that with Exchange 2003. You actually just cycle those services to give it a fresh address space.

Evan: And further, when we fail from the first node, the active node, to the passive node, we actually take those resources offline on the first node. So in Exchange 2000, it was not unusual to see that the services would stay online on the initial node, even after you had failed over to the second node. But in Exchange 2003 you'll see that those resources actually go offline after the failover, if there are no virtual servers still running on that first node.

Otto: The next question: Due to additional costs of having a clustered solution, one proposal that has been given in the past is to run the passive node off of less expensive hardware, understanding that you would run in a slower or less performance-heavy state if on the second node. Are there any concerns running on differing hardware on each node?

Allen: The same limitations would apply for symmetrically designed clusters versus asymmetrically designed clusters. As long as that cluster solution is on the clustered HCL, then it would be a suitable cluster for us to use for Exchange, and that's with regards to what it is. If it's a clustered HCL solution, then it's good to go.

Otto: Are there some specific instructions to deploy full-text indexing on an Active/Passive cluster?

Allen: To do full-text indexing on a cluster is not really too different than on a stand-alone machine. I guess, there are two things you'd want to be aware of. On a cluster, of course, either Active/Passive or Active/Active, your full-text indexes will default to being on the shared disk, which is slightly different than on a stand-alone machine. But as far as the actual process to go through and configure it, it should be very similar.

The other thing you may want to be aware of is that full-text indexing on a cluster is somewhat more complicated to deal with (especially when failures and other such issues occur) than it would be on a stand-alone machine. There are some KB articles out there that you may want to be aware of, that will assist you with resolving clustered issues with full-text indexing.

Otto: Does Exchange Server 2003 support mixed mode for Exchange the same as 2000 did, and is that still required for Exchange 5.5 compatibility? In addition to that, can you go from mixed mode in Exchange 2000 to native mode in Exchange 2003?

Evan: No. There's no real difference. Again, it all relates to the fact that, if you're going to run in the mixed mode environment, your SRS has to be installed in a stand-alone server. Even with 2003 in the environment, it is still a native mode switch, which would be 2000 or 2003 servers with all your legacy 5.5 removed from the environment.

Otto: The next question here: Does Exchange 2003 clustering support Kerberos against an EVS?

Allen: I'm not entirely clear on the question. We do use Kerberos, but primarily the big change in using Kerberos, from an Exchange clustering perspective, is that now we require that the network name resource provided by Windows be Kerberos-enabled. Fundamentally, the big change there is that we will now see a computer account be created in the Active Directory. This is something that will be a little bit new for people who are used to, for instance, the 9175 event that we saw in Exchange 2000, if there was an account for the Exchange Virtual Server in the Active Directory. Now we do require that there is an account in the Active Directory, so that we can do Kerberos authentication against that security principle, so some of those KB articles that are out there for Exchange 2000 will no longer apply the same to Exchange 2003. Hopefully that answers the question.

Otto: The next question here: You had mentioned that IPSec slowed the transfer between two nodes. Are there any numbers on what this performance hit is, or maybe any recommendations for hardware-based IPSec that can help mitigate that performance impact?

Allen: Yes. Since discussions about the performance of IPSec will probably be outside of the scope of this WebCast — there may be some information out there about it, but it probably wouldn't fall underneath the domain of clustering. It's just a new feature that we do support, and we do allow now back-end clusters connected to front-end servers with IPSec, but as far as performance, we'd probably need to look elsewhere to get information on that.

Otto: The next question here: Now that all services are dependent on the SA, if MS Search fails, which it has in our current environment, would that initiate a failover for the Exchange group? If so, can we disable that feature?

Allen: This is actually a very interesting question. It's important to note that if your MS Search resource fails in Exchange 2000, where it's dependent on the information store, depending on how you configured it, if it's in the default configuration, even that case would still cause your entire Exchange Virtual Server to fail over to the other node. So there isn't any change to that behavior in Exchange 2003 just because of the dependency change. But note that MS Search is really only used for full-text indexing. So if you're not utilizing full-text indexing within that Exchange Virtual Server, there's absolutely no harm in removing the effect to the group setting, of even deleting the MS Search resource. There are a number of different things you can do to isolate MS Search or remove MS Search from your Exchange Virtual Server if you're not using it.

I would also recommend, definitely upgrade to the latest service packs, either upgrading to the post-SP3 rollup for Exchange 2000 on your cluster, or upgrading to Exchange 2003. There have been a number of changes that have resolved issues we've seen that cause MS Search to fail in clustered environments.

Otto: The next question here: Do you know if there are any plans for more than eight-node clusters in a Windows 2000 Server?

Allen: No. I'm not aware of any plans for greater than eight-node clusters. Eight-node clustering right now is the most nodes that we support in Windows Server 2003, and therefore it's the most nodes that we support for Exchange 2003.

Otto: We have a follow-up message about one of mailbox move questions we had addressed a little earlier. The Mailbox Move function performance is also supposed to be much improved in Exchange 2003 as well. Do you know if that's accurate?

Allen: The question is probably beyond the scope, because there are several things that have changed in that respect, when upgrading to Exchange 2003. One of the things that I can tell you is that it is now a multi-threaded move process, as well as being a scheduled move process. So with respect to those two things, they are two improvements in the Move Mailbox process that are brought about by the implementation of Exchange 2003.

Otto: I know that this question might actually be a little general, or can depend a lot on the specific scenario, but the question is: What is the time needed to upgrade an Exchange 2000 node to Exchange 2003?

Allen: That definitely is a little bit general. It's going to depend on the environment. There are way too many factors that you can put in there like CD-ROM speed, what kind of media you're installing from, and also the performance tuning that you've already put into place for failover, and how long your EVS is going to be offline when performing the rolling upgrade to Exchange 2003. My best recommendation would just be to test it. A test environment is going to be the best way to figure out how long it's going to take you to do it in production.

Otto: Would an upgrade from an existing Exchange 2000 Active/Active to an Exchange 2003 Active/Active be supported, given that Active/Active is not recommended for 2003?

Evan: Yes, absolutely. We still do support for two-node clusters, the Active/Active scenario. That means that if you have an Exchange 2000 server and you move both virtual servers to one node, you can do the rolling upgrade to Exchange 2003 on the other node. Now the end of that question, given that Active/Active is not recommended for 2003, it's important to note that Active/Active is not recommended for Exchange 2000 either. It's basically not our recommended scenario for either Exchange 2000 or Exchange 2003, because of the virtual memory issues that you may run into in Active/Active.

Otto: Is there a list of each registry-tuning key available with their explanations? I've heard a lot of Exchange 2000 tuning tricks won't actually work in 2003.

Allen: There's not really a master list of registry tuning keys, unfortunately. But depending on what sort of tuning you're trying to do, if you're trying to tune failover performance versus, say, virtual memory configuration, there are a number of different sources for information. I'd say your best source for information, for the registry keys that are known, are the planning guide and the deployment guide; both are available in the Technical Library. You'll see that there's a lot of tuning information for performance and for virtual memory, and potentially for failover tuning as well in those documents.

Otto: Regarding rolling upgrades from an Exchange 2000 cluster to 2003, if you upgrade the passive node and then perform the EVS upgrade, after you've failed to the new 2003 node, if you were to run into any issues are there perhaps any stability problems we might encounter by failing back to a non-upgraded Exchange 2000 node?

Evan: Once you've upgraded the passive node and you've failed the EVS over to that previously passive node that now runs Exchange 2003, and you've issued the Upgrade command, you'll be unable to fail back over to the original node that has Exchange 2000 on it without first upgrading it to Exchange 2003.

Otto: Is there a rollback available for a 2003 rolling upgrade?

Evan: There is no rollback once it has been upgraded to Exchange 2003. In other words, once you've right-clicked and you've selected Upgrade Exchange Virtual Server, there is no rollback.

Otto: The next question here: Can you provide, perhaps, some more information on mount points? Currently I've only found two KB articles regarding mount points, and I'd like to answer questions about disaster recovery and dismantling of clusters. Do you know if there are any resources that we can use that might provide that information?

Evan: It actually sounds like these are possibly two different questions. The first question on mount points, there are some resources on mount points out there. I imagine the two KB articles that you found are probably two of the main resources. Most of the resources that I have found have been some Windows KB articles. If you could actually send back some more specific questions on mount points, other than that, I can give an overview of mount points, maybe to supplement what Allen said earlier.

Basically, the point of mount points is to decrease the number of drive letters that you need to have for each of your clustered Exchange Virtual Servers, the point of which being that all nodes in the cluster need to have the exact same set of drive letters, because they're all together in the same cluster. So the recommendations that we make for Exchange 2000 and Exchange 2003 clusters include things like separating the log files from the database files on different physical disks, includes things like having a different physical disk, potentially, for the SMTP mail queues. This allows us to separate where the I/Os go within your SAN configuration.

The limitation, of course, in Exchange 2000, is that if we only have 23 drive letters available to us, then even with all the fancy RAID hardware that's out there nowadays, you still have the limitation of only having 23 physical disk LUNs visible to the system, each one of which has its own drive letter. That means across an eight-node cluster, we'd have several, only two or three potentially, physical disks available to each virtual server in that eight-node cluster.

When you allow mount points, which are also sometimes known as junction points — if you want to do some searches on that, you may find some additional information — but when we only mount points in Windows Server 2003 clusters, what it lets us do is it lets us actually combine, into a single drive letter, a number of separate physical disk LUNs. It allows us to actually separate the logs from the databases, but have them both be on, say, the R drive.

The second half of the question was talking about disaster recovery and dismantling of clusters. There is some information on disaster recovery and dismantling of clusters, specifically the removal of Exchange Virtual Servers in that portion, inside the "Exchange 2003 Deployment Guide," the link for which is actually inside the presentation.

Otto: We've got a couple of users who are asking for some clarification on why Active/Active clustering is not recommended, and if it's not recommended, why does it exist. I'm assuming for backwards compatibility, but can you maybe summarize for us?

Allen: Sure. Let me try and explain that. Basically, the primary reason why Active/Active clustering isn't recommended is due to the issues that we can potentially hit with virtual memory fragmentation. We covered that a little bit earlier in the slides and in the Q&A, so I won't belabor that particular portion of it, but it's important to really note there are some myths about Active/Active that people should probably be aware of. The primary myth that we get about Active/Active clusters is that it somehow increases performance, and that's just simply not true.

It's very important that the people who are listening to the WebCast understand that Active/Active clusters don't increase your performance at all. There's absolutely no benefit to having an Active/Active cluster from a performance standpoint. In fact, because of the scaling limitations that we place on Active/Active clusters, because of the virtual memory fragmentation issues (things like the 1,900 users concurrent and the monitoring of certain performance monitor keys for virtual memory fragmentation), we really can't scale an Active/Active cluster as high either.

Plus, the third one that I like to call out, and one that's especially important to me, is that an Active/Active cluster is much harder to manage, because you have, essentially, twice as much management because you have two Exchange Virtual Servers as opposed to just one Exchange Virtual Server. So there are a number of reasons why Active/Passive is better than Active/Active, and there aren't any reasons why Active/Active is better than Active/Passive. So that's why we end up recommending an Active/Passive configuration.

The reason why Active/Active is still supported, you're pretty much right, Otto. It's because it was something that we released with Exchange 2000 two-node clusters. Because it was made available to the market, there are some customers who, for whatever reason, prefer the Active/Active solution. We can't take away that functionality even though there really isn't anything that's a benefit to an Active/Active cluster. Hopefully that answers the question.

Otto: The next question here: Any recommendations for volume shadow copy and backup for Exchange 2003?

Allen: That question has the potential to get way outside of the scope of the clustering presentation, but note that there is a lot of information out there, most of which comes from the third-party vendors who provide the solution. In Exchange 2003, the listeners may be aware, there is a new feature called VSS, Volume Shadow service (also called Volume Shadow Copy service), that allows us to do online snapshots of Exchange. This is a great feature, but it's one that we at Microsoft have provided a framework for, but none of the real code to make it happen.

It's sort of in the same sense as the antivirus code, the antivirus API that we provide. We provide the framework to allow it to happen, but it's up to the third-party vendors to actually do the work. So if you're interested in VSS or Volume Shadow Copy solutions, you probably want to check with whomever your backup vendor and your SAN vendors are, to see if they have some sort of solution they can provide that ties into our supported framework.

Otto: We're being asked to clarify an earlier question regarding licensing, if we can. Perhaps we can point the user to some more information as well. But the question is: Is an Exchange Server 2003 Enterprise license required for both nodes of an Active/Passive cluster or only the active node of a cluster? For example, SQL Server 2000 only requires the active node to be licensed.

Allen: I believe that both nodes of the cluster would require licensed copies of Exchange Enterprise Edition; although, I would refer that to your licensing specialist for the specific licensing framework that you qualify for.

Otto: Outside of the additional resources that we've got listed here, do you guys have any recommendations for books that people can look at to help them with the clustering aspect of 2003, or is there anything available yet that you know of?

Allen: At this stage in the game, it's probably too early to have books that are available on it. One thing that it is worth pointing out over and over again, because it's a very great feature of Exchange 2003, we do have these documents, that are essentially like books, available at the Tech Info Library site. These are things like the planning guide and the deployment guide, and there are a number of additional guides that will be coming out in the next couple months. These documents are ranging from 20 to 30 pages, up through several hundred pages. They are essentially books on the various portions of planning and deployment. Although I'm not aware of any published printed books yet, I would definitely recommend taking a look at these deployment planning and the other various guides available at the library site.

Otto: It looks like that's all the questions there, so I'm going to go ahead and wrap up the session. I certainly wanted to thank our presenters for coming out and giving us a great presentation and rolling through the Q&A here. In addition, of course, I'd like to definitely thank our listeners for coming out and listening to us. We certainly hope that this information was useful to you and your business. We hope that everyone has the opportunity to tune in again soon. Future Support WebCasts and upcoming events are always going to be available on Support WebCasts site at http://support.microsoft.com/webcasts/. Feel free to take a look at the list and see if there is anything there that might be beneficial to you, and we'll see you next time. Thanks, everyone, and have a great day.