Provide Feedback on this Broadcast

Note This document is based on the original spoken WebCast transcript. It has been edited for clarity.

Kevin Clarke: Historically, tape backup and restore methods have been time-intensive and, as a result, often not validated. Additionally, while it was possible to create snaps in previous versions of Microsoft® Exchange, they were not high-fidelity backups, potentially causing corruption. Today's presentation will introduce Volume Shadow Copy Services (VSS) for backing up and restoring Microsoft Exchange 2003 databases.

(Slide 2) In discussing this, I hope to address the benefits of using VSS with Exchange 2003, versus previous backup methods. While this topic is considered level 200 to 300, it does assume that you have some understanding of the mechanics of Exchange, specifically disaster recovery, the concept of transactions and how they are committed to the database, and the normal backup and restore procedures for Exchange.

(Slide 3) First, we will review snap backups and how they are used in Exchange backup routines. We'll discuss the differences between snapshot and snap clone, Volume Shadow Copy Services and Microsoft Exchange Server 2003, and what Exchange data is actually backed up using VSS. We'll have an overview of how the VSS components work in conjunction with one another, and we'll briefly discuss what events are logged when backups are performed, including what diagnostic logging levels should be included for analysis and troubleshooting. Finally, we'll touch on some finer points of VSS with Exchange 2003, and then we'll entertain some questions and provide answers in the remaining time.

(Slide 4) A shadow copy or a snapshot is a full-fidelity image of the data being backed up, made at a specific point in time. The requestor stores the shadow copy image, while the main system is free to resume servicing and using the original data. So as the diagram shows, when we have t₀, obviously we have no data, but then at t₁ we have data, and we make a snapshot of that data. Afterward, data continues to be written to the disk.

Without VSS, when copying open files such as a database, we'd have an incompletely written file. What I mean by this is if you were to drop it back in, because the file was not completely written, it would not work, because it may be corrupt. Additionally, with the advent of VSS, we can grab a snapshot of a database that is not in flux. The operation is very fast and allows the store to go back into production with little impact on performance. Moreover, the snapshot can be backed up to tape as a point-in-time backup.

As we'll see, starting in the next slide (slide 5), there are two methods for creating shadow or snapshot copies: clone and copy-on-write. We'll discuss their differences over the next few slides.

The snapshot clone is also referred to as split mirrors or full copy. A clone is a complete copy or plex of the original data volume. Created through either software or hardware mirroring, clones remain synchronized until the administrator breaks the mirror connection at some point in time. From this point forward, the source data and the shadow copy volume are independent. The original volume continues to take application changes, while the shadow copy volume, an exact copy of the original data at the time of the break, is effectively frozen in time.

Some of the pros include that it's generally quite fast to split the clone off by breaking the mirror. Additionally, because the mirror volume can then be mirrored to still another volume while removed from the premises, the clone can be viewed as having better disaster protection than copy-on-write.

Some of the cons are that it's somewhat cost expensive, because you had redundant disks for high ability. For example, if you had 0+1 RAID, you have a complete set of redundant disks spares, plus you will have a hot spares to re-mirror after the split is complete, allowing your redundancy to be put back for the high availability you had before the split. Additionally, one of the cons is that you may have to resync the redundant disk before the mirror is valid again.

(Slide 6) The snapshot type copy-on-write is a method that creates shadow copies that are differential rather than full copies of the original data. Like the clone method of creating shadow copies, the copy-on-write method can produce shadow copies using either software or hardware solutions. This method makes a shadow copy of the original data before it's overwritten with new changes, as shown in the slide.

When the change of the original volume occurs, but before the write is made to the to-be-modified block, it is read and then written to a differences or disk area, preserving a copy of the data block before it is overwritten with the change. Basically, the provider is keeping track of the deltas. This is commonly referred to as copy-on-write. With the blocks in the differences area and unchanged blocks in the original volume, a shadow copy can be logically constructed that represents the shadow copy at the point in time in which it was created.

One advantage of the copy-on-write method is that it creates shadow copies very rapidly, because it's only writing the changes to disk. The disadvantage to this method is in the restoration. In order to fully restore the data, the original data must still be available. Without it, the shadow copy is incomplete and non-restorable. So, overhead increases with use and a not completely redundant data set. Also, there is no inherent fault tolerance. However, some array hosts do host their disk areas on top of array flavors, so you may have some redundancy there.

It's also important to note that some vendors use a combination of these flavors, and you really need to investigate to understand what they're doing, so you can properly implement your end-to-end solution for backups.

(Slide 7) Here's the copy-on-write depiction. This slide is a simple conceptual demonstration of the disk area, where the deltas of the particular sectors are tracked. In addition to the original copy of the block table, you'll see that sectors two and five are highlighted, and those would actually be the deltas, if you were to change those. So we would keep track of those in the difference area.

Now to come back to the original idea of a clone or split mirror, here's a slide (slide 8) that depicts that concept again with these snaps. We're doing it on a volume level, not more granular, as we'll discuss later. Essentially what we have here is the exact same thing. The block table is exactly copied over to the other side. And after it's normalized, the mirror is broken, and then they're basically independent.

This slide really sets up the next slide (slide 9). The point I want you to come away with here is that after normalization, the mirroring, you end up with an exact point-in-time clone or plex. The nice thing about this is that it can be presented to a host as a separate or replacement LUN, which is a logical unit, and with no reliance on its source. This is perfect for testing on another system, as we'll discuss in the next slide.

(Slide 10) VSS for transport gives us tremendous functionality with much less pain than we would have experienced in the past. Although multiple servers can access the same storage array, they do not share access to the same storage unit, a LUN for instance. Otherwise, we would have had corruption of multiple servers that are trying to write to the same data simultaneously.

For servers to share the same data, the traditional sequence is to first back up the data on one server, and then restore it to a second server. This process can be quite laborious. With VSS and a storage array, with a VSS-aware hardware provider, it is possible to create a shadow copy of the source data on one server and then import that data on another server. This process essentially is a virtual transport, and it's accomplished in a matter of minutes, regardless of the size of the data.

Basically, the transport process is accomplished through the following steps:

1. We create a read-only shadow copy of the source data on Server 1, which is attached to a SAN.
2. We then mask off or hide that shadow copy from the Server 1.
3. We unmask the shadow copy to the second server.
4. Optionally, if we want to, we can clear the read-only flags on the shadow copy.

Then the data is ready to be used.

VSS transport is an advanced solution that works only if the hardware provider is on the storage array. So shadow copy transport can be used for a number of purposes, including, for instance, transport for tape backups. Backing up to tape is one of the most I/O-intensive operations an application can perform. Not to mention, on a production LAN, shifting all of this data will be time-consuming, and this high-traffic process can result in considerable congestion on the production network, thereby degrading application performance. So you can see this will be very handy for backup, testing, and data mining, to name a few uses.

The next slide (slide 11) shows a diagram of what I'm about to talk about. Conventionally moving gigabytes or terabytes of data could take hours or days, but now you can accomplish this within minutes, and without taxing your production server or network. Keep in mind, the requirements to deploy the shadow copy transport are as follows: Microsoft Windows Server™ 2003 Enterprise Edition or Datacenter Edition, hardware shadow copies only, a third-party VSS hardware provider, and a third-party VSS requestor. We'll talk about the provider and requestor in just a moment.

Before getting into the specific VSS components, let's have a quick refresher on Exchange disaster recovery. To do so, we'll need an idea of how Exchange stores data. The Extensible Storage Engine (ESE) uses write-ahead transaction logs and checkpoint files to help ensure the least amount of data is lost during system malfunctions. Transaction logs record all the changes that have been committed in the memory database, while checkpoint files record which log transactions have been written to the on-disk database file. So the transaction log plays an important role in backing up and restoring the Exchange database.

(Slide 12) To design and implement Exchange backup and restore applications correctly, you must understand the role transaction logs fill within Exchange. Exchange database transaction logs record transactions before they're committed to the database. The transaction log records the beginning and the end of each transaction, so that sufficient information is available to later undo or complete unfinished operations on the database.

Basically, the transactions are written to memory and the transaction logs. Then during CPU idle time, these transactions are committed to the databases, and the checkpoint file is incremented, keeping track of which transaction logs were written to the database. This is the concept of write-ahead logging. With an understanding of that, we can talk about recovery.

There's point in time recovery, which is used when the log files are damaged or lost. The transaction logs and the database must be retrieved from backup, and then rolled forward as necessary.

A roll-forward recovery is used when the transaction logs are intact on the disks, but one or more database must be retrieved from backup. The database is recovered by rolling forward the transactions from the logs.

Additionally, full backup recovery is used when only a full backup is available, and it's not necessary to roll the database forward through any pending transactions. VSS gives us the capability to restore back to the original location or to an alternative forest or server.

What Exchange choices do we have with VSS when backing up or restoring? With the introduction of Exchange 2000 came the idea of storage groups, where we can have minimum of four stores in each and a maximum of four storage groups per server. Each storage group had its own transaction log shared among the stores within it. Bearing that in mind, there are two supported types of backups with VSS: full and copy.

Full backups back up the database and transaction log files and purge the transaction logs. A copy backs up the database and all transaction logs, but does not purge the transaction logs. Copy backups are not intended for use in recovering failed systems; instead, data mining or testing would be more appropriate.

(Slide 13) This brings us to our choices of what we can back up. The minimum selection is the storage group, because we're required to truncate the transaction logs. Because more than one store may depend on those transaction logs, we need to purge the ones that have already been committed. While we can snap multiple storage groups at the same time, due to snap validation costs, meaning we want to make sure that it's a high-fidelity snap, the best practice is to snap individual storage groups. Our choices vary upon restoring. We can select an entire storage group for restore, or a single database, or any number of databases from a single storage group.

(Slide 14) Here is what gives us this functionality. The Volume Shadow Copy Service coordinates all components, which can be broken down into three main categories: requestors, writers, and providers. And actually the server itself is a fourth. A requestor is an application using the VSS application interface to invoke a creation of a shadow copy. So this would typically be a backup application. Also, I want to point out that NTBackup does not use VSS to back up Exchange.

A writer represents applications in Windows services. For example, SQL Server™, Exchange, Active Directory®, and more. I have subsequent slides that will list some of these. So basically the writer is application-aware and participates in the shadow copy creation process.

Finally, providers are what ultimately create the shadow copy, either through the system provider or hardware snapshots. We can flesh these ideas out a little bit more when I show you some of the lists of vendors in subsequent slides.

(Slide 15) Here is a peek under the hood of VSS with the Windows Server 2003 storage stack. It's broken down with the various components in either the user mode or the kernel mode. For brevity, I haven't specifically addressed the Virtual Disk Service (VDS). However, it's an important part of Windows Server 2003 file services. Specifically, the Virtual Disk Service provides the infrastructure used to manage storage resources, disks, and volumes, so that both directly attached and network storage resources can be carved up and used.

This becomes more important because various vendors previously required their own interface, which resulted in more specialized training. Now this management can be done through Windows. Also, note the difference in colors, where blue is Microsoft and OEMs are depicted in the purple.

(Slide 16) Here's a more simplified version of the component architecture. Basically, it's the Volume Shadow Copy Service that orchestrates the requestor, writer, and provider.

(Slide 17) This slide depicts a partial list of vendors. Neither this slide nor any other is intended to recommend any vendor over the other. In fact, you'll need to check with each vendor for support status regarding Exchange Server 2003 and product features. Because more vendors are continually joining in the efforts of VSS, this is a dynamic list, and a more updated list can be found at either http://www.microsoft.com/partners or the link noted on this slide (http://www.microsoft.com/exchange/partners/default.asp).

(Slide 18) Remember, the requestor is a component that requests the Volume Shadow Copy Service shadow copy sets of selected file systems. As you can tell, we have quite a few vendor partners that are already involved.

(Slide 19) Windows Server 2003 ships with quite a few writers right out of the box, such as Active Directory, FRS, WINS, WMI, and so on. Additionally, we have increased functionality in some server-based applications, notably SQL and, pertinent to this discussion, Exchange, which we'll talk about in just a moment.

(Slide 20) Providers manage running volumes and create the shadow copies of them on demand. There are three types of providers: system providers, hardware-based providers, and software-based providers. A system provider is supplied as a default part of the Windows operating system installation. A system provider uses a copy-on-write scheme to copy all sections of the disk that have been changed.

A hardware-based provider is one that manages shadow copies at that hardware level, by working in conjunction with a hardware storage adapter or controller. The work of creating the shadow copy is performed by the host adapter, storage appliance, or RAID controller outside of the operating system. These providers are implemented as a user-mode .dll component, communicating with the hardware that will expose the shadow copy data. Therefore, hardware-based shadow copy providers may need to call or create other kernel-mode components. Hardware-based providers expose shadow copies of the entire disk or the LUN, but not dynamic disks or logical volumes created by storage managers, such as the Logical Volume Manager.

Finally, a software-based provider intercepts I/O requests at the software level between the file system and the volume manager. These providers are implemented as a user-mode .dll component and at least one kernel-mode device driver, typically but not necessarily a storage filter driver. The work of creating these shadow copies is done in the software.

(Slide 21) ExWriter.dll is part of the Exchange 2003 install. To ensure high-fidelity snaps, the writer is able to pause the writes to the .edb and the log files and stop the .stm from growing while the hardware performs the snap. Note that all writes are suspended by Exchange during the snap process.

Clients are hour-glassed for submits. You will see this on a long snap, when a client is blocking on a write-to-database transaction for the submit. You may not always see this, but because of this, the better user experience would be with cached mode. Either way, this will be practically imperceptible to the end user.

(Slide 22) So what happens during the backup? First, we prepare for backup, and the administration functions against the storage group are prohibited. We can't do anything as an admin at that point.

Then we prepare for the snapshot. Volume dependencies are checked, and we freeze, meaning that we flush the current log file generation to disk and suspend any writes to the log files and the .edb file. The .stm file will not be allowed to grow, as we mentioned, and read-only access is still allowed. So anything that is already there, we can still access that from the client.

We then thaw it out, and new log file generation is created. We make calls into the Jet that allow writes to the log files and the .edb file, and the .stm file will be allowed to grow again.

OnBackupComplete is mandatory for a good backup. The requestor must validate the shadow copy image by running ESEUTIL after backup, to ensure consistency. Then, any truncation of the log files happens. And then we resume administrative functions against the storage group.

One of the common questions that is posed is: Does Exchange need to be installed on the backup server to run database consistency checks? The answer to that is no. Exchange does not need to be installed on the backup server. However, the ESM tool installed on the backup server is required to run the consistency check.

Another big one that I wanted to point out is: Can integrity checks be skipped for fast recovery snaps? The answer to that is that for full backups where the log files are truncated, database integrity must be verified prior to calling BackupComplete. For copy backups that are taken between full backups where log files are not truncated, the integrity check may — and the requestor is taking a chance here, and we're definitely not endorsing this — potentially skip this. By skipping the integrity check, the provider postpones identifying any possible database corruption until the next full backup. So this is obviously not what we want to see happen.

(Slide 23) During the restore, the administrator needs to take the affected storage groups offline. If these are not offline, the restore process will fail. Incidentally, vendors may actually do this through CDOEXM.

After all the checks are passed, data is restored back to the production location. This can be done with either the storage device volume surfacing or streaming of content back into place. In the case of restoring the entire storage group, the writer leaves the checkpoint file from the backup. If restoring only the database, the writer deletes the current production checkpoint file.

Starting the storage group invokes database hard recovery. Basically, Exchange views this in the same way as a typical power failure. So this is a good place to note that it's not the writer or any VSS component that is responsible for planning the log. This functionality goes back to the ESE.

I wanted you to see this slide (slide 24) for a couple of reasons. First, I wanted you to note the times between the sequence — and if we had a little bit of animation we might be able to see it a little bit better — it's basically how quickly this procedure commences. Second, know that there are many components and checks that all must coordinate and cooperate with one another. So if something comes back that it cannot complete, the backup or restore may be aborted.

To hone in on this possible failure, detailed event logs can be obtained by increasing the event log level to the maximum, or increasing the diagnostic logging in the ESM, on MSExchangeIS, System, Exchange VSS Writer. The Exchange writer metadata document (slide 25) contains Exchange-specific values and parameters that a requestor needs, so it can correctly specify the appropriate components for backup. This is backed up along with the data, so that upon restoring, it knows what options were taken during the backup.

For more details on how to decipher these, refer to the white paper, "Design Companion for Exchange 2003 Backup Applications Using the Volume Shadow Copy Service."

(Slide 26) I want to touch on a few finer points. As I stated earlier, Exchange 2003, out of the box, only supports full or copy backups. Remember, during a copy backup the transaction logs are not purged. During the backup snap creation, log buffers are flushed, and a new E00.log file is created. But the checkpoint is not advanced.

All databases have to be mounted for the backup to truncate the log files. It is required that the requestor perform a page-level integrity check on the snapped database to ensure a high-fidelity backup was created. This can be done by running ESEUTIL with the /K switch.

VSS sets the snapped logical unit to read-only, and can only be made read/write through calls through the VDS, the Virtual Disk Services I talked about before. This goes back to when we talked about the shadow copy transport. So making this snapped LUN writable comes into play when moving data across a SAN, for instance, and then using the data by another server. Again, VDS is necessary for this.

(Slide 27) VSS must be used to perform the restore if the backup was created using VSS. The ExWriter performs some sanity checks. It checks whether the databases are dismounted. It checks the circular logging state. Restoring individual databases is prohibited if circular logging is enabled, when either the backup was created or restored. This is basically because some transaction logs may have been overwritten, so the only type of restore possible is point-in-time.

All databases in the storage group must be dismounted to perform a restore of either the entire storage group or a single database restore. You cannot use VSS to restore directly into a recovery storage group. Unsupported workarounds are available. I wanted to point out that a Requestor must use the BreakSnapshot API and Virtual Disk Service functionality if you want to perform a logical unit replacement with a VSS restore.

For more information on VDS, refer to the white paper "Windows Server 2003 Active Directory Fast Recovery and Volume Shadow Copy Services and Virtual Disk Service." This is located on www.microsoft.com/windows/storage/productinformation/whitepapers.

(Slide 28) A snapshot is not intended as a complete backup solution. When searching for solutions, you need to ensure that the requestor does page check summing for feedback on corruption issues. Not all snapshot solutions are created equal. Pure copy-on-write-only solutions assume that you will stream to tape or disk immediately. Snaps should still be copied to tape or remote storage for archive.

Snapshot is not a complete backup replacement, but an alternative to streaming. We have no page scrubbing and no immediate feedback on corruption issues. Not all snapshot solutions are created equally, as I pointed out before.

Basically, the bottom line is that we still need to archive the backup. So we haven't gotten rid of the tapes quite yet.

(Slide 29) ExWriter is not told if multiple storage groups exist on the same LUN, the logical unit. A common question is whether the integrity check can be skipped for fast recoveries, and the answer to that is that we can indeed do that, but we really frown upon that. I touched on that a little bit earlier.

We must restore to the same logical drive letters. Currently, there is no native support for VSS restores to a recovery storage group. We cannot mix and match shadow copy and legacy streaming backups. In fact, if done at the same time, one or both will fail.

Vendors should never try to manipulate the checkpoint file or truncate the log file. This is actually done by the Exchange writer. You cannot start another snapshot until the BackupComplete is called. This means any validation steps will affect time between snaps.

Going back again, I've touched on it a few times, but it's important to note that the integrity check must be done. This is important to have a high-fidelity backup.

(Slide 30) That having been said, I wanted to point out that there are some additional resources that you can review. We have some Knowledge Base articles, and there is actually a hotfix update package (833167) that's available. And we have some white papers as well.

Additionally, there's a TechNet WebCast (833805) that was already done on emerging enterprise storage technology, so that's basically the shadow copies on a file-level system.

(Slide 31) Here are some more additional resources, as well as a list of partners (http://www.microsoft.com/exchange/partners/default.asp), which is continually growing. This last link is a pretty neat link to the "Volume shadow copy overview," if you wanted to go in there and dig around at a more granular level.

That pretty much sums up the presentation for the time being. Marilyn, if we have any questions, I'll hand it over to you.

Marilyn Loftus: Thank you, Kevin. At this point, we'd like to hear from you, our listeners, about this topic. If you find you need more complex technical assistance that might be outside the scope of this discussion, go to http://support.microsoft.com/, or call Microsoft Product Support Services directly to speak to a support professional.

If you would like more information on a future Support WebCast or to review any of our sessions on-demand, visit our main Support WebCast site at http://support.microsoft.com/webcasts/. For this particular session, you'll find a downloadable version of the slides and on-demand streaming media within 24 hours, and a full written transcript within three weeks.

The first question is: I am running Small Business Server 2003 with SQL and Exchange. What are some recommendations for running NTBackup to do a full backup (we are a small business, and our data can fit on a single AIT tape) and still capture MySQL database and Exchange mail stores while these services are kept running overnight? Currently, I am running an internal SQL database backup routine to a local server file location, so that the backup file is captured by the later running of NTBackup. The Exchange stores pose a different problem, for which I am interested in finding a solution.

Kevin: Because you're running SBS, there are some limitations on using VSS. As I mentioned before, the NTBackup is not actually a requestor with VSS. But because SBS 2003 serves as both a Windows Server 2003 domain controller and an Exchange 2003 server, the SBS setup program disables the Exchange writer to prevent a problem with the system state backup conflicting with the writer.

There is a Knowledge Base article that I'll point you to, 820852 (also see 828481 and 830575). Basically, in your circumstances, a full online backup may be a better solution.

Marilyn: The next question is: Does any of this change when the backup server is UNIX TSM on Sun?

Kevin: I'm not entirely sure what our audience member meant by that, but to take advantage of VSS, we'll need Windows Server 2003. That's actually part of the file system services. So I wouldn't be able to address anything with UNIX. Please contact Sun regarding TSM.

Marilyn: On to the next one. I'm not sure if you can answer this, but I'll ask it: How do you compare this with Veritas Storage Foundation for Widows? What do you recommend for a DR site?

Kevin: I'm going to have to sidestep this issue, just because there are so many vendors and partners. It wouldn't really be fair for me to make any comparison or make any recommendations for any partners. You can contact Veritas for more information.

Marilyn: Okay. The next question: Are you aware of any vendors that offer brick-level backups to single mailbox via VSS?

Kevin: Again, I don't want to point out what some vendors have been working on and what they have already put out there. I think that would kind of defeat the purpose of VSS.

I would point out though, that with Exchange 2003 we have the recovery storage group, which we didn't really talk about, because it's outside of the scope of this presentation. But basically we have an opportunity to, on the same server, restore a production database to a recovery storage group while the production databases are still running. And from there we can use ExMerge. So that is a valid alternative to using brick-level backups.

823176, "HOW TO: Recover or Restore a Single Mailbox in Exchange Server 2003."

Marilyn: The next question: Can you discuss VSS backup of Exchange and Windows OS? Are there issues or problems with using VSS to back up both?

Kevin: Keep in mind that there is a significant difference between the Volume Shadow Copy and the ability to make copies of Exchange, for instance. So you don't want to enable the Volume Shadow Copy for Exchange and the file system on the same volume. I'm not sure that I addressed that, but I'm not sure that I really understand the full scope of this question.

Marilyn: The next question: There is a 10-second time-out for VSS to perform its snap clone. What is the purpose of this limit, and why can it not be configured?

Kevin: That's a good question. There certainly is a 10-second time-out for that. Basically, VSS is actually what's orchestrating everything to work together. So if we are unable to make it work within this allotted time, we'll error out. I can't really speak to exactly why it cannot be configured, but I presume that it's just because it's so tightly interwoven, all of the components, that it needs to work together well.

Marilyn: The next question is: Is Windows Server 2003 Enterprise Edition or Datacenter Edition required to write a VSS copy to a network location, or is this enabled in Standard Edition?

Kevin: I believe you should to be able to do that in Standard Edition. But keep in mind, if you were to try to use the shadow copy transport, for instance attached to a SAN, then you will need Enterprise Edition or Datacenter Edition.

Marilyn: The next question: I am running two Exchange servers in a cluster on a SAN. Would I need a separate piece of hardware for VSS?

Kevin: If you're talking about the hardware provider, no, you would not.

Marilyn: That looks like the last of our questions submitted. So I will give you the link to the Contact Us page, which is http://support.microsoft.com/servicedesks/webcasts/feedback.asp, and then select the WebCast Comments link.

We have one more question: Can VSS back up cluster-aware applications?

Kevin: Yes it can. In fact, I believe on one of my slides I had that one of the out-of-the-box writers is for cluster. For instance, Exchange is a cluster-aware application. So the answer to that is yes.

Marilyn: Thank you, Kevin.

We hope this presentation has been helpful for you and your business, and we look forward to your participation in upcoming WebCasts.

Thank you, and have a great day.