Do you find the Support WebCast transcripts helpful?
Let us know!

Note This document is based on the original spoken Support WebCast transcript. It has been edited for clarity.

Paul Simmons: Hello, everyone. Welcome to today's WebCast. My name is Paul Simmons, and for the last two years, I've been working on Microsoft® Windows® 2000 in Microsoft's Premier Directory Services Support team. Before that I was a consultant for a number of different consulting companies. Today I'm going to be giving a WebCast on Active Directory® disaster recovery.

First, let me talk about what we're going to be talking about today. We're going to be talking about Active Directory disaster recovery. It's a level 200 WebCast, and we're going to be covering concepts, features, and how-to information.

First, some feedback that I received from the last WebCast. I tried to include lots of screen shots, a demo or two in here so you can actually get a good feel on how to use the product.

I want to start off with a definition of disaster recovery (slide 2). I define Active Directory disaster recovery as resolving problems on Microsoft Windows domain controllers that affect client, domain, and forest operations in the least amount of time with the least amount of pain with the best possible results. These factors involved in this definition are important, especially when we're talking about the time and we're talking about companies and businesses that need to get back up and working. Doing things in the most ideal way is not always as important as doing things the fastest way that gets the company back up and running and in production as soon as possible.

I want to start off by talking about preventative maintenance (slide 3). How we can avoid using the disaster recovery scenarios entirely. Ideally, if we use good hardware and test it regularly, we can avoid a lot of the problems that you're going to experience. Another important aspect is you want to always test your deployments of Active Directory or applications in a lab before deployment. This is extremely, extremely important that we actually test things before we put them out into operation. A lot of times we can prevent situations that require disaster recovery before they occur.

Another important aspect along the same lines is to practice the disaster recovery scenarios in a lab. Lots of times a company will have a disaster recovery scenario. They'll have a procedure for how to do things, but they've never actually taken the backups, tested the backups, verified that they're useful, verified that they're working, and then actually performed the disaster recovery in the lab before the time arrives that they need to do it. When the time comes they realize their backup software wasn't working the way we intended, this doesn't back up exactly what we needed. There's a critical aspect we're missing or there's something that we failed to realize beforehand because we didn't test it in the lab.

Another important point in preventative maintenance is to remove single points of failure, specifically in Active Directory, we're talking about domain controllers (DCs) here. At no time do you ever want to have, other than at the very beginning, a single domain controller in a domain. I can't think of any reason to have only one DC in a domain, you need to have at least two for backup purposes. As far as one domain controller going down, if all else fails, if a backup fails and you have another working, valid domain controller, you can always format, reinstall, bring up the other domain controller, and replicate with the good DC and you haven't lost any of your Active Directory information.

We get into trouble with companies that for whatever reason have gotten down to having one domain controller in a domain and then something happens like the Active Directory database is corrupted or the backup that they had that they thought was valid wasn't, and now we actually have no good copy of Active Directory. So you always want to remove single points of failure and keep at least two domain controllers in a domain, especially if this is the root domain, a lot of times you try to minimize the amount of resources involved in doing a root domain and you'll only put one; you need to make sure you have at least two.

The last important preventative maintenance is backup the system state before and after every major state change. Specifically here I'm talking about schema changes or other applications that interact with Active Directory or any major deployment of new users or new groups or new policies. You'll want to back up before and after, so if things don't proceed the way that you anticipated you can always return to the previous state.

With Active Directory disaster recovery, we have three recovery options (slide 4). The first and simplest recovery option is a rebuild. It involves Winnt32, Dcpromo, and Re-replication. This is where we rebuild the machine, wipe it out, format it, reinstall the operating system, re-promote it as a domain controller, and re-replicate. There are some additional steps involved if you're going to bring the domain controller in with the same domain controller with the same name. If you were unable to Dcpromo the machine properly those involve steps like metadata cleanup and removing the domain controller object from Active Directory before you try to bring it back in.

The advantage of this is the time involved. We have a known recovery time and we have a known set of results. Whereas trying to actually resolve a specific problem or perform a recovery in a situation that we're not sure of the results, a lot of times in a few hours this can get us back up and working, and we have a set amount of time that we know it's going to take to do it. So there is a big advantage in doing that.

The second option is a restore. In this case using Windows NT® backup to restore to a known good state. When we do this restore we basically restore Active Directory to the known good state and then we re-replicate with our existing domain controllers in the domain to bring us back to the current state of Active Directory, so we haven't missed any of the changes that took place while the domain controller was non-functioning.

The last is a repair. When I'm talking about a repair I'm speaking specifically of the Active Directory database. We would really only consider using a repair of the database as a last resort, and perhaps not even consider it at all. If we use Esentutl to do a repair of a database, it actually repairs the structure of the database, and we can't actually be certain of the results of the data after the repair is done. So there is a possibility for losing data and a possibility for losing critical data in doing a database repair of this type; really in almost all circumstances, a restore of the Active Directory is preferable to a repair.

I want to talk about some of the recovery tools that we use in doing these different restore scenarios (slide 5). The first tool is Ntbackup. When we use Ntbackup, we can do system state restores and full backups of the entire system.

Ntdsutil. Ntdsutil is like the Swiss Army Knife of Active Directory disaster recovery. There are lots of different utilities that we use in there. There's a Symantec database analysis tool. It's also a wrapper around the Esentutl tool that allows us to do database validation and repairs. This is the tool that we use to actually do an authoritative restore, where we increase version numbers of objects in the Active Directory to make them authoritative. There's also the ability to do metadata cleanup where we actually in and remove entire instances of servers, domains, and sites out of Active Directory; so it's an extremely important tool.

Like I said before, Esentutl is the database repair and validation utility. Winnt32, obviously before rebuilding the operating system, Dcpromo for demoting a domain controller or promoting a member server to become a domain controller, then we also have component-level recovery utilities such as FAZAM, which is shipped in the Windows 2000 Resource Kit, Supplement 1, and Dfsutil to do component-level recovery of group policies in the case of FAZAM or DFS configuration information. With FAZAM, we can actually back up individual group policies and restore them, and Dfsutil allows us to do an export of DFS metadata, and an import DFS metadata at the component level instead of doing a full disaster recovery of the database.

The first recovery tool is Ntbackup (slide 6). Some of its features are that it backs up Active Directory in Online mode. This means we don't have to go offline or go into Disaster Recovery mode or Active Directory Repair mode to actually back up the directory, so we can do it while we're online, and we can schedule backups.

Some of the things we can back up are the system state, which is Active Directory, boot files and registries, the registry, and a few more files, as well as doing a complete system backup and backing up all the contents of the hard drive.

There are a couple resources to be aware of. One is how to back up and restore the system state. The other article that I have listed here is the files and folders that are not backed up using the Ntbackup tool, so it's important to look at those and see what is and what is not being backed up in different instances of using Ntbackup.

Here are some of the limitations of using Ntbackup (slide 7). One of the major limitations of using Ntbackup, and it's not necessarily a limitation of the tool itself, but a limitation of backups involving Active Directory, is the time restriction for a backup. By default there is a 60-day time limit on system state backups for Active Directory. The reason for this is twofold. One is the password change. The Kerberos password to be used for authentication is changed every 30 days. The history or the number of passwords that a domain controller stores, is two; so we'll store our current password and the previous. So if we restore to a state that's past 30 days we'll actually authenticate using our previous password, but the domain controllers actually store our previous password as well, so that's not a problem. But if we go beyond the 60-day period, so that we've actually gone through two password changes, we'll be unable to authenticate and we'll get "access denied" errors when trying to replicate.

The second portion of this backup limitation is something called tombstone lifetime value. By default, the tombstone lifetime value is 60 days old. In Active Directory the replication that we do of objects in Active Directory are based upon version numbers, and because it's a multimaster replication mode just like Exchange or WINS, we can't actually go in and physically remove an object from Active Directory, because if we do that then the next time it replicates, a server that has that object will replicate it over.

The way we delete an object and keep it out of Active Directory is instead of deleting it, we change a couple of different things. On the tombstone object we set the IsDeleted value to true. When it's deleted we actually set the WhenDeleted value to the time of the deletion. So once those values are set, Active Directory realizes that this object has been deleted and it replicates that change out. So the fact that that object has been deleted will replicate out to all the other domain controllers.

Another function called garbage collection that happens every 12 hours on domain controllers, comes around and checks for objects that have been deleted and that have surpassed their tombstone lifetime value. So they basically have been deleted for 60 days. At that point it, goes ahead and removes those objects from Active Directory. The problem with using a backup that is over 60 days old is it can reintroduce those objects that existed at the time of the backup, but have been deleted. At this time there will be no tombstone objects to match, so those old deleted objects will be reintroduced in Active Directory.

The last limitation is the fact that there is no support for a schema rollback in Windows 2000. The only option we have if there are problems with the schema is do a recovery of the enterprise from tape backup. This is another important aspect, a very important thing to be careful of when you're actually modifying the schema or updating the schema, is the fact that after the schema has been changed, we can't go back and just change the schema back to the state that it was before; that's not an option. What we'd have to do is a disaster recovery, full restore on the entire database on all the domain controllers. In a large environment, that's really not an option, so we need to be really careful and test schema updates in a lab before we actually roll those out.

The next tool I want to talk about is Ntdsutil (slide 8). A couple of the things we can do with Ntdsutil. Metadata cleanup. Like I said before this is where we can remove orphan domain controllers or domains. We can remove objects. This is what we would do if one of our domain controllers had crashed, we were unable to Dcpromo, and we wanted to reinstall it with the same name. If we try to do that, we basically would have it reinstalling with the same name but a different GUID, so we want to actually remove all the previous instances of that domain controller out of Active Directory, and one of the ways we do this is with metadata cleanup, and you can do that through Ntdsutil.

The integrity check and repair are database functions that are actually wrappers around the Esentutl utility that check to see if the database is damaged and actually can try to perform either a soft recovery or a repair of the database.

The last thing you can do with Ntdsutil is authoritative restores. This is where we actually mark selected objects in a domain as being authoritative, and we do that by incrementing their version numbers.

In doing a restore, I want to talk about the two types of different restores we can do. We can do an authoritative or a nonauthoritative. The first one we're going to be talking about is a nonauthoritative restore (slide 9). So what is a nonauthoritative restore? It's restoring to a known good point using Ntbackup. You reboot into Active Directory mode and sync all the changes.

When would we want to do something like this? Well, we could do it when we only had a single DC in a domain, which I already mentioned is not a good idea to have only one DC in a domain, but because there is no replication involved, we don't need to worry about our version numbers because this is the only version of Active Directory that we have. In this case a nonauthoritative restore would be fine.

Another way you would want to do this is if you are going to restore configurations to a domain controller that was not able to replicate or had not yet replicated. So if we had a domain controller that some of the Active Directory settings like connection objects and things had been deleted on that domain controller, which prevented it from replicating, so obviously those changes had not replicated out; we could then do a nonauthoritative restore on that machine, restore those connection objects, which would then allow it to replicate and replicate all the changes that it had missed.

There are only going to be a few selective reasons to do a nonauthoritative restore. A lot of times in a large environment if we really mess things up, if we delete a critical object or things like that we're going to wind up doing an authoritative restore.

Another option, which is probably going to be preferable in most cases to doing a nonauthoritative restore, if we don't have application data that we're concerned about. If our domain controller is really just a dedicated DC, it doesn't have any other functions in life other than being a domain controller; it's not an Exchange Server, it's not a SQL Server, it doesn't have any other third-party applications running on it. What we can do is just do a Dcpromo it down; or rebuild it from scratch, rerun Dcpromo, and bring the machine back into the domain. Like I said before, this brings us to a known good point and then we can sync the deltas (changes) and bring us back to where we were before, so sometimes there can be a time advantage of doing things this way.

The second type of restore, and the one that we'll be covering mostly today, is an authoritative restore (slide 10). What is an authoritative restore? An authoritative restore is restoring to a known good backup using Ntbackup. Then we go in with Ntdsutil and make objects on that domain controller as a master copy for Active Directory, and we do this by incrementing the version numbers of the objects and attributes in Active Directory by a specific number. By default we use 100,000, but we can actually specify the number that we increment those objects by.

Now when would we want to use this kind of restore? We'd want to do this restore when we have accidental deletion or modification of objects or containers in the Active Directory or corruption of attributes in the Active Directory.

There is another alternative to doing this. Instead of actually doing a recovery from a tape backup, what we could do is find another domain controller that had the objects that the changes had not yet replicated to and make those known good objects authoritative in Ntdsutil. In that case, because those version numbers were higher and it was authoritative, the changes would replicate back to the domain controller that had the problems or that the objects had been deleted on. Later on I actually have a demo of doing this, so hopefully if this is a little unclear at this time, it will become a little bit clearer later on.

The way we do an authoritative restore (slide 11) is after we have a good backup, we boot into offline restore mode. We reboot the computer, press F8 as it is booting up, and we want to log on to Active Directory disaster recovery mode, or DS restore mode for short. Then we'll log on with the offline administrator account.

The offline administrator account is the account you use to log on with in Active Directory disaster recovery mode. You are prompted to provide a password for the offline administrator account while running Dcpromo.exe. It is important to remember this password because it's the offline administrator account that you log on with, not the domain admin account, when you want to perform a disaster recovery. These accounts are separate. I wanted to make that distinction.

After we boot into Active Directory restore mode, what we would then do is a restore of our system state, if that was the method we were going to do. If the machine that we were doing this on already had the good copy of the data, we would skip the restore obviously, and we would go straight to Ntdsutil and we will mark those objects as authoritative on that domain controller. We can do one of two different things: we can restore the entire database or we can actually just go ahead and restore a specific subtree, so we can do an OU or sub OUs and their objects underneath them instead of doing the entire database. One of the best practices that we're going to want to do there is to use the smallest and most restrictive path to these objects that we want to restore. We don't want to restore the entire database if we don't have to. If the only objects that we've deleted have been four or five miscellaneous objects that are localized in one OU, we don't need to restore the entire database, we just use the DN (distinguished name) path to that OU and restore that specific OU.

An important tip that I can give you is there is a Q article (Knowledge Base article), Q256588 on restoring Active Directory over Terminal Services. Lots of times in large enterprises you don't have the support staff at every remote site to have somebody on hand locally that has the technical depth to perform a disaster recovery. These people may be centralized in one location or there may not be someone there to do that.

One of the advantages we have with having Terminal Services capabilities on all of our domain controllers is the fact that we can actually do this restore via Terminal Services. This Q article works us through the steps of doing this.

Basically what it involves is going in and setting in the Boot.ini you set a /SAFEBOOT:DSREPAIR /SOS option in the Boot.ini file. You set that as your logon option, and then reboot the server; and you can do all of this via Terminal Services. After it reboots it will then automatically reboot in the Directory Services Restore mode. Terminal Services is still working in Restore mode, so you just go ahead and after you wait 5 to 10 minutes for the machine to reboot, log back in via Terminal Services and you can finish the Active Directory restore via Terminal Services. Then when you're done you can actually set the option back to a Normal Boot mode, and then the machine will reboot normally. This can be a huge time saver and money saver as opposed to actually sending person on site to perform the restore.

The next set of recovery tools that we're going to talk about are Winnt32 and Dcpromo.exe (slide 12). Everyone should be fairly familiar with these. It's the basic installation and promotion of a domain controller; so you install Windows 2000 and promote the domain controller. Lots of times this is the most simple pain free way of taking a domain controller that has ceased to function and needs to be restored, because it's capable of being done by fairly low-level support staff. They can build the machine, especially if you have a specified image or a procedure on how to do that, and then rerun Dcpromo. It will replicate, get a current good copy of Active Directory, and you're back up and running in a very limited about of time.

Another option is that you can actually maintain a standby server that can be shipped out to a remote site, in case you have hardware issues that cause a domain controller to go down. Then you can perform these steps to get back up and running.

I want to go over a couple of different scenarios (slide 13). I want to go over a hardware failure, I want to go over deleted objects in Active Directory, then I want to go over some of the things to remember when we're restoring FSMO roles, or Flexible Single Master Operations. So when you hear me refer to a FSMO that's what I'm speaking of, I'm speaking of the five single master operations that we have in Windows 2000. Then lastly I will do a demo of authoritative restore.

The first scenario is we have the domain controller that experienced catastrophic hardware failure (slide 14). The machine blows up, a hard drive goes down, or whatever. The machine dies and we need to get it back up and running as soon as possible. Now the tools that we're going to need to perform this task are a valid backup and we're going to need as close to an identical set of hardware as we can get our hands on. So when I mentioned valid backup, I'm talking in this specific instance where we have a hardware failure, we have actually a whole system going down and it could be, if it's the hard drives that have crashed, we're going to need a valid backup of the entire system. Depending on what files are corrupted or what kinds of things have gone wrong, a system state restore will actually get Active Directory up and running, but it may not ensure that we boot. So it's important to back up the entire system drive to ensure that we can get back up in case of a complete hardware crash.

The process that we will do (slide 15) is we'll actually replace the entire server or replace the specific hardware, restore from tape backup, and then we'll just go ahead and replicate again. At this point we actually haven't damaged any Active Directory data. We haven't lost any objects. It's just that server itself is down, so all we want to do is get the server itself back up using normal recovery methods. Then we'll worry about Active Directory by actually performing a rereplication and allow you to get back up and replicate all of the changes that have taken place since it had gone down. Alternative again is formatting the machine, doing Winnt32 and repromoting the domain controller.

Another important point that I want to talk about here when we're talking about hardware failure and we're talking about restoring on hardware is restoring to dissimilar hardware (slide 16). There have been mixed results with restoring to dissimilar hardware. There is an article out, Q263532, "Disaster Recovery of Active Directory on Dissimilar Hardware." This article greatly improves the success of restoring to dissimilar hardware. It has lots of important information on how to perform this, on things you can do ahead of time, the types of files you need to have backed up to get this to work. It also has the kinds of hardware conflicts you can run into.

Some of the requirements or things that we recommend having to improve your chances of having a restore to dissimilar hardware work are having the same number of drives and drive letters. This is essential, this is a requirement. You have to have the same number of drives and you have to have the same drive letters. Those drives need to be the same size, if not larger than their predecessors.

You need to have a complete backup of system state and the system drive. You need to have the same number of NICs, the same video card, HAL, kernel, and number of processors. There has been some testing with restoring from different vendors, from Compaq computers to Dell computers or vice versa, and they have gotten that to work on occasion, but that is not a guarantee, it's not a guarantee that it's going to work. So to improve your chances you want to have as close to the identical hardware as you can get your hands on. Sometimes that can be difficult because hardware changes quite rapidly.

Another couple of things you want to do are remove teaming network cards on the target, and you want to make sure that you have knowledge of the disk drive controller and configuration information, because those are going to need to be identical as well.

The next scenario I want to talk about is restoring deleted objects in Active Directory (slide 17). If critical objects have been deleted from Active Directory you want to recovery these objects without re-creating them manually, and we're given a valid backup. In this case we're talking about just a system state backup. We don't need all the other hard drive information. We're just looking at a system state backup to restore these Active Directory objects.

Now like I said before there are two different options we can use this for (slide 18). We can do a backup or a restore and a rereplication or we can find a server that has not yet deleted these objects, that these changes have not yet been replicated to, and we can actually go ahead and mark them as authoritative.

The first resolution is we restore from a recent tape backup. It contains and leaves objects. We mark these objects as authoritative using Ntdsutil and then we reboot in the Normal mode in the lab and replicate. The alternative is finding a domain controller that hasn't got the changes; we go ahead using Ntdsutil to mark these objects as authoritative, and then reboot in the Normal mode and allow them to replicate. In this step, because we're able to find a server that had not had the deletions replicated to them yet, there was no restore required.

An interesting thing that we've seen some of our customers implement that was a very good idea and I would like to pass it along, is a way to protect yourself from having to do the restore portion of this, to actually do the restoring from tape. Some of our customers have actually taken a domain controller and set it aside, and set the NTDS connection objects so that it replicates only once every four or five days. This backup domain controller therefore doesn't get deletions or changes like that for a number of days. This gives you time to identify problems in some of the changes that you made, and then when you've done that, you can just go to this server, find the objects that are going to be changed or that are problem objects, and you can mark them as authoritative. Now you can go ahead and force replication to reverse all the changes that you did previously on the other servers. This is a way to give yourself a four or five day buffer on changes that you make, if you notice something that's gone wrong. If a junior admin or somebody goes in and deletes an OU by accident that contain 500,000 users, you give yourself a four or five day window. So instead of actually having to do the backup you can just go to that domain controller, mark that OU as authoritative, and then force replication to prevent that problem before it gets too widespread.

I want to talk a little bit about FSMO recovery, Flexible Single Master Operations (slide 20). There is a good article (Q223787) that actually discusses all of the steps of seizing and transferring FSMO roles. I've had a couple customers ask for specifics when they're designing a disaster recovery plan, they ask, how should we plan for our FSMO roles? What is the order that we need to restore the FSMO roles in? Which one is the most important? Which one should be first, which should be second?

The important thing to note is to actually understand what each of the FSMO roles does and when it's going to be needed. Typically, and in the scenario that the one customer had given me, they were wondering if they had an entire enterprise go down, all their domain controllers go down, which should be the first one to restore? Should it be the PDC, should it be the RID master? It's important to realize that, if you're going to have to restore your entire domain from backup anyway, you do not really need any of those FSMO roles immediately. It's important to know what they do. It's important to realize that the domain name master is not going to be needed until you create a new domain or move the last domain controller in the domain, so the timing of when you do that as compared to the PDC FSMO doesn't really matter.

The more important aspect of FSMOs is realizing when you need to seize one and when you need to transfer one. If we have a FSMO role holder go down, the domain controller that holds our PDC FSMO, and it goes down, what we do at that point really depends upon how we're going to bring that server back up. If I have a restore, I'm going to do a system state restore and bring it back up. I don't need to transfer. I don't need to seize any of the roles unless that role is going to be needed before the time that the recovery takes place. So if I won't be able to actually do the recovery for two or three days, and I realize I'm going to need the PDC role available, then we may want to transfer the role, if it's possible. If that domain controller is unavailable then we would need to go ahead and seize the role.

A note about seizing roles. The difference between transferring a role and seizing a role is that transferring a role is the preferred method and is graceful. It contacts the other domain controller that holds the current FSMO role, verifies with it that they both are aware that the role is transferring, the role transfers, and then that information gets replicated out; eventually all your domain controllers are made aware of the new FSMO role holder.

When you seize a role it only happens if the current holder of the role is unavailable. So we make an attempt to contact the current FSMO role holder. That fails. We then go ahead and mark, in our version of Active Directory, that now I am the new FSMO role holder and the other domain controller never, ever gets that information. So if later on that domain controller starts working again or we fix it, it still believes that it is the FSMO role holder, and those two DCs are going to be in conflict.

There are a number of different problems that can occur when you have different DCs that both feel they are the FSMO role holder for various things, specifically like RID pool, you can start issuing RIDs that are identical from two different domain controllers and you can start having domains and domain controllers that have the same GUIDs, which can cause massive headaches.

So the most important thing to realize is that a server that has had its FSMO role seized needs to be taken off line immediately and never, ever, ever brought back on line. Format it twice, just to be sure. Never bring that DC back on to avoid any kind of problems like that.

The next slide (21) is actually a screen shot, so you can see what Ntdsutil looks like. As you can see on the slide we've actually gone to a command prompt and typed ntdsutil. It's a menu-driven command-line utility, so we typed in ntdsutil, and then we have a series of things that we can type in. For doing FSMO roles, we type roles and then type a question mark (roles ?) and it gives us a list of the options that we have. As you see there, we have all of the options for seizing and transferring all of the different FSMO roles.

An important thing to note is, if you actually try to seize a role, it will first attempt to transfer the role before it tries to seize it, and make sure that it fails. Then it will go ahead and seize that role.

Now we're going to move on for our finale here (slide 22). We're going to give you a slide-by-slide demo of doing a disaster recovery, system state backup, and deletion of objects, and then we're going to restore those objects, make them authoritative, and then do an authoritative restore.

To start off, we have an Active Directory, we have a domain, we have an OU (OU812), and in that OU we have four users. The next slide (23) is going to show us what one of those users looks like, and we're actually using a command here called repadmin /showmeta, and they we actually give the distinguished name of one of the objects, one of the users in this case, Eddie. So we go ahead and we're dumping out all of the attributes of Eddie, and the important thing to note here is the second-to-last column, the version numbers. He's a brand new user, so all his version numbers are going to be a 1 or a 2 or a 3. A couple of things change immediate after the user is created, but this is what a default user would look like when you run this utility. So we can see that the version number starts off at 1.

The next thing we want to do is a system state backup (slide 24) I'm skipping some of the actual dialog boxes, trying to save time as far as doing the demo, so there are a few more boxes, but typically they're just acknowledgements where you would click Next. Following the backup wizard for Ntbackup, we're going to do a system state only. This is good for Active Directory; but like I said, if we wanted to back up the entire server, we would need to back up everything on the computer as well as the system state.

Once that has completed and the wizard will go through complete and it will tell you your backup has been completed. After we have done that, you'll see that someone has gone into our OU and deleted the four users (slide 25); they're gone now. It just so happens that these were extremely important users and the boss is now yelling at us to get these users back as fast as possible.

The next thing we're going to do is a system state restore (slide 26). Now the step that I skipped here is actually booting into Disaster Recovery mode, so before we actually get to the point where we're going to do the system state restore, we're going to change the Boot.ini, or for local just reboot. As it's booting up, press the F8 key and go into Directory Services Repair mode. Once we're in Repair mode, this is the point that we'll do the system state restore, because we need to do this while Active Directory is off line.

Another option that I want to take the chance to bring up here is under advanced options for doing the restore I've seen some customers be confused at the fourth check box, the one that says, When restoring replicated data sets, mark the restored data as the primary data for all replicas. A lot of times they think that doing this makes the information authoritative, that this is actually the step that makes this an authoritative restore, and that is not the case. This check box is the same as when you first set up a Distributed File System (DFS) share, and it asks you about the DFS information. You've got two different replicas on the DFS share, and it asks which one wants to be the primary set of data and which should be the secondary set of data. That is what this is similar to, so this has nothing to do with marking the objects in Active Directory as authoritative. This is for DFS replicated information.

Once the restore has been completed, we go through the wizard, and this says, "The restore has been completed. Do you want to reboot your computer?" You say no, if we're going to do an authoritative restore. If we reboot it at that point, it would come on line and then we would re-replicate, and we would replicate the old changes that we had before and we would be back to the same situation that we were at. We would have no users because they've been deleted.

In this case after we've done the restore, now we've actually restored those objects, and we want to make our objects authoritative over all the other domain controller's versions of the objects, which are deleted, and which have the isDeleted attribute set (slide 28). So we open the command prompt, we go to Ntdsutil, and we type authoritative restore. A shortcut for authoritative restore is a r, and the shortcut commands in Ntdsutil work, so we type a r and it brings us to the authoritative restore command prompt.

In this case, we had the option of doing a restore of the entire database, which we don't really need to do because all the users affected are in one OU. So in this case we're just going to type in the specific distinguished name of that OU, so we type restore subtree ou and then the path of the OU. Then after we hit ENTER, it asks us are we sure we want to perform this Authoritative Restore, and we say yes.

Then we can see some of the information that the Ntdsutil spits back at us (slide 29). It tells us that it found five objects and has updated the records on those five.

Another important thing to note on this screen about four or five lines down, where it says, "Increasing attribute version numbers by 100,000" An alternate command to typing restore subtree, is we could do I believe it's restore ver inc, which is the version increment, and then a number. So we can actually specify, instead of doing the default, which is a hundred thousand, we can actually restore these objects by using the ver inc command, specify the number by which those objects are incremented. This could be important if we already have objects that have been incremented to 100,000 or 200,000, and now we do a backup back to a time when they were at one's and two's and just doing a regular restore is only going to bring those up to 100,001, which is not going to be high enough to be authoritative over their current state.

There could be situations, although rare, that you would need to actually use the ver inc command and restore those attributes to increments of a higher number.

Okay, once this has completed successfully, I wanted to show the difference in the version numbers, so this is just a small snippet of the command line (slide 30). We typed the same command, where we did repadmin /showmeta, and we did it on one of the objects. If you look now, it's a little bit difficult to see because the numbers are starting to run into each other, but the second-to-last column, our version numbers are now all incremented by 100,000; so instead of being container name (cn) equaling 1, or the version equaling 1, the version number is now 100,0001. So we can see that all those version numbers have been incremented by the commands you ran in Ntdsutil.

What we do at that point is reboot the machine and it brings it back up in the Active Directory Normal mode, and then those changes replicate out. All those changes that we have made and all those versions of those objects are now authoritative and will have precedence when it replicates out to all the other domain controllers and those objects will be restored.

That concludes the presentation portion of my WebCast. I think now we'll hand it back over and move on to the question and answer section.

Jason Bennett: Thanks for the presentation there, Paul. If you'd like to have a copy of the PowerPoint slides, be sure you download the file from the Web site. This content (slides, transcripts, streaming media) will be available from the Past Support WebCasts page.

To access all this information, including details about the upcoming Support WebCasts, an easy to remember URL is http://support.microsoft.com/webcasts/. We will answer all questions that get to us during the live WebCast. The Q&A portion of the Support WebCast is intended to encourage further discussion of the Support topic. One-on-one product support issues are outside the scope of what we're able to address here today. If you need technical assistance please submit an incident on the Web or call Microsoft Product Support Services and speak to a Support Professional.

We have received quite a few questions coming into the queue today during this presentation, so we're going to do our best to get to all of them.

The first question I have: I have tried to do a system save restore of an Active Directory domain controller and found that, after restoring the system state, the Ntds.dit file was missing from the backup. I think the backup was performed at the Windows 2000 Service Pack 1 level. Is there a known issue with pre-Service Pack 2 backups of Ntds.dit or this is a product support issue?

Paul: If I understand correctly, they did a backup of the Ntds.dit in SP1 and now they're in SP2. I don't know of any known bugs where that would occur that would prevent the Ntds.dit file from being backed up.

Jason: It doesn't sound like they've got to Service Pack 2, it sounds like they just performed the backup at Service Pack 1 and found that that file was missing. But you're not aware of any issues to that?

Paul: No, I'm not aware of that.

Jason: You should probably get on to http://support.microsoft.com and submit an incident. Look in the Knowledge Base.

Next question: How can you create more than one domain controller or is it possible when you're using Small Business Server 2000?

Paul: I am not an expert on Small Business Server 2000. I typically deal with larger premier customers and they almost always exclusively use just the straight Windows 2000 product. I would not be an expert on the backup. There may not be any differences in the backup procedures for the Small Business Server, but if there were, I definitely wouldn't be an expert on that.

As far as I know there are no restrictions in straight up Windows 2000 as far as the number of domain controllers you can have in a domain. If there is in Small Business Server I'm not aware of it, just because I'm not familiar with that product. That may be something that I may need to refer to somebody else with and answer at a later time.

Jason: Okay. Do you think there is a way we can track down that information off line?

Paul: Yes, there probably is. I can just send a quick e-mail to the Small Business Server team.

Jason: Okay, that'll be great. Okay, next question: As far as single points of failure go, can the roles that the first domain controller handles be moved to a backup domain controller if need be?

Paul: The FSMO roles, in this case, yes. The roles that it handles, if they need to be transferred, can absolutely be transferred. We showed the screen shot of the interface that you would use in doing that, that's Ntdsutil where you transfer or seize roles.

Jason: Okay. Is this statement true? "Per Active Directory domain mid-to-large size, no one domain controller contains the entire Active Directory." If so, do we really need to back up every domain controller in the domain, or just a few selected, like one per site?

Paul: I think I see where they're coming from. The first statement I don't think that what they said was true, that no one domain controller has the actual entire copy of Active Directory. You have an entire copy of the Active Directory on every domain controller. They may be different because of the loose consistency, because of a multi-master replication. They may not always be at the same version numbers. They may not always have the same objects in all of them, because they could be in the state of replicating. But with that one exception, yes, everybody's got their own copy of Active Directory, and it's a complete copy of Active Directory within that domain, not necessarily the enterprise or the forest.

So the point about whether you need to back up each server individually or whether you need to back up a few selected servers really depends on your disaster recovery strategy. I've worked with a couple of different companies that the dollar investment, as far as how many backup systems they have for each one of their servers was an important point, so they actually only backed up one DC at each site. They actually had a local copy at every site, but in this case they had a known recovery plan. If the Active Directory went down on a server that wasn't backed up, they would format and reinstall to get back up and running, because they didn't have a backup of every domain controller. So that was an accepted practice for them. There was nothing on their machines other than them being a domain controller. So the machine going down, formatting it, and bringing it back up again was a completely acceptable option. They really only had the system state restores so they could recover from Active Directory problems like deleted objects and those sorts of things. That's a completely acceptable situation, it's just you want to make sure that you've looked at all the different options and all the different consequences of having a scenario like that.

Jason: Okay, fair enough. I understand that the password for the Active Directory Restore mode is stored in the old SAM of the domain controller. That is usually a pretty old password that we don't update at all. Here are the questions, and it looks like there are three, so I'll take them one at a time. The first question is: I assume that each DC can have a different restore mode password. Is that true?

Paul: Yes.

Jason: Is it a good procedure to boot in Active Directory mode every once in a while to keep that password in sync with the latest domain password?

Paul: You can definitely do that, but what I have typically seen is that a company or an enterprise has a SETPASSword that they use. Because this password really can only be used at the console, they set this password, they make it a strong password, and they make it the same for all of their domain controllers at the time of their creation. So at the time that they are building them, they will make them all the same and they store this, lock it away somewhere for when they need it. They really don't typically change that. That's how I normally have seen it to be done.

As far as actually changing that password after the fact, I actually have not gone through the process of changing that, so I'm not sure what the process for changing it is. But that's not typically how I've seen it done. Typically they create the password once for everybody at the time the DCs are created and they keep those, and they make them the same across all their servers.

Jason: Okay, here's the third part of that question: Is there a way to do recovery if the administrators don't know that password?

Paul: That would be no, not according to our normal recovery options. At that point, if you did not have access to that machine because you did not have the password, you would then want to do a format and a reinstall, bring that machine back into the domain, and then re-replicate with some of the current domain controllers. So no, at that point if you can't get in you won't be able to do the restore.

Jason: Okay. Is there a way to create a Ghost image from a domain controller to re-create others easily?

Paul: That's a little bit outside the scope, as far as using Ghost as a third-party product to create new domain controllers. I'm not a setup hardware specialist, I'm more of an Active Directory person. I know there are some issues with Ghost and specifically the GUIDs and some of the SIDs of things, but I'm not sure how that affects the domain controller. So I would really probably have to refer them to PSS on that and talk to a setup person.

Jason: Okay, thanks for that. I want to thank everybody for sending in good questions. Again, let us know if you've already had a question answered and we'll take it off the queue.

Is it possible to back up an Active Directory independent of a system state backup? I have tried to restore Active Directory to dismantle hardware several times and the results are usually bad. Microsoft's KB article on this procedure is not reliable, and the Microsoft Product Support team has not been able to help me. Do you know if this is possible, Paul? Can you make any comments on this?

Paul: I don't think that I fully understand their question about backing up Active Directory separate from a system state restore. Using Ntbackup, which is the only backup solution that I am familiar with, because I'm not going to go into third-party products and the methods for using the, the way that we back up the Active Directory is using the system state restore. I'm not quite certain what he's trying to do, and it sounds like he's already been in contact with PSS, and that would be my recommendation on that, because I don't think I fully understand what he's trying to do.

Jason: Unfortunately he seems to be making more of a comment about the current state of Active Directory. He ends with a question: Why can't Microsoft separate Active Directory from the system state?

Paul: Oh, I see what he's saying, if he just had a system state restore that did not include all of the Active Directory stuff.

Jason: That's not possible at this time, then?

Paul: Yes, not usually. You may be able to go in and actually specify different components, but I'm not sure that would get him exactly what he wants.

Jason: Okay. Do you know anything about our partnerships or relationships with backup software vendors like VERITAS or Legato? Are these products certified to handle Active Directory restores?

Paul: I'm not super up on that, but I do know that we do have a certification process where third-party software can be certified to work with Windows 2000, and I do know that especially with the backup software that there are a number of, like two or three backup solutions that have been certified to work with Active Directory and to do system state restores. But I would not want to venture a guess at which ones those were, because I would be afraid that I would say the wrong one, one that wasn't or forget one that was.

But I'm not sure the location to go and find those. That may be something I might be able to find off line for you or point you to a resource that would have a list of those.

Follow-up answer: To check check hardware and software compatibility, go to http://www.microsoft.com/windows2000/server/howtobuy/upgrading/compat/default.asp.

Jason: Okay. Will schema rollbacks be supported in .NET Server? Do you have any information on that?

Paul: I don't know what the current plan is, and if I did know, I probably couldn't say because it might change.

Jason: Okay, that's fine.

Paul: Even if I knew something hard and said that absolutely 100 percent we're going to be able to do that, it could change between now and when it rolls out. To be honest with you I really don't know what their current plan is.

Jason: Exactly. Thank you. Do the same backup limitations apply to a domain controller that is offline for an extended period of time? What is the lifetime of a domain controller that is off line?

Paul: Yes, this could be the exact same thing, because really there's not difference between having a domain controller that's off line for 60 days as opposed to doing a backup to a point that was 60 days old. At that point you've got the exact same scenario. You've got a 60-day old copy of that server, so all the same things would still apply. If objects have been deleted in that time while that server was off line then you're still going to run into the tombstone lifetime.

The password problem, I'm familiar with how it worked in NT 4.0. I still believe that the password problem might not apply, because I think the actual client is the trigger or is the mechanism that actually triggers the password changes, so if it's off line I don't think it will actually be able to change its password, so the password problem may not be a problem for being off line for that amount of time.

Jason: Okay. Where can I get more information about tombstone defaults and how to change them and how they will impact my restores?

Paul: I know when I was looking at it, there were articles in TechNet on how to do that. That was one of the sources I was referencing when creating this documentation. There is a good article that explains all the important aspects of tombstone lifetime, and I think there's actually one that goes in there and tells you how to change garbage collection intervals, how to change what the default tombstone lifetime is and those sorts of things. So I would refer to our Knowledge Base on that, there should be several good articles out there, but I don't know them by numbers.

Jason: Okay. What's the impact of changing the tombstone lifetime value past 60 days? Does this also change the password history value?

Paul: I believe those are two entirely different things. As far as what the impact of changing the tombstone lifetime value would be, the point is that it would just take longer for objects to be completely removed from Active Directory. I don't think that, in and of itself, would cause a problem, but there may be something I'm missing there, but I don't think that by itself would be a problem.

Jason: Okay. If Dcpromo demoted to a member server is run on a server that holds FSMO roles, are those automatically transferred to another domain controller?

Paul: Yes, when you run Dcpromo to demote a server, on a server that actually holds FSMO roles, you will transfer those. The only problem about doing it that way is that it's random, you don't have an option on which one you're transferring it to. It just looks for a domain controller, finds one and say's okay, here you are, you're the RID master, you're the PDC master, and it just passes them out that way.

Typically though, when you're in a disaster recovery situation, Dcpromo is not an option, because a lot of times when you've got a domain controller that's not functioning, it's not replicating, it's not communicating to the other domain controllers for whatever reason, it's also unable to do a Dcpromo down. But in the event that you are, then it's always preferable to do a graceful promotion down.

Jason: Okay. Can you restore with data from one domain controller to another, and if so what needs to be the same?

Paul: Are they saying doing a complete restore with data, or just doing the data itself? If you're just transferring files and data, that's not a problem. If you want to do a complete backup of one domain controller to another set of hardware on a different domain controller, then I would refer you to the section in the WebCast on restoring to dissimilar hardware. There's an entire article on that, and it has all the requirements and specifications. But if you're just talking about the data, then there shouldn't be a problem at all.

I guess that's not really specific enough to know what parts they are talking about.

Jason: Right. If that user wants to follow up, we can definitely take the follow up during the WebCast.

Okay. If you only have one domain controller in a child domain in your forest and it has a failure, making you have to rebuild it, what would be the best way to restore it? I know that some of the objects are replicated to the parent domain. How does that come in to play?

Paul: That's a good point. Really the only part that's actually replicated in a parent domain is the stuff that's going to be stored in the Global Catalog servers. In that situation, if you weren't the only domain in the forest, without doing more research on it right now, just to be safe I would probably do an authoritative restore. I don't know if that's necessary; it might not be. I'd actually have to look and see if there was a reason for doing that, but to be on the safe side I would do an authoritative restore because it won't hurt, but I don't know if there's anything that the other domains in the forest are going to need that would cause a problem if you just did a normal restore. I don't think so, but I'm not 100 percent certain.

Jason: Okay. Now we've got quite a few questions about the offline administrator password.

Paul: I know there are some articles in the Knowledge Base on the offline administrator password.

Jason: Yes. Are there options if you don't know the password itself? Can you reset it or recover it or change it?

Paul: No. If you're at the point where you don't know it, there is no supported way for Microsoft to change that password, at least not that I know of. If I am mistaken, if there is something that I have missed, then that's the same as having a password on a machine that's was a member of a workgroup and it was the only password on it, there's just not a way to get in there to change the password.

Jason: Okay. What can they do? What should they do if they don't know the password?

Paul: They could always call Produce Support Services to make sure that there is not something that we have missed, but at that point they're really in a bad situation in the fact that there's no way that we can hack into it and violate their security, even if we had a way to do that. So if there's a password on it and they don't know the name of the password there's nothing that we can do to help them hack that password and violate their security to penetrate that. They're really in a bad situation, if that's where they're at.

Jason: Okay. Along those lines, I think around that subject, I'm not sure exactly what slide, this user is asking the question and it might be referring to the offline password again. It says: Can this be reset with SETPASS later? It looks like its slide 11, which is the authoritative restore. Does that comment make any sense to you; can this be reset with SETPASS later? I'm not particularly sure what they're referring to.

Paul: I think they're talking about a utility called SETPASS. I'm not sure.

Jason: If it's not clear, if that user wants to follow up and give us a little bit more information that would be fine.

Paul: This is an article that should be publicly available in the Knowledge Base, article Q223301, "Protection of the Administrator Account in the Offline SAM," I obviously can't look through it now and find out all the information on it. But that is an article that specifically talks about the protection of the administrator account in the offline SAM, and I think this is the password that they're talking about.

Jason: Okay, thanks for that. Is it possible to restore a single object using the authoritative restore?

Paul: I saw that question in the list. I have never tried doing a single object, so I'm not sure if the interface actually allows us to do it. It's entirely possible that you can, but I've never tried it. So I'd have to say that I don't know and I can get back to them, I can try that this afternoon and get back to that person. I've never actually tried to restore just one user; I've always done them at the OU level.

Jason: Okay, great. Any recommendation for, I think this refers back to the question we had earlier about Legato, any recommendation for third-party software for backup and restore?

Paul: I can't make recommendations on third-party products.

Jason: Okay. This question: I'm not sure what book they're referring to, but they're referring to Chapter 10, "Disaster Recovery for Branch Office Environment," page 8 says only the domain and configuration partitions can be marked as authoritative. The schema cannot be authoritatively restored because it might endanger data integrity. Is this true, and how can we do a schema authoritative restore?

Paul: Yes, that's true, and that's where I was talking about. Schema rollbacks or schema restores are not supported in this version of the product. So that's just talking about exactly what I was mentioning earlier, that we can't mark the schema as authoritative. In other words, if we change the schema and we want to get back to it, there is not a way to do that. We'd actually have to do an actual physical restore of each one of the domain controllers that have had those changes replicated to them. There's not a way to go in there and mark a version of the schema as authoritative and replicate it out.

Jason: Okay. Can you talk a little bit about the Boot.ini parameters?

Paul: Talk about the Boot.ini parameters.

Jason: Yes, what were they again?

Paul: Actually I think it was in the WebCast. The article that has those parameters in them is in the WebCast, or at least the parameters that refer to backing up, to restoring in the DS Restore mode, which is /SAFEBOOT:DSREPAIR mode. The article is Q256588, which actually has the syntax for what you want to do to the Boot.ini file, that's on slide 11. If they search the Knowledge Base on Boot.ini, then they're actually going to find a couple more articles out there that specify all the different Boot.ini switches, which are used for a number of different things.

Jason: Okay. What's the impact of not having the files mentioned in Q article Q233427? What's the impact of not having those files backed up? Did you mention that in the WebCast or is that just one that they are referring to?

Paul: I think that is one that was mentioned about the files and folders that are not backed up. It's so you have knowledge of what files are and are not being backed up. As far as the impact of them, that would depend on each individual file, and what you were hoping to accomplish with the restore. There's not a specific answer. The purpose of having this article is so that you actually have knowledge beforehand of which files are and which are not being backed up. But as far as the impact of those, I'd have to look to that article, read and see specifically which files weren't being backed up. Then I'd have to figure out what those files do and what the possible consequences of not having a file backed up could be. So I don't have an answer offhand.

Jason: The next question: What was the Boot.ini switch for having the server boot directly into repair mode?

Paul: That was the DS Restore. It's /SAFEBOOT:DSREPAIR /SOS, and that was in that same Q article on slide number 11.

Jason: Okay. Several folks were asking about article Q263532 that I believe you mentioned. They're not currently able to find it on the live site, is that something that's going to be made available soon? It's restoring Active Directory to dissimilar hardware.

Jason: Perfect. If I maintain a standby domain controller, what's the recommended method for replicating to the offline standby domain controller? How will I keep the standby domain controller current?

Paul: I think there are two different things. One of the things I mentioned in the WebCast was having a standby server; in this case we're just talking about hardware, not necessarily having an operating system installed on it, so that we can ship out at a moment's notice and then run setup and then running Dcpromo on it, and bringing it up as a domain controller. In that case there's no maintenance we need to do on it.

The other thing that I mentioned in the WebCast was having a server that was sort of like a backup domain controller that you only replicated to once every four or five days. In that case what you want to do is turn KCC off, the Knowledge Consistency Checker, so that it doesn't automatically generate connection objects. You then manually (this is FRS stuff for people that are not familiar with it) create connection objects and set the replication schedule so that it only replicates one day a week (or whatever). That way you have a set window; you could say that it only replicates on Wednesdays, that way you know that any changes that are made won't replicate for several days.

That's what you would do is manipulate those manually created NTDS connection objects for replication. That's how you would accomplish that. I don't think there's an article on how to do that. That may be something we might want to look into making.

Jason: Okay. Let's see, next question: Assuming we know we deleted critical objects, how can you prevent further replication before finding the objects on another server and setting it to the master?

Paul: That's a good question. In your site, it's going to be almost impossible because replication to the other DCs in the site should typically happen within five minutes, so you're probably not going to prevent that.

As far as replication across the site, the easiest thing is to turn the machines off or take the ones that have already have the data replicated to them offline. Actually the best thing to do is to find the server the farthest away from these that you know has not had the data replicated to it, and take it off line, so that even if it replicates out to everybody else, you've got this one machine that doesn't.

But as far as trying to prevent it from replicating around, you're going to look at trying to delete connection objects or set the connection objects to disabled for replication. As far as replication between sites, you've usually got a window by default of three hours so you may be able to go ahead and do that. You're probably going to have it automatically replicate out to everybody in the site, but as far as stuff across sites, you should be able to locate a server at a different site that hasn't had it replicate. But there's not going to be any easy way to do it. It's going to be a matter of looking at what sites they're in and who it's been replicated to and who it hasn't, and then finding a server and taking it off line, unplugging it from the network or whatever, until you can do the restore on it.

Jason: Okay. The next question refers to restoring a completely dead DC server using Winnt32 and Dcpromo. The comment they made was: I was thinking of using Ghost instead of Ntbackup, because you've got to install the whole operating system first before you can even use Ntbackup. Does that make any sense with regard to what you were saying in the presentation?

Paul: Could you read the question one more time so I can try to understand what they were trying to get at?

Jason: Yes, they're saying in this case the complete backup is an Ntbackup but something like Ghost, right, and then I said well what case are you referring to, and they said oh, to restore a completely dead server using Winnt32 and Dcpromo. I was thinking Ghost instead of Ntbackup because you've got to install the whole operating system before you can even use Ntbackup. Is Ghost a third-party product?

Paul: Yes, so I don't want to comment on this stuff about using Ghost, but if you're talking about restoring a machine completely, you can do a complete backup of a machine on Ntbackup. They're right in the fact that to actually use Ntbackup you have to actually install the operating. In other words if I did a full backup of my domain controller and then something happened, it got a virus and I had to format the hard drive and do a restore from a time before I had the virus, to actually use Ntbackup to restore that I'm going to have to reinstall the operating system so that I have access to Ntbackup, to use Ntbackup to restore everything. So that's what they're talking about and said could you use Ghost to just ghost it and then put it back on there, and I don't want to make any assertions about whether or not you can use that third-party product to do that.

Jason: Okay. With respect to the restore from tape backup, I assume it's still a case of base install to an alternate directory and then a full restore using Ntbackup given the same hardware. Is this correct?

Paul: That sounds right; it sounds like what I was just talking about.

Jason: I think we might have already gone through this. Is there an easy way to perform a selective authoritative restore, i.e. restore all user objects from A to L?

Paul: Oh, no. You're going to have to do it by containers, so you're going to have to do it by your OUs. There's not going to be a way to restore objects that have the first name starting with so and so; there's not a way of doing that. There may be some really confusing, convoluted way to script something like that, but it would be very complex and basically searching through your Active Directory and looping through the NTDS utility to actually restore those, but I'm sure that would take a long time.

So there's not automatic, easy way to do that based on your user name or any objects like that. It's all done on the OU hierarchy, so you're going to need to do it by the container. Either do the entire database or do people that are in this OU or in this sub OU underneath that, and that's going to be the method that we're going to use for doing that.

Jason: Okay. This user gives a hypothetical. On Monday I make a complete backup, on Tuesday I create user A. On Wednesday I do an authoritative restore of the entire database of Monday's backup. User A remains on the Active Directory. Why is this? Should I completely re-create an Active Directory to a certain point in time?

Paul: No, I meant to mention this and I went right past this point. This is a very important point, and I talked about all the aspects of it but I didn't actually bring this point of it home. On a situation where you have only one domain controller, you've only got one copy of Active Directory. I had an OU that had four users and I made a backup of it, so I have a backup of those four users and their version numbers are all at 1. I deleted those four users. I then create another user with a different name and put him in that OU, and then I do a restore back to the point where I just had those four users. Because I've only got that one copy of Active Directory, that user will disappear and the four users will be back there.

In an enterprise environment where you're doing replication and you have multiple domain controllers, what happens is I had those four users, they've got the version numbers of 1, they replicate to all the other domain controllers, so every domain controller has those four users on them. At that point I then delete those four users, the deletion of those four users replicates out to all the domain controllers, I create the new user, that new user replicates out to all the domain controllers. Now at that point if I go back and do the restore, all I'm doing is on this one domain controller overwriting my copy of that OU, so getting rid of that one user locally and replacing him with the four users, in this case with a version number of like 100,001 on those domain controllers. At that point when replication happens I say, hey look, I've got these four new users with version numbers of 100,001, that's greater than your copy of the users that is deleted, so take my version of these users. Those four users then replicate back out, and then the other domain controllers say, hey, I've got this new user who's got a version number of 1 that you don't have a copy of; let me replicate him to you. So you're not actually deleting by doing a restore, you're only putting the new objects back in the Active Directory and incrementing their version numbers so they overwrite other existing versions of their objects that have a lesser version number. You're not getting rid of other objects that might be in that OU. So that's an important thing to note, is all you're doing is taking the version numbers on those objects and incrementing them by 100,000. You're not actually saying delete every other object that was in this OU on any other server and only have these objects. All you're doing is copying new data and incrementing their numbers 100,000. So if you're got multiple domain controllers and you've created other users in the OU, they're still going to persist. So at that point, if you didn't want those users any more, you would need to manually delete them.

So that's an important point, and I've actually had that as a support call and it took us a while to conceptually figure out what was going on, but that's the way the procedure works. You're not actually getting rid of objects doing a restore, you're just incrementing version numbers.

Jason: Okay. This might be the same question. I'm trying to read it, but it seems slightly different. A common thread throughout the discussion of doing restores from other domain controllers is to find an unaffected domain controller if possible, and then make the necessary objects authoritative there. My question is how to verify whether those objects still exist on a particular domain controller, and obviously take some time so once you find one, is it a good idea to unplug that domain controller from the network to prevent replication? Where does one specify the Active Directory replication schedule, and is there a recommended schedule?

Paul: By default the recommended schedule is once every three hours. In domain controllers in a different site, within the site, the replication interval is every five minutes. So those are the recommended settings that are obviously modifiable and depending upon your bandwidth. The answer to his first question about should he unplug it from the network to prevent it from replicating, yes. Where do you go to change the settings? You actually go into Active Directory Sites and Services, find the domain controller under NTDS Settings, and then you'll see all the connection objects. You right-click the connection object, go to Properties, and that's where you change the replication window, how often it replicates. By default it is 180 minutes intersite, so you can actually change that amount of time or look at the replication window for hours that replication is allowed to occur and you can modify it there. So that's really up to them that way, but it's in Active Directory Sites and Services, under the domain controller, under NTDS Settings, is where you go to change those.

On another note, there is another article that is a public article because everybody's having questions about this offline Recovery Console Administrator password. I think one of the users that is in the WebCast informed me on this article. There is an article (Q239803) that talks about a way to change the Recovery Console Administrator password. I haven't read through this, so I'm not aware of its contents, but there may be a way to do that using this article.

Jason: Great. The next question concerns a slide that you covered, I believe it is slide 19, it's the last of the slides, the Deleted Objects in Active Directory section. The user didn't quite understand what you meant in the first bullet point. Can you describe that?

Paul: Where it says set replication schedule once every four days on backup domain controller?

Jason: Yes.

Paul: Okay, I put "backup domain controller" in quotation marks because we're not talking about an NT 4.0-style BDC, I'm just referring to a domain controller that we're going to use to bail ourselves out of these situations. This is the same scenario I was just describing, where you go in to Active Directory Sites and Services, go into the domain controller, look at the NTDS settings and look at all the connection objects underneath that, and then you want to change the replication schedule. By default, it's going to be once every 180 minutes. You can change that to be once very 24 hours or once ever 72 hours or however long you wanted the replication schedule to be. That way it gave you a window of time before those changes actually replicated to that server, time to notice the changes and then get to that server and mark those affected objects as authoritative, so that they aren't changed. That's what I was describing in that bullet point.

Jason: Okay, thank you for that. About seizure roles, if I de-promote the server and then re-promote it, could it be back online?

Paul: Technically if you can demote the server, there's probably not going to be a reason that you're going to want to seize it. So if I understand correctly, if you're saying if I have to seize a role from it and then I demote it, actually at that point I wouldn't want to demote it either, because I'm not sure of the behavior of what happens. If I can't connect to another server to transfer the role gracefully (which is preferred), I'm probably not going to be able to Dcpromo down. But on the off chance that I can Dcpromo down, I still think that I'm the FSMO role holder even though somebody else has seized it, because they never contacted me when they did that. I'm going to go ahead and try to transfer that role to somebody when I Dcpromo down gracefully. I'm not sure of the logic of the code, whether it actually goes out and checks and finds the other instance of that FSMO role out there, or if it just randomly picks a domain controller and gives it the FSMO role, so you could again wind up with two domain controllers with the same FSMO.

The real answer to the question is that there's probably never going to be a scenario in which you have to seize the FSMO role; that you're going to be able to gracefully Dcpromo it down. After you Dcpromo it down, though, if you did get it to Dcpromo down, it doesn't have the FSMO role anymore so it's not the problem, so yes, you can bring it back up. But there are going to be very, very few situations where that specific set of circumstances is going to occur like that. More often than not, if you have to do a seizure on it, you're not going to be able to do a Dcpromo down. And if you could, one of the things from preventing you from doing a Dcpromo down is not being able to transfer the role. So, if you try to do a Dcpromo down and you can't transfer your roles off, it won't let you Dcpromo down. So those two things probably are mutually exclusive, but there may be a rare, rare instance where you get both of them to happen.

Jason: If you have a backup domain controller with replication set for every four days, how do you prevent a new user whose ID is not replicated to that server yet from hitting it at log on? If it's on day one and I add a new user or object, it will not reach the backup one until four days later, so the new user will be unknown to it.

Paul: Put the domain controller in its own site. To keep it from replicating with other domain controllers in its site and to make sure that all the replication is intersite replication (so it replicates it from one site to another site), you're probably going to isolate that domain controller that you set up this way and put it in its own site, a fictitious site, call it backup site. That way there are no other domain controllers in this site that it's replicating with, and you create the only manual connection object to another domain controller. That way if it's the only server in that site, and all users attempt to log on to a domain controller in their site, and there are obviously no users in that site, there's no other machines in that site so no one should be trying to log on to this domain controller.

There are probably some other things you can do, like some weighting of DNS records and stuff, but just putting it in its own site by itself should prevent anybody from actually trying to log on to that guy or authenticating that guy when they log on.

Jason: Okay. The next question: Seizing is safer if the role holder is on line since it tries to XFR first? It's probably a safer practice to XFR though. Can you make any comments on that?

Paul: I see what they're saying. If you just try to seize automatically it's going to try to transfer anyway, which is true. So if it can transfer the role it will, however, the disadvantage of doing it that way is that you don't have a choice about who the role goes to, which domain controller it goes to. So ideally, you're probably going to want to control who gets that new role, so you're going to want to do a transfer and transfer it to a specific target. Through the interface, you would select operations target of the person you're transferring the role to and then say transfer the PDC emulator role. That way it transfers that specific DC and there's no guesswork involved. Whereas if I do a seizure and it says, "okay, trying to transfer the role first," and then it says, "okay, I found Server X in lower Mongolia" and it transfers the role there, you didn't really have an option of where it went to. So the preferred practice is trying to do a transfer first, but as a safety step, we do try the transfer as a first step of actually doing a seizure as well.

Jason: Okay. Are there any caveats and things to watch for if you set one of your domain controllers to replicate after four days? We have only two domain controllers.

Paul: If you did that, the only thing is, like the other person mentioned, you probably want to have them in their own site and not have anybody use them for logon traffic. Basically what you want to do is isolate that DC so that its only function is to serve as a safety mechanism. This is because at any one point in time, your version of Active Directory could be four days old. You're probably not going to want anybody authenticating to it for log ons, for security policies, that kind of stuff, anything that could be important that could change on a daily basis. You're probably not going to want anybody to authenticate to them for that purpose, so putting them in their own site is definitely going to be a good idea. So that would probably be about the only caveat that I can think of right now.

Jason: Okay. The next question: Regarding the customer suggestion to have one backup Active Directory controller set to replicate every x number of days, doesn't that create a potential back door on a network? If someone's access was removed to a resource and they were knowledgeable enough, they could get themselves authenticated by that backup domain controller and bypass the new security.

Paul: That's a good point. Actually there may be a way to get around all of this. I'm trying to think of dependency services here, and there may be something that I'm not missing. If you just don't want people to log on to that controller, you may be able to stop the Netlogon service on that server. That actually might be a cleaner way of doing that. I don't think that will interfere with replication, I'm pretty sure it doesn't. Stopping the Netlogon service should prevent people from actually being able to authenticate to it. I'd have to actually research that, but I think that may be another option. You just stop the Netlogon service on that server so that all it does is replicate and it doesn't authenticate users. So that would prevent that problem. But that was just a brainstorm; I haven't actually checked that out to see if there would be other problems doing that.

Jason: Right. A system state apparently cannot be backed up on remote servers. Would it be a good idea to back up system state on all of our domain controllers to their own hard disk and then come around with a network backup and put those on tape?

Paul: I see what they're saying, using Ntbackup to back them up locally and then using some sort of third-party network backup software that backs up over the network to restore that. I don't see any reason why that wouldn't work. Not knowing about the third-party software and how it operates, I don't immediately see any reason why that wouldn't work. That sounds like that should be fine. You're just going to have to find a way to get that data back to that domain controller when you want to do the restore, because it needs to be local when you do the restore.

Jason: Okay. The next user asks: Did I miss something? Why did we say no to authoritative restore, then go in to the Ntdsutil and manually do an authoritative restore?

Paul: I don't think we said no to doing authoritative restore.

Jason: Okay. I'm not certain where they got that.

Paul: Oh, I think they're looking at the slide. The focus might actually be, by default, on No, when it asks you if you want to do an authoritative restore. If you're looking at slide number 28, I'm actually already in Ntdsutil doing the authoritative restore; I've actually already typed ntdsutil: a r, and done restore subtree OU=<path>. Then once I hit ENTER, it actually prompts a dialog box up that says Are you sure you want to perform this Authoritative Restore? By default the focus is on No, so you have to manually click Yes. We clicked Yes there, it's just when I took the screen shot the focus was still on No, so that may be the source of confusion there.

Jason: Okay. Regarding the backup domain controller scenario, would you have to put the four or five day old copy of the database server into its own site to control replication? I think you might have mentioned this before.

Paul: Yes, I did mention that. That would be a good idea to do. I don't think you have to, but that would be the way I would do it, is I'd put it into its own site.

Jason: Okay. Can you restore a single user authoritatively? I think you addressed that as well.

Paul: I saw somebody respond back that said yes, you were able to do that. That could very well be true. I just have never done it myself. I've never tried to do it in the Ntdsutil through the interface that way, but one of the other people replied back already and said that yes, you could do that. Maybe they've had a chance to try it. I have not yet had a chance try it, so if they want I can follow up with them later on to verify that. I think the answer is now yes, but I would need to verify that to be 100 percent sure.

Jason: Right, and it would be great to get a Q article or something. The next question and I don't know if I'm reading this correctly, it's a string of letters so I'm going to do my best with it. Can output from Ldifde or Csvde be used in a restore situation?

Paul: I'm familiar with Ldifde, but I'm not sure what they're trying to say. Could you use that in a restore situation? I guess I don't fully understand what they're asking. I understand what they're talking about, output from Ldifde, but I'm not sure what they want to do with that output once they have it. So that may be something that we will need to have them clarify exactly what they want to do, because I'm not 100 percent of their question.

Jason: Okay: Do you get the authoritative restore option when restoring from third-party software, such as VERITAS backup exec, or do you have to use Ntbackup?

Paul: I cannot speak intelligently on the third-party software. The last time I used a third-party backup solution was before the release of Windows 2000.

Jason: So they're referring to a specific option on the third-party software.

Paul: Exactly, I would have to refer them to that third-party vendor to check up on that.

Jason: That's fair.

Paul: The last time I was a system administrator I was on NT 4.0 before the release of 2000, so since then I've been here at Microsoft and have only been using Ntbackup.

Jason: Okay. Is the offline defrag the Ntdsutil compact a good way to verify that the database is consistent, and should offline defrags be done every so often?

Paul: That's a little bit outside the scope, but if they want I can follow up with them. I'd have to check up on some of it; I know we've got some KB articles out there on the preferred methods for doing offline defrags and things. I'd have to refer to those, I don't know off hand.

Jason: Okay. How does performing a full system restore to a newly rebuilt operating system affects its ability to boot and run applications that may have been part of the backup that are not now on the newly rebuilt server? Should the restore be done selectively instead?

Paul: Yes, here we're running into a tricky part where a system state restore is really only going to do a restore of Active Directory. It doesn't really care about third-party applications or even Microsoft applications that are also on the system. It's only a backup of the Active Directory files, the registry, and things like that. So, doing a system state restore won't fix an Exchange problem or a SQL problem, if that was a problem. If doing a system state restore restores keys in the registry, because we do back up the registry, that have since changed, that actually affected Exchange or SQL or one of these other programs or some third-party program, then it could have an effect on it, but you would really have to be more specific, and there's probably two million different scenarios in there to speak intelligently on.

In that case, if you've got other applications that are running on that server, I would recommend doing a full backup of the entire drive. That way you're not just restoring the system state, you're restoring the system state and all the application data back to a known state to where everything is consistent as opposed to backing up parts of the registry that may not be consistent with another part of an application that could be looking at those registry keys. So in that case, I would recommend doing a full backup of the server.

Jason: Okay. What's the reason of the junction Sysvol\Sysvol\Domain\ from the directory Sysvol\Sysvol\%Domainname% and where to copy back GPOs, in the case of an authoritative restore from GPOs or other Active Directory objects with a corresponding Sysvol objects? Does that question make any sense to you?

Paul: A little; I kind of understand what they're saying but not entirely. I think they're probably talking about backing up files in Sysvol. So probably to follow with them, part of the question is why there is a junction point where you have Sysvol\Sysvol. That's going to be outside the scope. I had that same question when I first started looking at Active Directory, why we had that Sysvol\Sysvol set up that way, and that would be a question I couldn't answer. All I know is it was a development decision for having it that way. I may not be able to answer that question for them, but if they want to resubmit it and clarify exactly what their question is, I may be able to follow up with them off line. {Editor's note: No further information was provided by the user.}

Jason: Okay. We are coming down to our last couple of minutes; I'm going to try to get in a couple more questions before we have to end. We'll try to take the rest off line. Next question: I want to take my live server into a lab environment. I have an empty root with two domain controllers and a child with four domain controllers. The FSMO roles in both domains are split between two servers. I am short machines in the lab and need to restore all roles in each domain to one server. Do I accomplish this by executing a full restore and then seizing the roles?

Paul: It depends on what they're doing with the other domain controller. It sounds like yes is the answer. He basically has a production environment where he has an empty root that has two domain controllers, and a child domain that has four or five domain controllers, and the FSMO roles are split. He wants to back up that structure and put it into a lab, so where he has one DC that's a parent and one DC that's the child to emulate his parent domain and child domain to have Active Directory for the two domains in the lab environment, but he's wondering which server should he bring back because the FSMO roles are split.

So in that case yes, do a full backup of one DC in each domain and then go ahead and the roles that you don't have go ahead and seize, because you've only got one DC in each domain, so it'll need to go ahead and seize the roles that it doesn't have. So yes, that's how he wants to do it.

Jason: Okay. We are really out of time. I'm going to have to cut off taking any more questions during the live WebCast. I do want to thank everybody for coming in today and asking really good questions. Thanks to Paul for coming in and fielding these questions for more than an hour.

If you've got any feedback on today's WebCast or any other WebCast you've seen, we're definitely interested in getting that feedback from you. You can send it to us in e-mail to supweb@microsoft.com. If you do that, just put the date of the WebCast or the title of the WebCast in the subject line, and that way we can more easily determine who it's going to and forward it on to the appropriate person.

Again, I want to thank everybody for joining us. I hope the information was useful. We hope to see you again in the near future. Thanks and goodbye.

Question: If you only have 1 domain controller in a child domain in your forest, and it has a failure making you have to rebuild it, What would be the best way to restore it? I know that some of the objects are replicated to the parent domain. How does that come into play?

Answer: A full restore of the entire system would be the "best" way to proceed.

Question: Is it possible to restore a single object using the auth restore?

Answer: Yes

Question: Is it recommended to migrate to the last service pack version when doing a restore or should I keep the same version I had before the problem?

Answer: Keep the same version that you had before.

Question: I tried to find Q263532 in the Knowledge Base on http://www.microsoft.com/. Will it be available eventually?

Answer: I'm working on that.

Question: Can you restore a single user authoritatively? Are their any Q articles that detail with examples all of the objects and attributes that can be individually restored?

Answer: Yes. See http://support.microsoft.com/support/misc/kblookup.asp?id=Q241594.

Question: Is the offline defrag (ntdsutil compact) a good way to verify that the database is consistent?  Should offline defrags be done every so often?

Answer: Yes, you should perform an offline defrag and a database integrity check periodically.

Question: How do we restore Active Directory on a server with different hardware than the original hardware that the system state was backed up on. Must I have the same video card, NIC, and disk controller? What is meant by HAL, kernel?

Answer: This is answered in the WebCast. See Knowledge Base article Q263532.

Question: You mentioned an enterprise restore after losing or corrupting the schema using AD disaster recovery processes. How many DC's are going to require a restore?

Answer: All of the DCs in the enterprise that had the schema replicated to them.

Question: Is there a limit on the version number? Does an object reach a point when the version number rolls over?

Answer: In theory, yes. I have never seen this occur in reality.

Question: For an authoritative restore of specific Active Directory objects and corresponding objects from SYSVOL: What's the reason to restore the system state data twice, once to its original location and once to an alternate location? Are there any other reasons to restore "to an alternate location" excepting to have a good copy of objects from Sysvol folder which will be copied back to its original location after the Sysvol folder has been published?

Answer: The only reason I can think of is for FRS. Manually copying the Sysvol files into Sysvol triggers replication.

Question: How can I recover from a hardware failure, using the same GUIDs, etc.?

Answer: Replace the server with identical hardware and perform a full system restore.

Question: How are Exchange 2000 and other integrated applications handled during a restore? I've had trouble synchronizing Exchange accounts with AD accounts after a restore.

Answer: That is outside of the scope of this WebCast.

Question: Just so I understand this. Doing an AD restore only requires that one DC be rebooted into recovery mode? The other DCs can remain up and running?

Answer: Yes.

Question: Can you share the same FMSO role between two different domain controllers?

Answer: No.

Question: One of my domain controllers was down and two other domain controllers were still up but I was not able to make changes, such as change a user's password. Why should this happen when all domain controllers are the same?

Answer: The PDC emulator FSMO role.

Question: Is it sufficient to back up System State on only one DC per forest, or one DC per domain?

Answer: At least one per domain, however, doing this increases the amount of work involved in restoring the entire enterprise.

Question: If a domain controller is also a global catalog, will the GC data get backed up via system state backup as well?

Answer: Yes.

Question: What happens if you seize a FSMO role and the server that it was seized from comes back online?

Answer: It depends on the role. Typically very bad things happen.

Question: What value is system state backup on a member server, other than local registry settings?

Answer: Not very much.  Registry settings and boot files and COM components.

Question: Could you talk about the tombstone interval and how that relates to a valid backup?

Answer: This is covered in the WebCast.

Question: If a DC is a AD integrated DNS, and there are only two DNS servers, and say if they both go down, can the recovery be done from other live DCs?

Answer: You would need to install AD integrated DNS on an existing server.