Troubleshooting Windows Server 2012 Virtualized Domain Controller Cloning

Windows Server 2012 introduces a new feature called Virtualized Domain Controller (VDC)—whichWindows IT Pro covered in “ Cloning Virtual Domain Controllers in Windows Server 2012.” Essentially, you no longer have to deploy a Sysprep-prepared server image and then manually promote a domain controller (DC). Instead, the cloned DC automatically performs a subset of Sysprep operations and pfigureromotes the DC with the existing local Active Directory Domain Services (AD DS) data as installation media, consuming administrator-provided settings such as computer name and IP address. This allows faster deployment of new DCs in production or test labs, simplified disaster recovery, and the ability to scale out in hosting and branch-office scenarios.

 As with all things in your computing environment, you might need to troubleshoot this process. This month, I want to share the techniques that Microsoft established for VDC during its development over the past two years.

Understand the Cloning Steps

Most cloning issues turn out to be related to human error. My most important point in this article is a common realization among users: “When I fail to clone a DC, I usually find that I missed a preparatory step.” There are a number of requirements to cloning and steps you must follow exactly in order to succeed. There’s no cloning wizard, and that puts responsibility on you to get the procedure right. When you flub a stage, cloning fails.

With that in mind, I’ll distill the process down to a handful of operations. In essence, you’ll prepare the environment, prepare the source DC, and create the cloned DC:

1. Validate that the hypervisor supports VM-Generation ID and therefore cloning.
2. Verify that a Server 2012 server holds the PDC Emulator role and that the Primary Domain Controller (PDC) is online and reachable through remote procedure call (RPC).
3. Authorize the source DC for cloning.
4. Remove incompatible services and programs, or add them to the CustomDCCloneAllowList.xml file.
5. Create DCCloneConfig.xml.
6. Take the source DC offline.
7. Copy or export the source DC, and add the XML files if not already copied.
8. Create a new virtual machine (VM) from the copy.
9. Start the new VM to commence cloning.

To see the end-to-end steps, review “ Virtualized Domain Controller Deployment and Configuration” on TechNet. It’s worth the read, trust me.

Know Your Symptoms

If you make a misstep in these steps, cloning fails and the VM will boot up in Directory Services Repair Mode (DSRM). This mode protects your environment from accidental duplication of DCs. Once you fix the problem and remove the DSRM boot flag, you can restart the DC and it will try to clone again. Naturally, knowing the DSRM password is crucial. You can set the password during promotion of the source DC, or you can use Ntdsutil.exe later. Fancy folks maintain the password automatically—check out our blog post “DS Restore Mode Password Maintenance.”

If you miss step 5 above and you’re using automatic IP addressing, you’ll end up with a duplicate DC. This one is a bummer to fix (see “DC cloning fails with no DSRM, duplicate source and clone computer”), so if you get nothing else out of this article, remember step 5! This issue is as old as AD itself, but in the VDC era it becomes more likely.

 Finally, you need to understand where cloning ends and promotion begins. VDC ultimately results in a DC promotion and replication, and if those go wrong you return to the classic DC troubleshooting of the past decade.

Know Your Methodology

Figure 1 shows a diagram I like to keep around as something to jog my memory. It’s based on the most common issues I saw during internal dogfooding, public beta testing, and worldwide training of Microsoft support and field engineers. So don’t feel bad if you run into one or more of these—you’re in good company!

Figure 1: A Cloning/Promotion Methodology Diagram

First, ensure that cloning succeeded. If it didn’t, determine whether the server is in DSRM. If it isn’t in DSRM, the DC is now a duplicate and you should be following the steps in the Microsoft article “ DC cloning fails with no DSRM, duplicate source and clone computer.” If it is in DSRM, the safeguards against duplication worked and you should examine the Directory Services event log.

Which leads me to another of my reiterative points: Always check the Directory Services event log! This log contains all the cloning events (2160–2228 and 29218–29267) and should be your first stop for any troubleshooting. The log will reveal whether you made a common mistake:

• Is the hypervisor supported?
• Does an incompatible application need to be in the CustomDCCloneAllowList.xml allow list? Does the CustomDCCloneAllowList.xml contain valid entries?
• Is the PDC emulator online and available through the RPC protocol?
• Is the DC a member of the Cloneable Domain Controllers group? Is the permission Allow a DC to create a clone of itself set on the domain root for that group?
• Is the IP address or computer name either duplicated or invalid in the dccloneconfig.xml?
• Is the AD site invalid in the dccloneconfig.xml file?
• Is the IP address not set in the dccloneconfig.xml file, and is no DHCP server available?
• Does the dccloneconfig.xml file contain syntax errors that prevent correct parsing?
• Did DC promotion fail after cloning began successfully?

There are less common issues, too. Was the maximum number of auto-generated DC names (9999) exceeded? Is the MAC address duplicated? Is the PDC Emulator replicating so that it knows the security group change happened and its partners know it holds the PDC role?

I highly recommend using the New-AdDcCloneConfigFile Windows PowerShell cmdlet because it helps you avoid XML syntax errors—but don’t treat it as a crutch. It has no idea whether you typed in a bad IP address or an invalid computer name. Microsoft did its best to make the event log entries actionable. For example, in the first example below, it’s clear that I tried to clone a source Server 2012 computer on an unsupported hypervisor (in this case, it was a Windows Server 2008 R2 Hyper-V host). In the second example below, I accidentally added the name of an existing computer into my dccloneconfig.xml file. Because I can’t create a new DC when another one already exists with the same name, cloning fails.

Event ID	2169
Source	Microsoft-Windows-ActiveDirectory_DomainService
Severity	Informational
Message	There is no VM Generation ID detected. The DC is hosted on a physical machine, a down-level version of Hyper-V, or a hypervisor that does not support the VM Generation ID.
Additional Data	Failure code returned when checking VM Generation ID:%1

 

Event ID	2199
Source	Microsoft-Windows-ActiveDirectory_DomainService
Severity	Error
Message	failed to create the following cloned DC object because the object already exists.
Additional Data	Source DC: %1 Object: %2

The System event log and %systemroot%\debug\dcpromo.log might also contain useful information about cloning failures. Remember that once cloning is complete, DC promotion still has to finish. For a complete list of events—as well as expected cloning log behaviors—review “ Virtualized Domain Controller Troubleshooting.” As the author of that article, I warn you not to read it while operating heavy machinery!

Core Servers

By now, you’ve noticed that Server 2012 defaults to Core installations, which has no graphical shell and requires that you perform operations with a command prompt or PowerShell. This fits well with the VDC philosophy of deploying scalable managed environments with minimal OS overhead. But when was the last time you read an event log from the command line? There’s no eventvwr.exe tool on Core servers.Never fear! You have a few options:

• Run Wevtutil.
• Run the PowerShell cmdlet Get-WinEvent.
• Enable the Windows Firewall rules for the Remote Event Log Management groups to allow inbound communication. That allows remote tools such as Event Viewer to connect. You can use Group Policy to deploy this policy to all your existing DCs, as Figure 2 shows, and it will be on the clone copy if you run into problems.

Important: Don’t try to add the graphical shell back to a server while in DS Repair Mode. Microsoft doesn’t support this. The server won’t boot correctly, forcing you to discard the VM and start over.

Figure 2: Using Group Policy to Allow Inbound Communication

Reattempting Cloning

Once you review the Directory Services event log and fix your problem, you still have a clone server that always boots in DS Repair Mode. You can use MSConfig to remove this flag: Select the Boot tab, and then under Boot Options clear Safe boot and click OK. If you’re on a Core server, use BCDEdit:

bcdedit.exe /deletevalue safeboot

In either case, when you restart the computer, cloning will read the dccloneconfig.xml file and try again. There’s no limit to the number of times you can try, but let’s hope you get it right on the first attempt!

 

Don’t Forget the Knowledge Base

If the Directory Service events aren’t making sense or you just can’t figure out what’s gone wrong, don’t forget about your pal the Microsoft Knowledge Base. It contains all of the known VDC issues, along with guidance and repair steps. Here’s a query for all of the Server 2012 VDC articles if you just want them in your back pocket. I hope you find the Server 2012 VDC feature compelling: We designed it with your business and feedback in mind.