Sometimes the designers of applications are so conscience about rollout dates and budgets that these goals outweigh the good design and fault tolerance of the applications they're developing. This scenario recently occurred on a project I was involved with. The project manager wanted to reduce the number of servers in a commercial application he was implementing for a client, so he designed the forest with only one domain controller (DC) and one DNS server. The application configured in the forest was highly dependent on Active Directory (AD) schema. In other words, if there were a problem with DNS or AD, the commercial application wouldn't work.
In the rush to get the code written and meet the deadline for the commercial application, the project manager and his team never performed a backup of any of the systems—not even a System State backup—which set them up for disaster. At this point, Murphy's Law was locked and loaded. And you know that Mr. Murphy's timing is going to be perfect.
A few hours before the client wanted to start testing the commercial application, Mr. Murphy showed up. The Microsoft Customer Relationship Management (CRM) Security Service failed to start. When a service doesn't start, typically an unknown dependency, a wrong password, or incorrect permissions is the culprit. Not this time. It was a missing object from AD. There was an error in the application log, but the object's globally unique identifier (GUID) wasn't in the error message.
That's when I got a call from the project manager. After he described the problem, I said, "Well, let's do an authoritative restore. Where's the System State backup?"
In a very low tone, the project manager replied, "We don't have a backup of the System State."
"Then what about the other DC?"
"Uh...we have only one DC."
Right about then, I realized that I didn't immediately know of a way to fix this problem. I thought about using the Lightweight Data Interchange Format Data Exchange (LDIFDE) tool, but I didn't have a clue of what to query for.
I knew that the object was a tombstone and not deleted yet. A Google search on the recovery of tombstone objects in AD resulted in the Windows IT Pro article "5 Must-Have AD Tools," October 2004, InstantDoc ID 43879. One of the tools described in the article was AdRestore, a free utility written by Mark Russinovich that's available on the Sysinternals Web site (http://www.sysinternals.com/ntw2k/source/misc.shtml). One of the switches for the adrestore command is -r, which enumerates any deleted objects in the sequence of their deletion.
I downloaded this tool to a VMware virtual server on which I have Microsoft Small Business Server (SBS) 2003 installed. I created a few machine and user accounts, then deleted them, after which I ran the adrestore -r command. The utility found all the accounts I deleted and gave me the option to restore the tombstone objects. After I restored them, I opened up the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in to see the results. The objects were back in the list, although the accounts at this point had to be enabled—a minor glitch that I could easily fix when I used AdRestore for real.
I performed a System State backup on the problematic DC, then ran AdRestore. Sure enough, AdRestore's output showed that the object for the CRM Security Service had been deleted. As Figure 1 shows, the output included the object's GUID. I opted to restore the object, then I rebooted the DC.
Holding my breath, I highlighted the CRM Security Service on the CRM server and tried to start it. It started right up. What a great tool AdRestore is! I performed another System State backup of the DC while the project manager called the client and told him that he could start testing the commercial application. Then, as the project manager and I headed for the coffee machine, I yelled out, "Thank you Mark Russinovich and Windows IT Pro magazine!"
