In March, we asked you to tell us your Worst Disaster Recovery Nightmare story. We've selected Andrew Barr of the UK as our winner. Andrew will receive a copy of Disaster Recovery Pro from UltraBac.
The top 5 stories we received, including Andrew's are posted below.
Winning Entry
Well it all started when I arrived at another school, they had five servers all running windows 2000 and I was there to install a new server to replace their old Active Directory Server.
It all started OK, I asked if they had a full backup, YES they said (always check yourself!), so off I went. Installed the new server added it to the domain. Made it a DC and forced a replication. Turned all the other servers off, So far, so good. Started to copy users data from the old server to the new server and the new server just froze, no blue screen or reboot just sat there with no mouse and no keyboard response. Right, lets reboot the server, nothing, ok lets start from scratch again..... Rebuilt the new server, turned the old server back on, checked the AD, nothing no users groups OUs just the default users.(OK this was going to be fun) re-installed win2k server on the old server, got the backup tape, inserted backup tape and went for the full restore; only users data backed up and no system state, marvelous. 11pm at night with only users folders, time for an early night and an even earlier morning. At 5:30am into the school armed with a directory of users folders and a windows 2000 server CD, began to install win2k on the new server. After 3 hours of installation of drivers, users data and setting up the AD it was time to script the user creation. With the aid of MS Excel and the concatenate command I set about creating scripts to add the users and computers to AD, reapply the security to the users folders and add the users to the correct groups. Now with the time coming up to about 3pm it was time to check that the pcs could still logon, all was well. Now for the other servers 1 print server (luckily only 10 printers in the school) 1 ISA firewall, 1 exchange server –were all rebuilt to get rid of any knowledge of the old server, and 1 apps server which just needed adding back to the domain. They now use the old server as a AD controller for backup. So at about 1am(the next day) all the servers where back up and running, the machines logging into the domain and the schools network administrator looking worst for ware, but with a smile on his face. All we had to do now was install the backup software and make sure that it backed up the system state, C drive and the user’s data. I also informed him of autoloaders, so that the other servers could be backed up from a single location with out them needing to change tapes.
So at 2:30am Saturday morning, it was time to go home, only a 3 hour drive – Nice. Since this fun couple of day, they have never had a problem with backups, servers crashing or nothing that a simple restore wont fix.
Other entries from the top 5
***This could have been my worst nightmare, but it worked out OK. Just thought that I'd share it with you.***
It was about 4 years ago and the network that I had "inherited" had one NT PDC with no BDC's on it. The PDC took a dump and my PFY had scheduled a checkdisk upon reboot. Needless to say, it kept failing because one of the disks was toast. Oh, ya. Did I mention that this had everything on it? Printing, internet proxy, network storage, database. Geez...
Nice planning. So, what I did was boot it up with a DOS boot disk, PKZIP'd the registry (the system disk was still FAT) and copied it to diskette, walked it to another NT machine, unzipped the registry, flipped the checkdisk bit from "1" to "0", then reversed the procedure to get the registry back in the PDC. It worked.
Not too bad for doing this all at 3:00 am after working all day. Thank God for those little beans that, when ground, make coffee.
Well, my story is not a true "enterprise" disaster recovery story, but never the less, it was my worst experience. It was 1996, and I was a college student in Pueblo, Colorado. My father's business, in Denver, was just starting its year end financials when their bookkeeping computer died. They called me since I have always been their unofficial IT support. They had waited until the last few days to get their numbers together, so it was urgent that it be fixed quickly. Unfortunately, it was the middle of midterms, and I had a test that night! Of course, I had to get my dad up and running, so after the 7:00am call, (very early for a college student), I took off to Denver.
Their bookkeeping software was on a 386 system, running Windows 3.11, and the hard drive was reduced to a constant click. With a quick trip to store, I had a new hard drive in hand. Formatted the drive, loaded Windows and his accounting software on it, and, here it the kicker, restored his data from 37 5.25 inch floppy disks! I made the 100 mile journey back to Pueblo just in time to walk into the classroom to take my midterm. I seem to remember not doing so well on that test.
Not the greatest story, I know. But thought I would share anyways.
Thanks.
On a Friday afternoon a SCSI disk in a RAID 1 volume failed. Right after that failure the controller bit the dust as well. This server was the main applications server for all the financial data and of course it was close to the end of the month so everyone was wanting reports. A exact spare controller was located and a spare disk was procured as well. The OS was reloaded on the server and attempted to bring up a new RAID 1 with new disk and the old disks. The volume would not mount so we decided to go to tape to restore after wiping out the old drive in the RAID 1 array.
Now this is about 40G of data we are restoring and it was on a DLT IV tape. At the time the restore started it was about 7am on Saturday morning. We left to get some sleep and come back about 5pm to see how the restore was going. After arriving on site we heard a weird noise coming from the rack. Upon inspection we found the tape had broken in the drive and was wrapped around the spools inside the tape drive!!! The only other tapes were offsite and we had no access till Monday. A older tape was found and we tracked down a spare Tape drive and started a restore. This was at about 11pm on Saturday nite. Finally on Sunday the restore was done but we ended up missing about 2 weeks of data. We waited till Monday and got the tape back from offsite and attempted a restore with it as well. Started the restore at approx 11am and then went to lunch about 12am. Got a phone call on the cell at about 1220pm saying that a weird sound was coming from the server room. I knew exactly what that sound was. Upon return the offsite tape did the same thing the backup tape did. We ended up having to re-enter two weeks of data. After that we switched to disk based backups.
My experience started in the early 90's with low disk space on the 89 MB HD in the 386 DX 25 I was sharing with my younger brother. In freeing up disk space I moved my completed AmiPro 3.1 documents in c:\lila\old to a 1.44 MB disk. Everything was fine until my Dad needed a 1.44 MB disk, but could only find 720 KB disks in our cases labled blank disks.
Realizing I had free disk space I issued the diskcopy command for my floppy using a 720 KB disk as the destination and then cleared the old disk by issuing format B: at the DOS prompt (A: was a 1.2 MB disk drive).
When I inserted my new 720 KB floppy after my Dad had stored info on my old 1.44 MB disk DOS acted like it needed formatting. I learned that disk copy only works with floppies that have the same capacity. Oops!
