It's always a relief when a previous expensive lesson that happens to highlight a common IT problem happens to someone else and not you. It's even better when you have the information that caused the problem and can cite it when upper management asks why you're spending the company's money on something. In this case, we have an incident that highlights the importance of backups and, more importantly, restores, in a well-documented and irrefutable fashion. Hopefully this isn't the first time you've heard about this monumental foul-up.
Last June, the Alaska Department of Revenue, which holds the keys to the $38,000,000,000 Permanent Fund that pays out a tax rebate each year to every qualified resident of Alaska, managed to completely lose the database that contained more than 800,000 PDF files of the applications for the funds they distribute (which in 2006, came to a little over $1600 per applicant). The Alaska Department of Revenue lost its database in a very simple way--by accidentally deleting it.
The Alaska Department of Revenue now found itself in a tragicomedy of errors. If I wanted to dream up a scenario to demonstrate the importance of checking your backups and restores, I couldn't come up with a better example. Not only did the Alaska Department of Revenue manage to delete the online version of the database, department staff also deleted the mirrored database when they formatted the mirror's hard drive. The Alaska Department of Revenue then decided to use its tape backups and, to continue the now classic tale of woe, there was no tape backup. In an interesting twist, the Alaska Department of Revenue didn't have a bad tape or a poor backup procedure; the problem was simply that the person or persons responsible for configuring the backup application had never configured the application to back up the 2006 version of the PDF database.
So the data was truly gone from the system. No mirrored drive, no backup tape. And the only things that didn't make this mistake one of the worst government foul-ups in the history of computers were two saving graces. The first was that the Alaska Department of Revenue was able to recover the 2000-2005 data from the tape backup, restoring the database to its pre-2006 update state. The second saving grace was the departmental policy that stored the hard copy of applications for the Permanent Fund for at least one year.
Replacing the 2006 database was done the hard way. Over the course of 10 weeks, four employees were able to scan the rebate applications into the database, working their way through 300 file boxes of paperwork to recover the lost data. This task cost taxpayers only about $200,000--money that could have been better spent on quite a few things. But the annual checks were sent to Alaska's residents in a reasonable time, and everything in the Department made its way back to normal.
The Alaska Department of Revenue has since improved its procedures, requiring daily certification of backups and monthly updates and reviews of its backup processes. Fortunately, the population of Alaska is fewer than 700,000 people. This type of foul-up in a more populous state could have required months, if not years to resolve. And an error on this scale could easily put a small or medium-size commercial organization out of business.
Take note of what happened to the Alaska Department of Revenue, and take the time to review your own policies and procedures for assuring your ability to use backups to recover your business.
