What do Dilbert, Billiards (or Pool), the US Navy and your SQL Server Backups all have in common?
Well, Dilbert is a fantastically hilarious comic that aptly describes failures and disasters both within the business world and within IT circles. And, in my mind there is no greater failure or screw-up in IT circles than when a critical system fails, IT professionals go to recover the system, and find that for some unknown (and unexplained) reason that mission-critical systems started silently failing a few weeks prior to the disaster in question.
Which, in turn, is where Scott Adam’s (the author/creator of Dilbert) blog post on the Illusion of Winning comes into play. Quickly summarized, Scott’s post details how if you were ever to play a game of pool with him, he’d beat you – and you’d be tempted to chalk his win up to something lame like luck. But, in his post, he points out that his win would be anything but luck. Instead, it would be all of the countless hours he put into playing pool when he was younger.
In fact, the post is so good, that I’ll wait here while you go read it.
Seriously, go read it’s a quick and brilliant read.
Back? Good. Now, there is a bit of cynicism in that post on a couple of points—but the the underlying assumption there is hard to miss: the more you practice something, the better you become at it—and the more capable you are of dealing with it (even under pressure).
Which, in turn, is probably why the US Navy is the only Navy in the world to train Junior Level flight officers in night-flight missions. Dangerous? Hell yes. Scary? I can only imagine that trying to land a 17-ton aircraft on a rolling/pitching deck in the dark is a hair-raising experience. But, the payoff is tremendous. By putting in regular, tough, hours practicing the basics (and advanced maneuvers) over and over at night, Navy pilots end up with a distinct advantage on the field when it really counts.
Confidence, Repetition, Skills, and Backups
Bringing us full-circle—to SQL Server Backups.
Because, as I constantly like to tell my clients:
When you don’t regularly practice your backups:
- You're not gaining familiarity with the concepts, techniques, and skills needed to be able to recover databases after a disaster.
- You're risking that minor change to your environment (think of service packs, network upgrades, security changes, etc) has 'silently' invalidated your backups—rendering them useless for that time you’ll actually need them (i.e. when recovering from a disaster).
- You're taking it for granted that your backups even work as expected—because you haven't actually validated that they’re working correctly.
- You're deciding that you'd rather learn the ins and outs (and pitfalls) of disaster recovery when your boss is standing over your shoulder, and everyone is waiting on you to get everything perfect so they can get back to work. (And, if you're so unprepared that you're trying to learn how to recover databases DURING the middle of a disaster, I’m going to go on record and say that your ability to pull everything off perfectly is going to run a HIGH risk of failure.)
On the other hand:
When you DO regularly practice backups:
- You're bumping into the kinds of problems and misconceptions about how backups and recovery work that will let you gain true mastery of the basics of backups and recovery.
- You're regularly validating that environmental changes have NOT invalidated your backups, and you're increasing your familiarity with how to troubleshoot the kinds of problems, 'kinks,' road-blocks, errors, and pitfalls that CAN and WILL crop up in the case of disaster.
- You're building comfort, familiarity, and the skills needed to be able to gracefully recover from a disaster – an invaluable skill to have when you need to recover databases under pressure.
- You're able to validate whether or not you have a realistic hope of being able to meet specified RPOs and RTOs in the case of different kinds of disasters.
So, when's the last time you actively tested your backups and disaster recovery procedures?