"Our agency worked 12 days straight taking about 2000 claims after Hurricane Charley. It was organized chaos around here. There were people everywhere in this office, employees running around in shorts and sweating bullets, and clients passing out from the heat. But despite all this, we had our data. We heard horror stories of other agencies whose offices were destroyed, and their service reps were out on the street in the hot sun writing claims by hand."
As Kathleen Russell tells it, in the aftermath of a hurricane, all hope for riding out a disaster lies in how quickly your organization can regain access to business data. Kathleen, who manages and oversees IT functions for the 39-employee Key Agency in Englewood, Florida, was keenly aware that the agency's continued existence depended on whether its data- and business-recovery strategies worked as planned when Hurricane Charley roared up Florida's southwest coast on August 13, 2004. Not only was the agency within striking distance of the hurricane, which made landfall about 10 miles from Key Agency's main office, about 12 percent of the firm's 17,000 customers were in the direct-hit area and sustained catastrophic property damage. Protecting and restoring data with minimal interruption to business operations is the ultimate goal of any recovery strategy. In a recent interview with Windows IT Pro senior editor Anne Grubb, Kathleen discusses how a cost-effective backup and recovery plan geared toward the needs and limited resources of a small-to-midsized business (SMB) enabled Key Agency to resume operations and start processing customer claims on site less than 24 hours after the hurricane hit.
Unlike the areas affected by the December 2004 tsunami, you had adequate early warning about Hurricane Charley. What was your timeline for preparing for the hurricane?
We started preparing on Thursday, August 12, the day before Charley struck, when we knew it was on its way up here. We powered down all our servers around 2:20 p.m., then covered them with plastic and placed them on old insurance manuals to lift them in case of flooding. Then everybody went home to hunker down. About 24 hours later, on Friday afternoon, August 13, the hurricane came ashore near Port Charlotte, where many of our insured customers are located. We're always wondering, when is the big one going to hit? And it did.
The agency owner called me at home at about 9:30 p.m. He'd been out already and told me that the main office and branch office (in North Port) were both fine, but the main office had no power. The branch-office server is in the main office, along with our main server, a fax server, and a backup server, all on Windows 2000 Server. So I'm thinking, "Okay, I don't have computers, I don't know how long it will be 'til I get computers because I don't know how long we'll be without power. All I know is, we're going to be completely swamped with claims, starting Saturday morning."
Shortly after I spoke with the agency owner, at about 9:50 p.m. I made a call to our offsite backup service, LiveVault, which provides a continuous, up-to-the-minute backup by copying our database via the Internet and taking a picture of the changes about every 30 seconds. We signed up for the LiveVault service through Applied Systems, which provides our agency management application. LiveVault and Applied Systems have an arrangement where if you lose your server—say in a fire or it's infected with a virus—Applied Systems will download the backed-up data from LiveVault into an online version of the Applied Systems application, so you can start processing claims even if you don't have your server.
After I spoke with the LiveVault rep, he started downloading our backed-up data into Applied Systems' online service around midnight. The download took about 3 hours. LiveVault didn't bring back everything, only the last 5 years—5.3GB—of data.
I arrived at the main office—which was dark and hot—about 7:45 Saturday morning. Our employees were starting to drift in. I got a call on my cell phone from LiveVault to say that my data was up and ready to go. Right after I hung up from that phone call, the first customer walked in the door to make a claim. He had lost everything; his entire house was just gone.
Did the agency have a plan for continuing operations during a power outage?
We have diesel generators in the building to power the servers during an outage. In addition, the agency owners have motor homes, which also have generators, so our plan was, if necessary, to operate the business out of the motor homes and access the Applied Systems application and our data there via laptops. But a funny thing happened. No sooner had we plugged in the generators and were ready to get our computers going again, when our power was restored! It was a good drill, though, because we found that our backup and restore plan worked. When we came in that morning, no matter what else happened, our data was available both to the branch office and to us.
We still do a manual tape backup daily and keep the backup tapes in a fireproof safe on site. Before the hurricane, I passed out copies of all the tapes to the different department heads. We also printed reports that we could have worked from, but they were run at the beginning of hurricane season in July, and the data was incomplete. Restoring the data from tape or using the reports to manually write claims would have taken much longer than restoring data and downloading it into claim forms through our offsite service. It would have been like looking for a needle in a haystack, trying to find the right data.
Our agency worked 12 days straight taking about 2000 claims after Hurricane Charley. It was organized chaos around here. There were people everywhere in this office, employees running around in shorts and sweating bullets, and clients passing out from the heat. But despite all this, we had our data. We heard horror stories of other agencies whose offices were destroyed, and their service reps were out on the street in the hot sun writing claims by hand.
It sounds as if being able to restore data and business operations so quickly gave Key Agency an edge over other agencies in serving customers after the hurricane.
Right. It all comes down to having that data available. If you have your data, you can work. Until you file that claim, nothing's going to happen; you can't help the customer. When you have a major disaster with so many people filing claims, you're either standing in line behind 20 claims or behind 20,000. Your customers will be even more upset if their claim is put off yet another couple of months because there aren't enough adjusters. Restoring our data and operations quickly allowed our customers' claims to be processed fast, compared with the people who were out taking claims by hand, then driving to someone's house and faxing them.
Did you reevaluate your disaster recovery plan or storage needs after Hurricane Charley?
Actually, I started keeping a folder, which I called the Charley folder, which turned into the Charley, Frances, Ivan, and Jeanne folder. We added information to the folder whenever another hurricane started our way. We got much better at our getting-ready plans. Soon, preparing for a disaster became old hat. One change that the agency will probably make is to outfit the main office with permanent generators, so that we can generate enough electricity to power the servers and the air conditioning. In hurricane season, when it's 95 degrees and 100 percent humidity, your servers—and your people—will overheat in a few hours without air conditioning.
Key Agency is probably like other SMBs, in that you have to put together a data and business recovery plan using limited IT resources. What can other organizations—especially SMBs—take away from the lessons you've learned?
For one thing, you might think you're prepared for a disaster, but you never really know until you go through it. After the hurricane, I was thinking, "Okay, we've been paying for this online backup—will they be there when I call them at 10:00 p.m.? Is our data there? Can they restore it?" It took such a burden off of my shoulders to know that the online service was right there, available.
The cost of the online recovery service is nominal, especially when you consider that you don't have to restore the data yourself. I'm not a full-time tech person; I'm an insurance agent who oversees IT for our firm. I don't have $60K in payroll to spend on an IT person. For a business of our size, the only viable solution is to outsource our data protection. As the manager here, I can't tell you how relieved I was to get our people the data that they needed. Customers today expect you to be up and running no matter what. I've heard our customers ask, " If this place is hit by a hurricane, how is my data protected? Will you be able to file a claim for me?" It's nice that we have the right answer for them.
