Back in October 2012, when the threat of Superstorm Sandy lingered over the University of Pennsylvania, Donna Jacobs, IT senior director at UPenn, did all that she could to minimize the impact on the university’s 6,500-square-foot data center in Philadelphia’s University City section.
Jacobs had three main concerns related to the facility. She wanted to be sure the power stayed on and that the basement, which houses key infrastructure including UPSs, wouldn’t flood. She was also concerned that the strong winds didn’t affect the equipment on the roof.
“Any one of these could have put us in a crisis situation,” Jacobs says.
There was also the possibility that Pennsylvania or any of the neighboring states could declare a “state of emergency” and close all traffic arteries, so Jacobs and her team decided to implement a two-pronged approach. They would keep the minimum essential personnel onsite at local accommodations and ensure that offsite staff had the adequate tools and connectivity to work remotely.
Throughout the week leading up to the storm, Jacobs and her team met daily, collaborating with other parts of the organization to determine potential scenarios that could impact the availability of the university’s services. They also engaged third-party maintenance and service providers to understand the level of local support that would be available, including the possibility of getting additional spare parts on site to cut down on recovery time. While they were planning the technical solutions for the problems they might face, they also engaged clients in frequent communication about not only plans and contingencies, but also precautions the clients themselves should consider putting in place.
Before Superstorm Sandy struck, Jacobs and her team also took a proactive step, deciding to terminate the university’s storage mirroring process. “The decision we made to break the mirrors as a precaution was a contingency, critical not only to protecting data from possible corruption if communication was lost abruptly, but also minimizing recovery time in the event of a more serious situation,” Jacobs says.
The organization had already made a decision to transition its online documentation to a storage provider before the storm hit. This allowed staff to have information readily accessible during the storm. “While people are concerned about the continuity of their business, they are also trying to manage threats to their family’s safety and personal property,” Jacobs says. “Having a cloud-based tool provides a permissioned, off-premise option for documentation, diagrams, contact information, and asset data.”
Jacobs and her team also implemented a data center crash kit, which included tarps, radio, batteries, and a number of nontraditional components like two different types of masks for other types of emergencies, such as a pandemic. Jacobs says the university encourages its IT staff to keep a “shelter in place” kit for a 48-hour time horizon that includes personal items.
She says it was also helpful to read the fine print on the university’s Disaster Recovery Agreement to ensure they could put their provider on standby and not incur a fee for doing so.
While Superstorm Sandy’s fury skirted the Philadelphia area, the controls and processes the University’s data center had exercised through tabletop simulations and other staged events proved invaluable—not only from a technical standpoint—but also in preparing the staff with experiences and tools for engagment in a “crisis” situation.
The takeaway, Jacobs says, is “the data center and its operation is a business—treat it like one. Disaster recovery is not business continuity, and while they do complement each other, it’s critical to know the difference.”
Renee Morad is a freelance writer and editor based in New Jersey. Her work has appeared in The New York Times, Discovery News, Business Insider, Ozy.com, NPR, MainStreet.com, and other outlets. If you have a story you would like profiled, contact her at [email protected].
The IT Innovators series of articles is underwritten by Microsoft, and is editorially independent.
