In November 2014, Faithlife, a provider of electronic tools and resources for Bible study, was hit with an unexpected outage due to a popular product launch. The company was an early adopter of OpenStack, operating 150+ nodes, but the solution was lagging behind when it came to uptime and performance. The outage, along with the setbacks that resulted from customers’ inability to access Faithlife products for an extended period of time, made the company realize that it was time to address its IT issues.
Richard Kiene, principal engineer at Faithlife, found himself scrambling to resolve the issues and halt any lingering consequences. When it came time to rebuild the existing stack, however, Kiene wasn’t willing to simply accept the setback and roll the dice to make sure it didn’t happen again. Rather than opting to recover the existing stack, he began considering alternative options to try to discover a learning opportunity hidden in a challenging situation.
Kiene admits that the company knew its storage was running at or above capacity before disaster struck, but it hesitated to make any changes. He also says that while the solution they were using enabled the company to scale massively with commodity hardware, it proved to be “unsustainable, operationally speaking.”
“Our technology just wasn’t ready for the scale we were throwing at it,” Kiene says. “We had a big product launch and that just pushed it to the max.”
Kiene decided to build and operate a private cloud across multiple data centers using a newly launched open source solution. And while the process certainly wasn’t free from obstacles, Kiene soon discovered ways to navigate around them. Kiene says a key takeaway throughout the process was centered on open communication to the entire company, especially top executives.
“When making radical changes, you need an anchor at the top,” Kiene explains, adding that it’s a helpful strategy for any IT professional trying to convince company management to take a chance on a new solution.
“You’re often told, ‘you can do anything you want, just don’t spend more money than we already do.’ To deal with this situation and have support when things inevitably go wrong, it really helps to have someone at the top who’s willing to stand up for you when other executives might get frustrated,” Kiene says.
He also says staying true to company values proved to be a big help throughout the process. At Faithlife, the team values smart, versatile learners, and automation, over expensive vendor solutions, and using those factors as guidance helped to steer the company in the right direction.
Kiene explains that during his case study, he learned that the new solution was easy to deploy, had small failure domains and a surprisingly simple architecture. It was also must faster. When its new cloud deployment moved on from the lab to one of the company’s production data centers, Kiene knew there was great potential for the solution.
“We went from a traditional solution to cutting-edge stability, and we did about 50 times the hardware build-out in year and a half, with the budget remaining unchanged,” Kiene says. “We also opened up another data center.”
Renee Morad is a freelance writer and editor based in New Jersey. Her work has appeared in The New York Times, Discovery News, Business Insider, Ozy.com, NPR, MainStreet.com, and other outlets. If you have a story you would like profiled, contact her at [email protected]
The IT Innovators series of articles is underwritten by Microsoft, and is editorially independent.
