The well-known motivational speaker Zig Ziglar is famous for his quotes, and one of his most popular is “Expect the best. Prepare for the worst.” While he may not have had disaster recovery or backup in mind, the phrase clearly fits in many situations. In fact, it’s that mindset that saved digital sign and display designer and manufacturer Dualite from a major ransomware hack last year—a network-wide attack that encrypted everything from desktop PCs to its ERP files to its engineering and art files.
Back in 2014, IT manager Chris Rolke noticed that Dualite’s aging backup system, which consisted of multiple tape drives across multiple servers, couldn’t keep up. “We back up about 13 terabytes of data on a regular basis, and we didn’t feel we could trust a system that was becoming obsolete,” he explained.
Even more critical was the fact that the backup window was much too long. “It was taking us 20 hours to do a backup set,” he said. “At one point, we were running four separate tape drives on four separate servers just to do backups.”
The nail in the coffin was repeated attempts to back up and restore using existing technology. Not only did it take multiple tries to back up data, but restores based on multiple tapes often led to unrecoverable failures.
That was enough for the IT team to replace the system with the Unitrends Recovery 824S hardened, Linux-based backup appliance. Rolke preferred a Linux-based appliance to a Windows- or Mac-based system because he felt that it takes more specialized expertise to hack into Linux-based technology. That choice turned out to be fortuitous.
'Very Bad Things Happening'
The night of June 10, 2020, started like many others. Rolke was performing a data refresh from Dualite’s Oracle ERP production to a test environment, a process that could take several hours. It was late, and he decided to get a few hours of sleep while processes were running. But what he saw when he woke up to check on the progress at 2 a.m. was shocking.
“The system was still down, so I got in another way and started seeing some very bad things happening right in front of me,” he remembered.
Rolke immediately got in his car and was at the office by 3 a.m. trying to figure out what was going on. He started unplugging anything and everything feeding lines to different areas of the building to minimize the damage and then started shutting down all running applications.
He finally determined that the cause was a ransomware hack that had been funneled through Microsoft’s PowerShell task automation tool, and that it originated in Ireland using stolen credentials. He immediately called coworkers in, and by the time he went home at 7 p.m. that night, Oracle was back up and running. Because the system had previously been set up to back up the ERP system hourly, Rolke’s team was able to restore the system back to just a few hours before the attack.
The team then spent the next few days getting application servers and data storage servers back up and running, and then rebuilding desktops and other core functionality.
The reason it took only a few days to recover, Rolke said, was because the team had upgraded to a more modern Linux-based backup system years earlier. “If we had still been on the old system, I think we’d have been out of luck,” he said.
Now that the worst is behind him, Rolke has had time to reflect on what else he could improve in his backup and security environments to be ready for the next ransomware hack. In addition to upgrading to the newest version of Unitrends, the team made significant changes in how passwords and VPN credentials were handled. It also disabled PowerShell across the board, giving access only to Rolke.
“We are doing everything we can to lock down our systems even more tightly,” he said.
