What is your worst IT nightmare? A ransomware attack? A flood in the server room? A configuration change that brings down the network? For Scott Hogle, CEO of Spire Energy, it’s a riot near his data center. Hogle remembers when a series of angry civil rights protests in downtown St. Louis two years ago made it nearly impossible for employees to report to work at the company’s data center nearby.
“I couldn’t send my engineers into work because their safety was in danger,” he said.
Hogle was one of four IT executives who participated in a panel on disaster recovery and business continuity at TierPoint’s BraveIT 2018 conference. Disasters, he noted, come in many forms.
“A disaster can be a lack of access to your data center, an act of God, damage to your IT infrastructure, malware, or even terrorism,” he noted, adding that what is important is how you react to it.
Hogle and co-panelists Chris Crosby, CEO of Compass Data Centers, Marc Carl, CEO of ControlScan and Chad Root, director of managed services for Ready Computing, Inc., shared their insight on disaster recovery in the session “What IT Nightmares Keep You Up at Night?”
They all agreed on one important point: While all IT departments face the risk of natural or manmade disasters, a good disaster recovery plan can prevent those unplanned events from becoming business catastrophes.
Panelist Advice on Avoiding IT Nightmares
DR strategies will vary based on the company’s needs and budget. The panelists shared their different experiences and advice on DR planning:
Know your RTO and RPO
Your recovery time objective (RTO) is the maximum downtime your business can sustain, and recovery point objective (RPO) is the point in time to which you want to recover. Both RTO and RPO will influence your selection of DR solutions and services. For ControlScan, which provides payment processing and other critical services for retail, the RTO and RPO are, essentially, “Now!” So the DR plan includes a hot duplicate IT environment to take over immediately should the primary system fail.
“If we go down, 13% of the gas stations in the country go down,” noted Carl. “We can’t afford to wait for a restore, so we run two completely separate environments that all applications and data touch at the same time.”
Don’t be distracted by “black swan” disasters
Too many IT departments worry about once-in-a-lifetime. But worrying about a comet strike can blind you to the dangers of more common IT disruptions, such as equipment failure, cyber-attacks and human error. “Folks focus on the black swan events, but with just 20% of the effort they could take care of the more common disasters,” said Crosby.
Design a layered notification plan
There’s no need to notify everyone for every IT disruption. “If it’s a small outage, we activate level one of our incident response plan, which means we notify the leadership, IT and a select few business people,” said Hogle.
Keep PR and customer service in the loop
For major disruptions or which affect customer data or applications, be sure to communicate with the PR and customer service teams. “They’ll need to manage the message so that you don’t lose all of your customers,” said Carl.
Invest in Disaster Recovery and Business Continuity
The annual budget should include business continuity and disaster recovery operating expenses and investment. They’re too often ignored in favor of revenue-producing projects. But imagine with it might cost your company to be offline for a week. As Root noted, “You’ll spend far less overall preparing for a disaster then it’ll cost you to react to one.”