Why Practice Should be Part of Your Incident Response Routine

Floods, fires, and power outages can happen around the clock. A cybersecurity incident is no different. Case in point: an intruder targeted RJ2 Technologies at 3:30 AM on a Saturday, said Heather Simek, vice president of the Schaumburg, Ill.-based MSP.

In the middle of the night, hackers encrypted servers on both the MSP and customer sides. The intruders also deleted the backup and disaster recovery across the board, including cloud backups, said Simek, who shared her story with other MSPs during the ChannelCon 2022 panel “I’ve Been Compromised! Now What?”

“There was no history. We were starting from scratch,” said Simek. While the team did have an extremely high-level incident response plan in place, it had gaps they always meant to fill in but never did. Instead of a nice clean backup to start with, they were scrambling.

Partner vendors were also unprepared, working with a skeleton crew and slow to respond. Looking back, she would have had more than a rough idea of what to do during an incident response.

“There were a lot of holes, holes that nobody thought of,” said Simek.

Related Blog: How Two MSPs Got Punched in The Mouth—Things You Don’t Consider Until You Get Hacked

Having a plan for a potential disaster is one thing, but practicing it is another. Practice gives you opportunities to find the types of gaps Simek and her team experienced. Practice also gives you a chance to talk to your partners about how they respond to disaster. “Find out what their plan is because you might see a hole there that they may not see,” Simek said.

In Simek’s case, the vendor had the key to a clean backup, and Simek’s team was able to download it from the secondary to the primary cloud. With that break the MSP was able to spin up some servers and get the data back down from the cloud—a process that ended up taking weeks.

Practicing Overrides Overwhelm 

There are hundreds of things to do during a cybersecurity incident and prioritizing them in the moment is almost impossible. People without a plan can experience similar mistakes: blanking out, wasting time and burning out from unfocused panic. The intensity, the stressors, the spike in cortisol—all of those physiological responses are normal in a high stress situation.

“Your brain is going to try to put you into survival mode,” said Edlin Garcia, a Ph.D. student studying mental health and IT professions at Indiana University. “If your response is, ‘I don’t want to take care of this right now,’ that’s your brain trying to protect you.”

It’s common to blank, stare at the screen or otherwise freeze in the face of a security incident or a weather emergency that threatens a network.

“Not only is the stress going all the way up from the top, it goes all the way down to your helpdesk engineer who is getting those phone calls from the customers, trying to figure out what's going on,” Simek said. “It impacts everybody across the board.”

Understanding the bigger picture and how your vendors can help you with messaging and other tasks can help reduce stress—but only if you have a good plan to follow and clear assignments for everyone.

Having a live run-through helps in various ways. If you practice your incident response plan a few times a year, people build a muscle memory response when the pressure is low.

Most MSPs and MSSPs have time and financial limitations, so planning for everything—and running every client through monthly drills—might not be realistic. Decide what’s going to help in the event that your clients are locked out of all their servers and go from there.

“I think you need to be honest about your limitations and also how much risk you have, not just for yourself, but for the hundreds of thousands of businesses that you guys all support,” said Andrew Liverman Anderson, CEO of DataStream Insurance.

He recommends simplifying a system that really supports your clients and guides them through what needs to happen during an emergency response. “Complexity is the enemy of consistency,” Liverman said.

Related Content: MSPs Double Down on Cybersecurity | Trend Watch

Practice Uncovers Mistakes

Having a plan is just part of data preparedness, according to Blair Dawson, who handles the cybersecurity data privacy practice with McDonald Hopkins in Chicago. On the legal side she helps people plan for incident responses, disaster recovery and insurance coverage.

“There's the planning and then there's the practice, which means knowing ahead of time what team members are going to handle the incident, who they are, what their roles are, and practicing with them at least once a year,” Dawson said.

In her experience working on email compromises and ransom events, practicing the plan helps answer some important questions you’ll need to know in the moment.  Who is our carrier and how do we contact them? Is there communication between yourself, your carrier and your vendors?

“When you're doing the criteria for the business continuity and the disaster recovery, you can prioritize what's critical, what's necessary and what can be put off for later.” Dawson said.