Worst-Case Scenario: MSP Shares Harrowing Cyberattack Experience

Like millions of other Americans, Jay Tipton was looking forward to the upcoming Independence Day holiday weekend this year. It was Friday, July 2, just a couple more hours and the festivities could begin.

The CEO of Technology Specialists, a Fort Wayne, Ind.-based MSP, was visiting a longtime client’s site, reprogramming a stubborn phone that couldn’t be accessed remotely, but it was also a chance to check in, say hi to old friends and wish them a happy Independece Day. As he was wrapping up, he noticed Outlook shut down on his laptop. Well that’s weird, he thought, but, ehh, not a big deal. But that’s when the real fireworks began.

By the time he got to his car, Tipton’s office manager called to say she couldn’t get into the MSP’s ConnectWise or Kaseya accounts. Then she called back to say a client called and couldn’t access its machines either. Within just a few minutes, she’d received seven more calls from customers, all complaining their machines were acting crazy and files were popping up on their screen. Something was wrong.

By the time Tipton got back to the office and looked at one of his own screens, there was no denying it—the MSP had been hit by a ransomware attack.

“At that point, we knew a little, but not full details like how many people it hit. At first, we thought it was just us. And that’s the worst feeling you’ll ever have,” he said.

Of course, it wasn’t just Technology Specialists. The attack launched through a vulnerability in Kaseya’s VSA software is estimated to have impacted up to 1,500 companies, including many MSPs, who are increasingly the target of cyber criminals because they serve so many small businesses.

Tipton recently detailed his experience in an episode of CompTIA’s Shoering Up Security series with host MJ Shoer, executive vice president and executive director of the CompTIA Information Sharing and Analysis Organization (ISAO), in order to help other solution providers better prepare for a potential attack on their own businesses.

Expect the Unexpected

Historically, Technology Specialists backed up customer data to three disparate locations as part of its disaster recovery plan. Unfortunately, all three remote sites were targeted and hit at the same time during the attack—something the MSP hadn’t thought could or would happen.

“Normally all three are not even on at the same time,” Tipton said.

It was gut-wrenching to think customers, some for more than 20 years, could be impacted. Tipton blamed himself. The ordeal left him mentally, physically, and emotionally drained. He shared his story to hopefully spare other MSPs a similar experience.

“I didn’t eat for three days. I lost 10 pounds. I was here 20 hours a day the first three days and three weeks later probably have 300 hours in,” Tipton said. “You need to have a plan for an attack, but you can’t plan for the emotional impact. I went through the whole gamut of emotions. Customers entrusted me with their data, and they got sick.”

While the MSP lost a handful of clients because of the attack, the vast majority of his 60+ managed service customers and 100+ project-based customers have stuck with the company through and after the crisis, Tipton said.

“99% of them read the articles and understand that there’s nothing we could have done to stop it and that nothing is 100% safe anymore,” he said. “There are a few that don’t get it, won’t ever get it, will never understand, and say it’s all our fault. I can’t change their minds, so I’ll just shake their hands, part as friends and go on with life.”

With so much to be done and so much to worry about, Tipton said he assigned coordinating the recovery and remediation process to someone else—a smart move, he said, because he was too close and too emotionally tied to his clients and the situation.

“I told my office manager to run with it. I was going to focus on getting medical clients back up and running so I said I need you, take this and run. It’s not because I didn’t want to do it, but I kept jumbling the list of things that had to be done. One thing changing in our disaster recovery plan is that Jay is not going to be the coordinator,” he said.

An Attack on You Is an Attack on All of Us

Once the word spread among CompTIA members that Technology Specialists had been hit, more than 80 tech vendors, distributors and other MSPs offered their support, including offers to fly in to be extra resources onsite if necessary.

Tipton said several peers, ex-employees and even customers have helped with the remediation process, from delivering new devices to checking in to make sure he’s staying on task with what needs to be done.

“I was totally overwhelmed with the offers of help, hardware, software, and even money. I think that was the hardest thing,” Tipton said. “All the offers of help from people I guess I have helped and given answers to over the years. Even when dealing with this, I have stopped and made some time to help others in the same boat as I am. I have shared a program my ex-business partner wrote to clean up the hard drives. I have always been the one giving, this was quite different accepting help. A big pill to swallow when you are the one normally giving.”

The response from CompTIA members to Tipton’s plight (as well as other MSPs) is testament to the brotherhood of MSPs, according to Shoer, who noted that Tipton had once helped him out when he was an MSP himself.

“I had an issue, I put out a request. I didn’t even know you at the time, but minutes later my cell phone rang, and it was you with an answer to my problem,” Shoer told Tipton in the video. “I’ll never forget that. I know I’m not the only one that you’ve done that with. It’s only fitting and good karma that so many people stepped up. That’s one of the best parts of this industry we work in. Everybody has really adopted that stance. This was an attack on all of us. That’s how industry responded.”

Customers Aren’t the Only Ones to Worry About

Another potentially overlooked aspect of a ransomware attack: the impact on employees. Just like an MSP’s leaders, its front-line sales, technical and administrative staff are worried about their customers, their jobs, their reputations. Tipton recognized that and tried to assure them early on that this too shall pass.

“They are rock stars,” Tipton said of his team. “We had a brief meeting on [July 5]—it was all-hands on deck, and we talked about the overtime and once we get past this, we will work on getting it paid. But for now, it’s been a straight 40-hour plus 8-hour overtime pay per week. They were good with it. Everyone has been with us for eight years or more, so we know the good and the bad times. We have a lot of trust in each other.”

Since the July 2 attack, Tipton and Technology Specialists’ team has spent a lot of time second-guessing their actions and looking for things they could have or should have done differently, lessons learned if a similar incident happens in the future.

Thankfully, all the data managed by Technology Specialists could be restored but it took a long time, mostly due to slow download speeds. “Everybody was trying to hit the same data centers at the same time [for recovery]. You never think about comparing [your situation] with who knows how many other MSPs trying to download at the same time. You can’t really plan for that,” he said. “Now I have a big wall of Post-It notes full of things to change going forward.”

Sharing the experience of surviving a cyberattack is no small ask. Tipton and his team still have much work to do and much to recover from, including emotionally. But he hopes sharing his story will resonate with other MSPs and help them better prepare for the future.

“We still have a few systems left to get clean and a ton of cleanup work at each site. It will be a while before we’re done with this,” Tipton said. “But I want people to be aware of our experience—the whole other side that you don’t hear about. You can’t fully prepare for it, but you can at least be aware this is going to hit you like a tsunami.”