Over the last several years, I have been blessed with the opportunity to serve with the non-profit Information Technology Disaster Resource Center (ITDRC). For those of you unfamiliar with ITDRC, this organization is called upon during major natural and man-made disasters strike communities throughout the world. Their mission is to help people in need by re-establishing communications and providing technology resources. ITDRC works with non-profit NGOs, federal, state, and local emergency management teams to assist in recoveries nationally and internationally.
Several weeks ago, I was able to attend my second ransomware recovery effort. As an IT professional, it’s hard to grasp the full extent of the impacts of government technology and modernization on a community. In society, we have become accustomed to technology, reliable networks, with constant access to our data and internet. We have become reliant on technology without realizing our dependance.
When I arrived, I found a beautiful community that was struggling to provide day-to-day services at a level of service their citizen’s had come to expect. This community was small, tight nit, and connected. While their resources were limited, I quickly learned that this community was ready to do what it takes to meet this challenge. I was lucky to find a well-rounded team, that had skillsets in business operations and technology. Everyone was willing to share and be supportive of each other. The local team embraced support coming to help. The first evening, I spent several hours trying to assess and understand the impact of the ransomware and the previous accomplishments. My goal was to understand the current situation and needs to determine the next steps to recovery.
My first task was to evaluate a few systems that had been impacted. I quickly gained insight into the damage that the Royal ransomware left. Most files had been encrypted, hard disk space was consumed leaving disks full, and even some were corrupted and unbootable. This variant was aggressively working to prevent analysis and forensics. We found that it attempted to steal focus and would crash the windows explorer process every few seconds.
When you arrive early in a ransomware event, infrastructure becomes one of the most critical components to your recovery. It starts with network. The key to recovery is evaluating the network, hardening it, and ensuring you have reliable connectivity throughout your environment. Without this, communications suffer, resources will spend time waiting, your progress will slow, and recovery efforts will be hampered. Without a reliable network, systems teams will become frustrated with their restricted ability to complete tasks. This burnout will occur fast in a critical recovery.
The second component to recovery is understanding the need to investigate the event. Figuring out a way to preserve critical data is paramount in a response. While this effort may slow recovery, it’s important to understand how the threat actor moved through the environment. How do you protect yourself without knowing how they got in, and what they did once they established privilege? Most environments make the most of available capacity and resources. Typically, these systems don’t have storage capacity to duplicate resources consumed daily. How do you get storage onsite quickly to enable the copy of critical evidence that will be used to investigate what happened? Answering this question can change your path to recovery.
Once you have established a basic understanding of how the attack happened, you can work towards building protections against these types of attacks. It’s critical at this point to start prioritizing what security procedures you need to implement. During an event, it can become tempting to implement new protections and re-invent the wheel. While it’s a great opportunity to implement new process, it can have a big impact on your recovery time.
In my limited experience responding to high impact security events, I have compiled a list of factors to consider.
Are you leaving behind an environment that the organization will be able to support?
During an event, everyone’s focus is on recovery. Many times senior leadership moves to tunnel vision. Recovery at any cost. While it’s important to move forward and make changes to enhance your defenses, it’s critical that sustainability is considered. Are you leaving an environment that is supportable to the staff expected to manage it?
Security is not about impenetrability, it's balance.
Every organization has a soft spot, something that is vulnerable. Every year, this target moves and evolves. It’s important to realize that no-one is immune to cyber security events. If you become consumed building Fort Knox, it will hinder your ability to recover and deliver services to your customers.
Trust.
If you have been breached, this doesn’t come naturally. Your organization’s ability to trust individuals to assist in your recovery will drastically impact the time to recover. Technology is complicated, it takes a lot for a technoligist to step in and understand someone else's environment. It's critical they have access to build an understanding of how each component works. Technologist waste critical time trying to discover what is inside the black box. Build a method to track/monitor resources, provide access and documentation to components, and establish trust quickly. Evaluate and asses your resources based on merit and available information. The reality is it can’t get much worse.
Engage your vendors early.
Engaging your critical infrastructure and application vendors early is key. Many enterprise vendors have teams ready to help. Notification can reduce the time for resource assignment significantly and assist is speeding up recovery.
Establish recovery priority.
While core infrastructure is the base to recover, each department within your organization needs a priority. What is the order to recover services?
Logistics.
While this sounds simple, it’s critical to establish a team to deliver everything from the simplest, to the most complex needs. This is a key component to successful recovery. Do you want your resources tied up trying to figure out what to eat and drink? During recovery, team members are working long hours and doing their best to bring services back. There are also basic IT supplies that need to be considered.
It's a marathon not a sprint.
Ransomware events don’t conclude as quick as they start. These types of events have far reaching impacts. While they may only affect technology, they have many similarities to natural disasters. Imagine a tornado or hurricane hitting your home. It will take countless months to recover and rebuild. While the next item on this list is critical, embodying this philosophy is crucial. When you are recovering, it's critical that key team members get time to get away, rechange, and come back refreshed. Sometimes that means leaving early or taking a day or two off. Most people expect recovery to happen within days; but significant events are measured in months.
Continuity in IT leadership.
This becomes more difficult in small organizations with limited resources. This continuity establishes a shared goal, trust, a plan to move forward, and can assess progress. Every time a handoff occurs, days are lost, direction waivers, and the recovery slows.
Access to an organizational decision maker.
The ability to make decisions quickly and move forward directly correlates with the recovery time. Every minute wasted waiting for an answer hinders forward progress. Organizational leadership is key, vendors won’t be able to fill the shoes of executive leaders.
From this response, I found competing priorities, each not aligned with one another. Every aspect is important, not one more then any other. It creates an interesting dynamic, a dilemma, on how to find the middle of the road, how to get enough information while recovering in a timely manner, and how to build a secure environment without comprising the ability to administer it.
I'm thankful for the opportunity to be invovled and the lessons it has taught me.
