10 Critical Elements to Great Security Incident Response

Security incident response is the cybersecurity capability that can directly prevent a company from becoming headline news. It is one of the most important and unquestionably the first cybersecurity capability that should be built and matured. When everything else has failed, and it will fail, a documented, well-rehearsed and efficiently executed incident response plan can ensure that the failure is graceful and outcome effectively managed within the law for all those that are impacted. 

Ahsan and Chris have over three decades of combined experience in building and operating security incident response capabilities.  We’ve managed hundreds of security incidents, many involving sophisticated threat actors, requiring customer and regulator notification, and taking months to achieve full resolution. We have a deep understanding of the people, process and technology needed to build a highly-effective capability and want to share our perspective on 10 critical elements that we believe you must get right to build and operate this capability successfully.

Security incident response is a complex capability with a lot of moving parts and dependencies. The 10 critical elements in aggregate do not constitute a comprehensive and mature capability.  They are simply some of the most important elements that we believe that you must get right. They aren’t listed in order of priority but are instead categorized as things that should be done before, during and after a security incident.

Before an incident

You want to be as prepared as possible prior to a security incident.  You should have your people, process, and technology in place. There shouldn’t be any ambiguity around roles and responsibilities and you should have gathered critical artifacts required to support the capability. Make sure as many questions as possible have been answered.  Questions like: Who makes which decisions? What are the different severity levels of incidents? How and who should be involved and when should they be involved? Who has the authority to shut down operations? Do we have tools for secure communication? Who decides whether ransom should be paid?  How will it be paid? Who is the law enforcement contact? And so on all need to be answered before you have an incident. Much of what you come up with will need to be tailored to the specifics of the incident, but building these from scratch during a high-pressure situation will likely result in poor decisions, stakeholder frustration, additional impact to your business, and a longer time to resolution. 

Here are four elements that you should sort out prior to an incident:

(1) Establish internal partnerships - The larger an incident becomes, the more teams will need to be involved and the more politics will come into play.  Working with teams prior to an incident will save a lot of valuable time during the incident, for example trying to identify system or process owners. Including these teams in the creation of the final plan will increase their level of buy-in resulting in them seeing it as “our plan” instead of “the security team’s plan”. Be thorough and inclusive in approaching internal teams. It is important to meet with all major functions and stakeholders in the organization to:

• Identify when each team should get involved in a security incident.
• Identify the responsibilities of each team and the actions they will take during an incident.
• Identify the people to contact for each team in the event of an incident -- primary, secondary and tertiary contacts are ideal, but this depends on the size of the organization.
• Work with each team to create a playbook or checklist for the actions they will take during an incident to reduce the number of decisions that need to be made during an incident and ensure consistency from incident to incident. 
• Train each team on the whole incident response plan.
• Inform stakeholders and senior leadership of their responsibilities and how communication updates will occur.

(2) Establish external partnerships - Most organizations outsource some of their business activities.  As an extension of the business, outsource providers need to be included in building and operating the security incident response capability.  In addition, for larger incidents, we recommend leveraging external firms who have expertise in security incident response. Some key external relationships include:

• Any third party performing critical work on behalf of the company.  You can identify these while working with your internal partners in element 1 above. Also, most companies leverage cloud service providers who are critical to getting the information needed to conduct investigations.
• At least one third-party incident response firm and external legal counsel with expertise in security and privacy incident response should be on retainer.  The value of bringing in an external firm is twofold. First, they do this all day every day and have tools, people, experience, and connections that can be extremely useful in handling significant security incidents. Second, if the incident is large enough to make the news, then perception becomes a real concern.  Like it or not, it’s always best to bring in an impartial third party to help with a significant incident since that incident happened on your watch.
• Law enforcement.  There are a number of pros and cons that need to be considered prior to engaging law enforcement. The internal legal team should be consulted to establish a protocol for when and how law enforcement will be brought in.  However, you should start building relationships with federal and local law enforcement immediately so that you aren’t doing introductions in the middle of an incident. Both the FBI and US Secret Service have highly capable cybersecurity agents who can provide extremely useful support during incidents.
• Extensions of existing teams.  Some internal teams won’t have the resources, skillset or experience to fully and effectively contribute to an incident response and will need to bring in outside help.  This is common for communications and IT teams among others.

(3) Have a written plan - This isn’t a revolutionary idea, but it surprises us the number of organizations that have elements of a plan, but not a comprehensive plan in writing. An incident response plan needs to have a balance between static and dynamic components, regimented enough to ensure that incidents are handled in a consistent manner, yet flexible enough to accommodate the nuances of each unique incident. It needs to be tailored to the needs of the organization with key decisions, workflows, processes and stakeholders documented. Here are some key items that need to be included in the plan:

• The core incident response methodology and outline of the incident response process
• Description of roles and responsibilities
• Playbooks and checklists (e.g. standby statements) created by each team
• Contact information for internal and external resources
• Links to additional artifacts needed for incident response (e.g. network maps)

(4) Practice the plan - Every execution of the security incident response plan is an opportunity to create more team cohesion and to identify gaps, decisions that need to be made, and artifacts that need to be created.  While performing tabletop exercises is a valuable practice, they tend to happen only once or twice a year and as a result play a relatively small part in what we are recommending. The best way that we have found to regularly practice the plan is to manage smaller incidents as larger incidents. This makes team members go through the motions more often in a less stressful scenario so that when a large incident occurs the team already has some solid experience.  

During an incident

People’s behavior can and often does change when they are placed in high-pressure, high-stress situations like a security incident. Common negative behaviors include jumping to conclusions, becoming impatient or freezing, starting a parallel response initiative, reacting too quickly or too late, or in general just losing perspective. This is usually exacerbated by information that is scarce and dynamic -- especially at the beginning of an incident. We recommend that you focus on these five elements to keep your incident response on track:

(5) Leverage command-and-control leadership - This isn’t our recommended approach to address most corporate leadership challenges, but a command-and-control structure is often necessary in managing security incidents.  There are just too many people and actions to coordinate to have everyone going in different directions. The incident manager, is the single leader, who has the accountability and authority to ensure that the incident response plan is being followed by everyone.  We have found that in most cases, people respond well to, and are often relieved by someone stepping up and taking charge. In addition to the incident manager we also establish an Incident Command team which is comprised of the CISO and senior representatives from the legal, privacy, and public relations teams.  Incident Command offers support and empowerment to the incident manager, who is responsible for ensuring adherence to the plan. In turn the incident manager escalates major issues and decisions that could have significant business impact to Incident Command. Incident Command may have a different composition based on the unique needs of each company. 

(6) Delegate and support - Our intent with command-and-control leadership is in managing the overall incident response process (e.g. establishing incident severity, communication update intervals, meeting cadence, etc).  It doesn’t mean that the incident manager or Incident Command for that matter should be making the front-line decisions on technical resolution or incident support functions. Those decisions need to be delegated to the appropriate team which is why you have them involved in the first place. Delegation should be to a single named individual who is then held accountable and supported.  Never delegate to teams - teams don’t make decisions, individuals do. The incident manager and Incident Command need to validate that the proposed actions fit into the overall incident response approach, but legal decisions should be made by the legal team, communications decisions should be made by the communications team and so on.

(7) Follow the plan - Stick to the plan that you’ve created.  It’s the best way to keep everyone coordinated and focused on the same priorities.  It also helps demonstrate consistency to senior stakeholders who aren’t in the trenches working on the incident.  To them, the incident response is a black box and the only information that they receive will come from the incident team  (or through back channels from their reps on the team, but that’s a whole other conversation). If communication is thorough, tailored to the audience, and delivered with consistency that will build credibility. Going dark or being inconsistent with communication during an incident is the worst possible course. This will result in stakeholders questioning whether negative outcomes from the incident were avoidable with better leadership.  We are not suggesting that you should never make changes to the core process in the middle of an incident, however if such a decision is made then it must be documented and stakeholders informed. However we would recommend that if it can wait, that it’s better to remain consistent and deal with suggested changes in the postmortem.

(8) Manage incident resolution and communication - There are two core objectives in incident response: 1) resolve the incident and 2) manage communications about the incident.  One of the biggest mistakes made in incident response is trying to have the same people resolve the incident and communicate about the incident -- it just doesn’t scale.  It’s necessary to have an incident resolution team and a communication team. Those teams need to come together at regular intervals to ensure that communication is accurate and the technical resolution team is kept up-to-date on relevant updates from stakeholders.  But if the same people manage both aspects, then both workstreams will suffer likely resulting in a longer and potentially more impactful incident and a number of frustrated stakeholders desperately trying to get updates. Also, It is extremely important that you have a disclosure list of everyone who knows about the incident and they should be made aware of strict communication protocols around the incident. Communication should be handled by a specific person or team and needs to include internal (employees and contractors) and external (customers and partners) parties.

(9) Stay calm and productive - As security incidents grow in criticality, the impact, or at least potential impact to the company increases. More people will get involved in resolving and communicating about the incident and senior executives will want updates on progress and to weigh in on decisions and approach.  This can be a tense situation leading a diverse group of people with different motivations, seniority, temperaments, and incident experience. In this situation, we believe that elements 5-8 above are your best guide on what to do. Our advice here is on how to do it. The incident manager and Incident Command set the tone for the incident response team and that tone should be of calm, professionalism and urgency. This will be your best bet at moving the incident along productively while instilling confidence in your stakeholders.

After an Incident

Once the incident is over and all services have been restored, the last thing anyone wants to do is talk about the incident.  However, that’s exactly what needs to happen, sooner than later, so that lessons learned can be captured while they are fresh in everyone’s minds. The final element below is one of the most important on this list and when done properly can increase stakeholder engagement, reduce the likelihood of future incidents, and reduce the impact of the incidents that do occur.

(10) Perform an incident postmortem – The key here is brainstorming with key participants and stakeholders to get an understanding of what went well, what didn’t go well, and suggestions for improvement.  Get the perspective of anyone who was central to resolving the incident or who was affected by the incident. Getting this information can be done in 1-on-1 or group settings.  It can be completed in a meeting, email or survey. Decide on the best structure based on the organization and the incident. One suggestion would be to send a survey to all of the key contributors and ask them to pass it along to their team.  This will help you gather a broad set of opinions. Then meet as a group with the core members of the incident response team and get their opinions. Finally, reach out to senior executives individually through email asking for their feedback and offering to have a follow-up phone call if needed.  Aggregate and process the input, decide on the necessary changes, put them in an action register, assign ownership and follow-up until the actions are completed. A root cause analysis of the incident should also be performed with findings being integrated into the action register. This step creates a virtuous cycle of improvement of the security incident response capability and demonstrates to key stakeholders that you are listening to their input and serious about improving.

Conclusion

Having a strong incident response capability is essential to managing the cybersecurity risk your company faces. We hope that you have found value in us sharing our thoughts on critical elements for building and operating this capability. If you don’t currently have a comprehensive plan in place, we suggest you prioritize creating one.  We understand that this can be difficult if you have limited security incident management experience or resources. If that’s the case, please reach out and we can help point you in the right direction.

Chris Houlder is a Chief Information Security Officer, board member, and serial hobbyist.  He can be reached on LinkedIn and Twitter @chrishoulder.

Ahsan Mir is CEO of Rapticore, a cybersecurity startup. Ahsan has extensive experience in security operations, incident management and leadership. He enjoys reading, trail running, climbing and feeding birds. He can be reached on LinkedIn.