What's your process for prioritizing and escalating incidents?

Often it can be tempting to dig deep into investigating impact and that's sensible - you want to establish a "blast radius" as soon as possible. However, be careful not to get blinkered. It's often just as valuable to do a high level assessment and then focus on "stopping the bleeding" before doing any further triage. For example, in the event of a security incident where you've breached some credentials, it's more important to immediately cycle tokens before you overanalyse audit logs. As another example, it's more important to prevent sending out a large batch of poorly formatted emails to customers, before you look at exactly how many have been sent already.

Immediate Reporting: Report all suspected incidents, even if based on minimal information, immediately via email, phone, or confidential forms, and ensure you get a positive acknowledgment. This is applicable to all staff and company resources. Initial Impact Assessment: Evaluate the impact on IT, affected data, incident duration, and risk to data subjects. Make sure to document all findings. Incident Response Activation: If there's any indication, even minimal, of significant data loss, business disruption, or reputational risk, activate the response. The security officer will make this decision. Escalation: Consider security, safety, business, and revenue impact in your triage. Escalate to key contacts within 15-minute intervals.

Incidents (security ones or ones involving software and infrastructure) are shrouded in fog until the end of the resolution phase. Assessing an incident's impact is a way to map out the general area covered by that fog: what is not working as expected. The impact assessment is the first step towards proper severity scaling and a good indicator of the size of the response. To assess the impact, focus on the performance and product metrics: what is not happening as usual, what is not working, and what message broker is busier than normal, ... Look at the exposure to customers, operations, and legal obligations: are they not affected, partially or totally?

Prioritizing and escalating incidents begins with assessing their impact on organizational assets, operations, reputation, and legal obligations. Employing a matrix or scale helps rank impact from low to high, reflecting severity and scope of damage. A minor malware infection on a single workstation may warrant low impact, whereas a ransomware attack encrypting the entire network signifies high impact. This initial assessment guides effective allocation of resources and escalation protocols, ensuring swift and appropriate response to mitigate further damage.

Utilizing a matrix or scale to rank urgency from low to high aids in gauging the necessity for rapid response and resolution. A detected and blocked phishing email may constitute low urgency, whereas a data breach compromising sensitive customer data represents high urgency. This evaluation informs prioritization of actions, enabling timely intervention to contain and mitigate potential escalation or spread, safeguarding organizational interests and stakeholder trust.

High-priority incidents typically entail high impact and urgency, while low-priority ones involve low impact and urgency. Utilizing a color-coded or numbering system enhances clarity and aids in swift decision-making. This prioritization scheme ensures resources are allocated effectively, addressing critical issues promptly to minimize risks and uphold operational resilience.

Maintaining a public-facing status page creates a consistent and fast channel to communicate important updates to customers and external stakeholders. Aim to post customer-impacting incidents quickly and provide frequent (min every 30 minutes) updates until resolution .

Communication will keep stakeholders, users, and other teams out of the way of the Incident Response Team during an incident. That's why the Communication Officer has an essential responsibility through the Incident Response: to communicate outwardly about the incident and the response in a timely and regular manner. Communication must be done early and regularly. Any message should indicate a time estimate for the next update to avoid too much pressure and a precise time for others to check the chosen communication channel.

Documentation of an incident starts when the response begins and ends after the incident has ended. To make it easy, prepare a ready-to-use template in a shared document platform or wiki. The template should have a title, a start date, an end date, a severity, a list of participants in the response, and a list of impacted services. The documentation has multiple use cases: - help any member of the Incident Response Team to retrace what happened when they join the response or while they are working on the response - contain a list of findings, hypotheses, and actions throughout the response - during the After Action Review, serves as the basis for it An incident is an opportunity to learn; documentation is how to ensure it happens.