What are the best practices for documenting and communicating incident resolution?

Grouping the incidents into themes (technology/user story based) helps consolidate the findings and plan enhancements. Oftentimes, incidents are worked on in siloes and there is no understanding of their interdependencies. This leads to prioritization issues when resourcing is constrained.

Not only this, but having ways to correlate incidents, it’s a great way of capturing trends and detect future problems. Associating it with the correct Configurations Items(CI) and the dependent service(s) it supports, brings more insights and better decisions.

Having a formal process, with a formalised 'form', to complete, ensures all relevant information is captured. The 'form' should have a 'flexible' design to enable descriptive and other observed/ related information to be recorded as well. Limiting text input can be problematical when reporting issues.

1. Clear Documentation: Start by documenting the incident comprehensively. This should include initial detection, response actions, resolution steps, and post-incident analysis. Ensure clarity and conciseness, including relevant details such as timestamps, affected systems, and actions taken by each team member. 2. Standard Format: Adopt a standardized template or format for incident documentation to ensure consistency. This can include sections like incident summary, timeline of events, root cause analysis (RCA), corrective actions, and lessons learned. 3. Timeline of Events: Provide a chronological timeline of events from detection to resolution. This helps stakeholders understand the sequence of actions taken during incident response.

1. When an incident occurs, promptly identify it by recognizing any deviations from normal operations or any unexpected events that impact the system, service, or workflow 2. Have clear procedures in place for logging incidents. Determine where incidents should be logged, whether it's through an incident management platform, a dedicated email address, a ticketing system, or a shared document 3. When logging an incident, capture relevant details such as the date and time of occurrence, a brief description of the incident, the affected system or service, the severity level, and any initial observations or symptoms

Prioritization remains one of the most challenging steps in the process. Too often organization rely on a human assessment of Impact & Urgency to derive an overall Priority. While this is great in scenarios where nuance and judgement are key, most of the time a more objective approach is valued. I've found that considering three factors can simplify the conversation: - Criticality of system(s) involved - Nature of customer(s) involved - Recent volume of incidents for either above factor By understanding which systems and are relatively important - and by paying attention to recent activity - we can simplify the prioritization process.

Evaluate the impact and severity of the incident to determine its priority level and the appropriate response. Consider factors such as the number of users affected, the criticality of the affected system or service, and the potential business impact.

Yes, every service, application or product should have its associated criticality and overall impact on the business. If this is incorporated in the ticketing system, documentation or an App Registry containing the services' owners, teams and tech stack. Assessment and prioritisation are much more transparent. Improving response and resolution.

Would like to add that having a solid and accurate CMDB is key to a successful incident remediation, including communication of impact. The matrix for response/remediation times must be created (not just discussed) with business constituents

A key point here is where Service Level Management can help: SLAs define the expectations of the service and a good configuration map of the service management estate allows a reported incident to map back to a CI or service and urgency can be automatically determined based upon SLA targets.

When an incident occurs, the first step is to promptly assign it to the appropriate team for investigation. This team should thoroughly examine the issue to determine the root cause, assess the impact, and develop a plan for resolution. Clear communication is crucial throughout the process, ensuring all stakeholders are informed of the incident's status and progress. By systematically investigating and documenting the findings, we not only resolve the current issue but also enhance our processes to prevent future occurrences, ensuring continuous improvement and operational resilience.

Assign the incident to the appropriate team or individual based on their expertise. Begin the investigation by gathering all relevant information and analyzing the issue. Identify possible causes and work on finding a resolution. Keep stakeholders updated on progress throughout the investigation.

Additionally, having a matrix of skills up-to-date ease the correct assignment of an SME. Coupling with the proper tools, monitoring for correct analysis is necessary for any investigation. The investigation should follow a template in the form of a checklist. This ensures standardisation and avoids missing steps.

In my experience, unless ITSM maturity is very high, assignment to a resolver group must be done in the context of incident manager’s coordination, as providing timely updates to business constituents is key.

Resolver groups based upon affected service can help create auto-assignment rules so that reported incidents get routed to the most appropriate team quickly. For this to happen, it's especially important that all groups share a common toolset. Customers don't want to raise a ticket in one system only to be told they should be using a different communication means; the customer should be presented with a single means of reporting the incident then behind the scenes things are routed for them, rather than relying upon them to pick the right communication channel.

Not sure if capturing/defining a root cause is part of resolving the incident. Very often I see the concept of “root cause” incorrectly used, and as a result, Problem Management process doesn’t mature.

Rather than "The assignee", it's better to consider "the resolver group" - many issues occur when a ticket is in an individual's queue then that individual becomes unavailable. Keeping the record in the group allows team visibility. Agree that root cause isn't a necessity for the incident record; the priority is to restore some level of normal operation to the affected user - RCA is a nice-to-have, but safer to leave that in the hands of those performing Problem Management and focus upon the next incident.

While it is important to determine the root cause as soon as possible while the details are still fresh, some details may require time and effort. This work should be performed using problem tickets opened for various components that were involved and assigned to specialists or even vendors. In some cases, the business or application support teams need to correct any transaction or business impact has also been fixed, maybe even in connected or external systems. For example, the incident might have caused an order to fail, but the payment might have been processed. Unless the business impacts are fixed, the incident is not over.

Fix the problem and get things back to normal as soon as possible. Keep everyone in the loop about what's happening and when they can expect things to be sorted. Check that everything's working like it should and if needed test things out to be sure.

- Root Cause Analysis (RCA): Conduct a thorough root cause analysis to identify the underlying reasons for the incident. Document the root cause, contributing factors, and recommended corrective actions to prevent future occurrences - Document Actions Taken: Clearly outline the steps taken to resolve the incident. Include any changes made to systems, configurations, or processes. This information is valuable for understanding what worked and for reproducing successful resolutions in similar situations - Knowledge Base Updates: Update the organization's knowledge base with information from the incident resolution. This helps build a repository of best practices and lessons learned that can be referenced in the future.

Knowledge articles should be reviewed for applicability to the incident and if none were found, one should be written right away. This Knowledge Centered Support (KCS) approach ensures faster restoration through knowledge over time.

1. Document detailed information about the incident, including any troubleshooting steps taken, communications with stakeholders, resolution actions, and post-incident analysis. Maintain a chronological record of events to facilitate post-incident reviews and audits. 2. Once the incident is resolved, document the resolution details, including the root cause analysis, corrective actions taken, and preventive measures implemented to avoid recurrence. Conduct a post-incident review to identify lessons learned and areas for improvement in incident management processes.

Write down everything that happened, including what you did to fix it and what we can learn from it. Make sure you are using the same every time. Tell everyone involved what happened, what we did, what we are going to do to avoid such incidents in future. If possible make separate tools like wiki/knowledge sharing/sharepoint which will make all this information easily available and findable.

The owner of the Incident Management process should be the individual most concerned with process efficiency... but that doesn't mean those involved with the practice can't highlight process improvements.

Incident Management is reactive in nature, the “fire truck/fighters” who put the fire out! Details captured in the remediation should be used to drive the root cause analysis or the prevention of such incident to occur through either a permanent change or a documented workaround.

Where possible, start automating your incident data collection. With logging the information early upfront, with the impacted configuration items, events, triggers, technical owners and business stakeholders, it is possible to identify possible root causes and raise awareness earlier. The information can be collated into templated PIR forms, reducing manual data entry effort and improve accuracies. Automating the data collection at the start of the incident, such as using data from the CMDB and Event Management, it is possible to feed this data into the incident first time and reduce inaccuracies.

Working incidents is a waste of time, and teams need to strive to meet the SRE norm: no incident should ever be repeated. While that is a tough standard to achieve, it is important to analyze past incidents for commonalities, repeats, and patterns to code or configure the application differently and avoid them. While this is hard work, it can result in a 50% reduction in incident volume, leading to productivity improvements and more sleep at night.