You're facing a major system failure incident. How can you conduct a post-mortem analysis effectively?

One of the most critical components and concerns is to “fix the underlying problem” as opposed to “fixing blame”. Spending time focusing on political fallout instead of rational problem solving undermines the integrity of issue resolution process and further fractures the team dynamic.

An effective post-mortem analysis after a major system failure involves preparing a collaborative team, documenting the incident in detail, and identifying the root cause through structured analysis. The goal is to foster a learning environment that focuses on understanding the failure and implementing measures to prevent similar issues in the future.

1. Assemble the Team: Gather all involved stakeholders, including engineers, operations, and management. 2. Define the Incident: Clearly document what happened, including timelines and the impact on users. 3. Identify Root Causes: Use techniques like the "5 Whys" or fishbone diagram to trace back to the underlying issues. 4. Analyze Response: Evaluate what was done well and where the response fell short. 5. Document Findings: Summarize key insights, corrective actions, and preventive measures. 6. Share and Implement: Distribute the report to relevant teams and ensure the recommended actions are implemented. 7. Follow-Up: Schedule a review to assess if improvements have been effective.

Start before the event. Ensure your systems are well instrumented at every level, preferably into a consolidated toolset. Make sure you know when it really started and what was happening at the time.

You'll need to go back to before the incident started and follow it through until after it was resolved. Start your focus on gathering information and data which should lead you to the root cause. Then create a plan of action to prevent it from happening again. Document all that was learned from it, then share this with those it may affect in the future.

Don't just look for what happened this time. Often, major failures are the culmination of an escalating issue. For example, the disk is almost.filling each night, but the cleanup job is preventing a failure, until it doesn't. Or a job is gradually increasing in memory consumption until it finally runs out.

Con los datos en mano, el siguiente paso fue reconstruir la secuencia de eventos. Utilicé algoritmos de machine learning para identificar correlaciones entre diferentes eventos y establecer una línea de tiempo precisa. Recuerdo una ocasión en la que una falla en la red parecía ser la causa, pero el análisis de IA reveló que fue una actualización de software no autorizada la que desencadenó el problema. Compartí esta etapa de análisis en LinkedIn, donde colegas y expertos me ofrecieron valiosas perspectivas adicionales.

Utilicé técnicas de análisis de causa raíz (RCA) asistidas por IA para profundizar en el problema. Herramientas como Root Cause Analyzer me ayudaron a identificar no solo la causa principal, sino también factores contribuyentes. Un incidente que me marcó fue cuando descubrimos que una combinación de factores menores, pasando desapercibidos, se combinaron para causar una falla mayor. Publicar un resumen de mis hallazgos en LinkedIn generó una discusión enriquecedora con otros profesionales que habían enfrentado problemas similares.

The use of AI assisted tools, or even using an independent third party facilitator, will go some way to reducing confirmation bias and perhaps identify aspects not under initially consideration.

Force yourself and your team to drill down into 3-5 "Whys". Why did the service fail? It failed because of a bad code release. Why? Etc until you get 3-5 Whys documented.

Tras analizar todos los reportes obtenidos, podemos determinar la causa raíz del problema que causó el fallo del sistema. A veces he encontrado que el error es de un equipo que está EOL, por lo cual su software no es actualizable para evitar esta falla en un futuro, por lo que nos debemos centrar en tener un plan para paliar este problema crónico de forma inmediata. En mi caso fue desconectar el equipo switch que provocaba un bucle en red. Aunque puede que a veces no sea posible estas vías, siempre debemos tener un plan de acción acorde con los puntos de falla posibles en nuestra arquitectura de sistemas. Backups, equipos secundarios de respaldo, discos para nuestras cabinas de storage, etc...

one thing that must be taken into consideration is to establish a blameless culture and then ensure that you analyze the root cause of the system failure. Once you have identified the root cause, solve it and ensure that you have documented the findings for future reference.

Con la causa raíz identificada, dise?é un plan de acción para corregir el problema y prevenir su reaparición. En un caso, implementé mejoras en el proceso de despliegue de software para incluir verificaciones automáticas adicionales. Compartí el plan en LinkedIn y recibí comentarios constructivos sobre cómo otras organizaciones habían implementado soluciones similares o mejores.

First is to ensure that system are operational to allow business to function as priority and workaround are in place. Perform the root cause to determine the cause and implement a solution Set up a team to identify the process or control that was broken down and ensure that it is a learning lessons.

The true measure of a post-mortem analysis lies in the tangible improvements it generates. Therefore, it is crucial that the action plan is not only realistic but also effectively communicated and meticulously executed. By doing so, you can drive meaningful change and prevent recurrence, thereby enhancing the overall robustness and reliability of your systems.

If you cannot be the shining example, at least be the poignant reminder. Get over yourself and own a mistake. Everyone makes them, and if you have not taken down production at least once in your career you are not doing the hard and rewarding work, or you have not been in the game very long. Mistakes happen, learn from them so you don't do the same thing again, and share your experiences with others. They need to learn from it too. It's not a failure unless you fail to learn and teach others.

Mistakes are made everywhere but the key is to learn from the mistakes and reduce the likelihood of impacts in the future. Share this information with require teams so that there are lessons learnt.

Después de implementar las soluciones, es crucial reflexionar sobre lo aprendido. Realicé sesiones de retrospectiva con mi equipo, donde discutimos qué funcionó bien y qué podría mejorarse. Utilicé plataformas como LinkedIn Learning para buscar cursos y webinars que me ayudaran a fortalecer mis conocimientos en áreas específicas. Publicar estas reflexiones en LinkedIn no solo ayudó a consolidar mi aprendizaje, sino que también ofreció una oportunidad para que otros compartieran sus experiencias.

Finalmente, documenté el incidente y el análisis post-mortem en un informe detallado, asegurándome de incluir lecciones aprendidas y recomendaciones. Subí este informe a LinkedIn, etiquetando a colegas y expertos que podrían beneficiarse de la información. Esta práctica no solo ayudó a difundir conocimiento, sino que también me permitió construir una reputación como alguien que comparte y colabora activamente.

Por experiencia propia, siempre funciona mejor una reunión tras la resolución del problema con los miembros del equipo y comentar de la forma más simple el proceso de resolución del incidente. También es aconsejable, informar a los supervisores, que aunque no sean los encargados de solucionar la falla, sean capaces de entender los requerimientos necesarios para que el problema sea abordado y entendido por todas las partes.

One key consideration is to focus on the learning aspect of a post mortem rather than the blame aspect. The whole point is to identify what control or process broke down so you can fix it moving forward and not experience the same incident twice. Sure there will always be a desire and often times a need to identify who was at fault for a variety of reasons to determine maliciousness or liability, but in the post mortem it is to help the entire team learn. So keep the meetings constructive and professional and document everything.

1.- We all make mistakes. 2.- The values of the organization influence the behavior of the employees of that organization. 3.- People achieve high performance when they receive positive encouragement and support from leaders, peers and subordinates. 4.- We learn from our events to understand why mistakes are made and to improve our systems to prevent them from happening again. 5.- Deviations from expectations are often malicious and are often caused by people trying to get the best result from a complex and variable work environment. 6.- The way leaders respond to an event is important and directly affects a team's ability to learn and improve.