How do you document root cause analysis in a concise and effective way?

Create a "war room" and invite your SRE/DevOps team to participate. I've done it on Slack and MS Teams. Have a channel dedicated for your team's RCAs, in which you can create threads and date them. These will serve your lessons learned in the future (+ fully searchable!). Elect a "Captain" to be responsible for guiding the team into collecting and documenting significant events in the thread. Those can be logs, alarms, tickets, metrics (graphs). An example of "Captain" guidance is: "Anne, could you please take a screenshot of the graph for response status around the time the alarm was triggered?". Not only this practice generates good documentation, but also educates your team on what matters in a RCA and creates a great team bond.

Utilizing ITILv4 concepts I have used the Fishbone diagram to help highlight potential root causes with Organizations & People, Information & Technology, Vendors & Suppliers, and Value Streams & Processes. This has been helpful to thoroughly identify the appropriate root cause and any additional contributing factors.

The identification of the root cause of an issue boils down to proposing and testing hypotheses. During the collection and analysis of the data, as the SRE/DevOps team walks the floor, each member is already creating a mental map of all possible causes based on the evidence collected. These hypotheses should be listed in the RCA thread and team members now set out to find evidence that further confirms or debunks them. Sometimes, the evidence will not be deterministic such as in identifying the entities responsible for the error, but matching timestamps with a reference to a functional path in the application can provide a sensible approximation that justifies attempting a fix. Generally, applying the scientific method generates rich RCAs.

There is usually a lot of focus on the immediate solution applied. There is always an inherent risk of this solution generating a dependency on a manual process, or on an operator, or not being applied to the whole category of assets that could eventually surface the same issue. In my experience, the solution applied during a critical event very rarely is a reproducible solution, therefore the need to shift the focus from the solution to the action that will mitigate the reoccurrence. E.g.: a failure largely reported by users can have a functional fix applied. A team focused on mitigation will also create an alarm that identifies the error (or its precursors) and engages the team before the blast radius expands.

Document root cause analysis concisely by outlining the problem, identifying potential causes, analyzing each cause's impact, and determining the primary root cause. Use structured formats like fishbone diagrams or 5 Whys. In renewable energy, pinpoint turbine failures by analyzing maintenance logs and weather data. For grid compliance, trace voltage fluctuations to equipment malfunctions. In manufacturing, track defects back to process deviations. For battery storage, investigate failures through performance logs and maintenance records. Clear documentation ensures effective problem-solving and prevention.