Too little and too late: A commentary on accident and incident reporting systems

Quite an older (1991) book chapter from James Reason, where he explores different facets of organisational accidents and limitations/caveats of incident reporting systems.

Consider the age of the paper before commenting on the rather outdated language, which I’ve mostly retained in my summary.

He argues that while reporting systems can provide important information, as sources of intel and information on real operating conditions they are “insufficient to support effective safety management. The information they provide is both too little and too late for this longer-term purpose” (p9).

Rather, organisations need to focus more on revealing and monitoring an organisation’s “vital signs”.

There’s probably nothing too new or surprising in this discussion paper, but Reason has such a poetic way of stating things.

He starts with a background in organisational accidents (those emerging from and within complex sociotechnical systems). Investigations are typically limited due to having an overly technical nature combined with other “strict stop rules”.

Stop rules are rarely made explicit but result in premature conclusion of the investigation when it has, ostensibly, “[described] the events just prior to the accident, identifying those people or equipment items whose failures were directly responsible for the system breakdown and specifying a list of local repairs that, had they been in place, would have thwarted the accident sequence” (p10).

Organisational accidents may have some common characteristics, including:

·????????The likelihood of an accident relates in part to the number of latent conditions, states or hazards within the system. These can then encounter the necessary local triggers to complete the latent accident sequence. He argues that “this view demands quite a different calculus than that employed in conventional probabilistic risk assessment” (p12)

·????????Expectedly, more opaque and complex systems will likely have more latent factors and more expected and unexpected interactions, emergent behaviours and failure modes

·????????Simpler, less well-defended systems need fewer conditions to bring about an accident

·????????“The higher a person's position within the decision-making structure of the organisation, the greater is his or her potential for spawning [latent factors]” (p12)

Reason asserts that safety “has two faces”. One is the “harsh face revealed by accidents, incidents and near misses” (p15) and reflects an organisation’s periodic vulnerability to a multitude of factors and conditions.

The other face is a “positive but largely concealed face of safety”, relating to an organisation’s potential for resisting future adverse conjunctions of hazard and failure. This resistance to failure may have two components (p15):

A)?????The scope and effectiveness of the system’s defences

B)?????It’s basic “safety health”, derived from the quality of its functions and activities

He frankly states that while “Some organisations assert that all accidents are avoidable”, they appear so largely only in hindsight. Further, it’s “hard to resist the ‘if only’ game” when reading accident reports since the investigators knew the outcomes and could readily piece together disjunctive signals into a coherent accident narrative.

However, for people involved in the accident, they “almost certainly did not realise that they were actors in a much larger accident scenario” and they “were probably doing the things they had often done before” (p15), successfully and without overt signs of impending failure.

Further, accidents are unwanted and unexpected events and we “cannot simply will their non-occurrence” (p15). That is, while it’s tempting to say safety is simple and we need to try harder, be more vigilant, or utilise more “intelligent unease, care and prudent precautions” (p15) to prevent accidents – this is misplaced.

He further argues that “Murphy [‘s law], Sod [‘s law], continuing hazards, ineradicable human fallibility and unhappy chance also have large parts to play. In short, accidents are caused by both deterministic and stochastic factors” (p15).

Organisation’s can be seen to operate in a safety space (not unlike Rasmussen’s dynamic safety model), where one end is relatively safer and the other is unsafe. Their position changes based on deliberate (non-stochastic) safety measures and stochastic factors outside of their control.

The implications are that “while intrinsically safe organisations can, to a large extent, "make their own luck", they are not immune to accidents” (p18) and the same argument works in reverse “even the undeserving can have runs of good luck” (p18).

Thus, there’s no such thing as absolute safety or absolute unsafety. Further, while this view “predicts a positive relationship between accidents and degrees of unsafety, the correlation coefficient is likely to be much less than unity” (p18). That is, while it’s intuitive to think that organisations that put more focus into preventing accidents will have fewer accidents, it’s not likely to be a stable or proportionate ratio and luck plays a large role, e.g. the undeserving can have a good hand.

Safety Information System Channels

This section looks at how organisations sample their relative position in the safety space via “vital signs”, or proactive safety state indicators.

1.??????Accident and incident reporting systems

This is said to be the “most widely used [information] channel, yet it is the least useful for effective safety management” (p20).

The info these systems provide is “both too "noisy" … and too late in the causal sequence” (p20). This doesn’t mean they don’t have value, however.

In any case, individual accidents tend to lead to local fixes and solutions, e.g. changes to procedures, engineering retrofixes, retraining etc. Over time they can lead to improvements, particularly locally.

Many highlight the need to systematically analyse incident data for recurrent patterns. However, more commonly, “the most conspicuous result of such analyses is the remarkable absence of significant patterns or trends” (p20).

Moreover, he argues, and I wholly agree, that the main use of incident data is often but inadvertently “to produce league tables of unsafety by which to identify "the good, the bad and the ugly" among their various activities or operating companies” (p20).

2.??????Unsafe act and near miss auditing

Reason argues that on the face of it, this channel of info “should be more valuable than that provided by [incident reporting]” (p20). This would also align with the various iceberg concepts of accident causation.

Nevertheless, he argues that audits and observations of unsafe acts and the like involve “serious measurement problems. The actual numbers of unsafe acts committed are almost impossible to determine. What is certain is that these numbers are extremely large” (p21).

He argues of more value than raw counts of acts/behaviour is the context surrounding behaviour. This is because behaviour “They are like the dragon's teeth in the fairy tale: eliminate one and another will appear in its place. Errors and violations are part of the human condition. They can be moderated and guarded against, but they can never be eliminated altogether” (p21).

Note that Reason isn’t critical of behavioural approaches per se, but rather the “league table” approach of numbers and the like. He does, however, indirectly support better human factors & ergonomics approaches to improving the environment and context rather than trying to fix pesky people – covered next.

3.??????The precursors of unsafe acts

This info channel deals more with the environment and psychological precursors of behaviour, decisions, actions etc – like poor workplace design, high workload, unsocial hours, inadequate training, hazard perception.

These are “one stage upstream from unsafe acts in the accident causation model and hence far fewer in number, but they … remain only expressions of higher level failure types” (p21)

While this intel theoretically would be of great use to supervisors and line management, in practice they’re said to be difficult to collect and process proactively but at the same time they’re easy to identify after accidents with the benefit of hindsight (and thus likely to be blamed on people for missing such “obvious” signs of failure).

These factors by themselves are rarely sufficient to cause accidents in complex and well-defended systems. Rather, their unintended consequences surface via multiple and often unforeseeable conjunctions with local triggering conditions and are subject to “vagaries of chance and hazard”.

Therefore, “they are neither particularly informative nor readily remediable” (p22).

4.??????Failure type indicators

Reason argues that this is the start of the intel level where we gain direct insights to the “vital signs” that reveal intrinsic safety health of organisations. It largely lies within the organisational domain and not within individuals.

Reason then discusses the types of factors which may be useful, including “general failure types”; I’ve skipped this section as, meh, it’s still more league table stuff with a different paint job in my view.

5.??????Stylistic or cultural indicators

Here, while the previous channel “provides regular and frequent checks upon a system's "vital organs", Channel 5 data yield the basis for longer-term assessments of the organisation's global safety style” (p23).

This includes things like top-level commitment, competence, and cognizance (he discussed these earlier in the paper but I skipped them but it’s largely factors involving boards, leaders etc.).

He notes “Since these "cultural" features are likely to change rather slowly, the sampling rate need only be at something like yearly intervals” (p23).

He also discusses a maturity curve approach.

Conclusion

The chapter is then wrapped up. Some key statements [** or my own inferences] here include (pp24-25):

·????????“The key to effective safety management lies in appreciating what is controllable and what is not”

·????????Organisations try to direct control over incidents, injuries etc. as if they were some directly controllable facets of performance like manufacturing widgets

·????????Instead of setting effective production and safety performance goals (which they control), they set (un)safety goals as reductions in accidents (which they don’t directly control)

·????????The feedback channels in production tend to be “relatively noisefree” and most of the factors which influence the quantity or quality of products “lie within the sphere of influence of the organisational managers”

·????????This isn’t the case for accidents etc. These “result from a complex interaction between two distinct sets of causal factors: (a) latent failures … and (b) local triggering events” and a healthy dose of chance

·????????Thus, “the feedback provided by conventional incident and accident reporting constitutes a very "noisy" channel …having only a tenuous connection to the efficacy of the safety measures employed by the organisation”

·????????A large stochastic element is involved and therefore, “"healthy" organisations can still have bad accidents, while many "sick" organisations may escape retribution for long periods of time”

·????????“Accidents, by their nature, are not directly controllable by an organisation” and “So much of their causal variance lies outside the organisation's sphere of influence”

·????????“Rather than striving to exercise direct control over the incidence of LTIs and accidents, organisational managers should seek to govern those internal factors relating to design, hardware, training, maintenance, procedures, goal conflicts and the like”

·????????And, those above factors are “the manageable factors determining an organisation's general "safety health”.

Link in comments.

Author: Reason, J. (1991). Too little and too late: A commentary on accident and incident reporting systems. In Near miss reporting as a safety tool (pp. 9-26). Butterworth-Heinemann.