Normalization of Deviation: Quotation Error in Human Factors

[This is an update and repost of an older post but now in a LinkedIn article format]

Interesting paper that's a bit different than usual. This one from Central Queensland University looks at the issue of misquoting/misrepresenting in human factors research, using Diane Vaughan's Challenger Launch Decision book and Normalization of Deviance (NOD) concept as the study control.

Of 187 extracts from 118 articles referring to Vaughan's book/concept, 24 of them (12.8%) were classed as inaccurate representations. One example cited changed the meaning of the original quote, while others incorrectly inferred certain points and details. 

The authors note that for a claim or idea (like Vaughan’s NOD), their data suggests that accuracy is better considered along a scale rather than as a discrete category, as some of the excerpts appear to be more accurate than other ones.

Interestingly, the authors used Vaughan's NOD concept to help explain from systems-perspective how misquotation in research can occur. For instance, they say that quotation errors rather than simply being things to blame on an individual may instead be “embedded in the banality of organizational life (in this case, academic life): culture, production pressures, and structural secrecy” (p10).

·        For culture of production, they suggest that the well-known axiom “publish or perish” pressures academics to publish. They say that while the production culture is designed to reinforce positive aspects of productivity, it may increase negative side-effects of pressure and shortcuts in the writing process.

·        For production of culture, they argue that the drive to quote and reference sources, of which is “ingrained within the rigorous “culture” of academia” (p9), drives norms to over-referencing, where references that don’t quite support the author’s arguments are included as supporting evidence.

·        For structural secrecy, they say three main elements contribute:

o   The growing and multi-disciplinary field of human factors under a “culture of production” makes it difficult for authors and reviewers to fully understand all of the material they cite.

o   Referencing conventions against quoting large slabs of text means authors need to reword statements, and in the process possibly misinterpreting complex themes into brief and neat dot points or phrases. They say that because “citation tends to be self-perpetuating (Steel, 1996), these distortions become normalized and firmly embedded in the literature” (p10).

o   Restricted institutional budgets for journal access prompts authors to rely more on secondary sources etc.

The authors say that quotation errors are, to some degree, a silent issue where the immediate effects, such as loss of credibility, are directed almost entirely at the author.

Instead, they further argue that “To misquote Vaughan slightly, we suggest that it is the normal everyday systems that promote scholarship that also allow mistake, mishap, and quotation error to occur” (p10).

Hopefully I haven’t misquoted anything here as that would be awkward.

Link in comments.

Authors: Jordan Lock and Chris Bearman, 2018, Human Factors.