Assessing Technology Claims: What Do You Know for Sure?

“It’s not what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” – commonly attributed to Mark Twain

This maxim is very true in my experience – especially when dealing with technology.? We are inundated with plausible claims from colleagues, vendors, journal articles, press releases, YouTube, etc.?

Decisions to develop, invest in, or even use technologies frequently need to be made before absolutely definitive evidence is available. ?How do you know claims about a technology’s performance and benefits are true and accurate??? That is, how much weight should be given to various information sources when making decisions?

For me, an adaptation of the “hierarchy of evidence” framework (link) used in the realm of health care is very valuable.?? My proposed adaption is shown below. ?

Claims backed by studies are higher up on the pyramid, especially studies involving multiple tests, conducted by neutral parties, and directly assessing overall appeal to users.? Evidence of lower quality is not necessarily wrong, but rather one should be less surprised if it later proves significantly inaccurate or of only restricted applicability. ?If decisions must be made based on such evidence, the decisions should be periodically revisited in light of new, better-quality data.?? Moreover, efforts should be made to find or develop that data.

“Data” with regards to evidence quality means measurements from actual experiments.?? Modeling is not a substitute unless the models are based on well-studied analogs with extensive, peer-reviewed test data.? This is because for any novel complex system there are numerous technical, commercial, and market issues which are only fully appreciated via real-world testing – i.e., “the proof is in the pudding”.?

Lastly, one must guard against the “repetition psychological bias”?(link) from low quality claims.?? The repetition bias is that the perceived truth of a statement tends to increase merely by hearing it more often.? ?

Claims from low quality studies are almost always repeated and reported more than that from high quality studies.? ?This is schematically illustrated as the width of each level in the pyramid.? The relative abundance of low quality claims versus high quality claims results from low quality evidence being easier to generate, often sounds more novel since it occurs when a topic has not yet been well studied or challenges common wisdom, and makes easier stories to tell since it is less nuanced.

How do you judge and weigh various claims about the benefits of various technologies???