Don't hate the player, hate the game
Boyd Baumgartner
Latent Fingerprint Examiner @ King County, WA | AI Software Development Enthusiast
‘Don’t hate the player, hate the game’ - Ice T
I’ve been in the fingerprint industry since 2004, the same year that the FBI made a fingerprint error that shook the forensic science community. After a predictably long investigation that somehow found bureaucratic ineptitude to be at the root of, and the solution to the issue, the foundation was poured. These findings served as a catalyst for the 2009 National Academies of Science Report on forensic science which uttered the clarion call: Change!? Little did I know back then, the call for change was a double entendre.? It was less for cultural, methodological or ethical change, and more for literal change, in the form of grant money.
You see, panhandling isn’t confined to the roadways of America, it exists in the academic, and non-profit sectors as well. It just takes the form of grants, charitable giving and endowments.? The American Association for the Advancement of Science has its own marketing team and you can even transfer your stocks and roll over your IRAs to them! They put out a position paper in 2017 on latent fingerprints.
The first two recommendations in the position paper begin with ‘Resources should be devoted to further research...’? A non-profit whose position is that it should be given more money to ‘advance science’? Who could’ve seen that coming?
I don’t mean to pick on the AAAS, they’re just like any other of a number of creepy relatives that come out of the woodwork whenever a family member dies or wins the lottery. There were previous papers that made the same call, and wouldn’t you know it, these same alphabet agencies and blue ribbon commissions have put themselves in the position to rule on the validity of any action that stems from their calls from on high.? Something I wrote about here: https://www.dhirubhai.net/pulse/nothing-truth-boyd-baumgartner-wxbqc/
Examples of these papers are:
PCAST Report: https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf
The recommendations of the PCAST Report include ‘The President should request and Congress should provide increased appropriations to NIST of (a) $4 million to support the evaluation activities described above and (b) $10 million to support increased research activities in forensic science...’
Human Factors in Latent Analysis:
The recommendations of the Human Factors report include some familiar language: ‘The federal government should support a research program.....’
It appears that the administration of justice is seemingly more important than the execution of justice in the form of backlog reduction.?
And this gets us to the point of this article. If grant funding is the goal, then journals are the arena and publication is the playing field.? Whenever you have a playing field, an arena and a goal, you have a game. The problem however, is that science is not a game. Enter, the Replication Crisis.
The Replication Crisis is the realization that many scientific studies cannot be replicated. Replicating empirical data is supposed to be an essential element of the scientific method, so the inability to replicate the results of a study or theory undermine its credibility.
How is this even possible?? According to “The Manifesto of Reproducible Science”, it stems from the following:
Low sample size, small effect sizes, data dredging (also known as P-hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect.?
According to a 2005 paper titled “Why Most Published Research Findings Are False”, the problem is stated in the summary.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
领英推荐
According to a 2017 paper titled “When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment” , the Replication Crisis is intimately tied to an inappropriate use of Null Hypothesis Statistical Testing.? A borrowed quote from a book referenced in that paper sums it up nicely:
“What used to be called judgment is now called prejudice and what used to be called prejudice is now called a null hypothesis. In the social sciences, particularly, it is dangerous nonsense (dressed up as the “scientific method”) and will cause much trouble before it is widely appreciated as such.” - Edwards, 1972.? Likelihood: An Account of the Statistical Concept of Likelihood and Its Application to Scientific Inference. Cambridge, UK: Cambridge University Press.
Let that quote sink in.
This XKCD cartoon memes the quote quite well.
What they don’t explicitly mention is this, the point of the article. Science is gamified. It’s how scientists contend for grant money in the first place. Gamification requires the application of game-like elements and mechanics to scientific activities, processes, and communication. This can involve using points, badges, leaderboards, challenges, and other forms of rewards or competition as incentives.?
If we look at research bios we get just that. This is the research bio of someone prominent in bias research in the forensic sciences.
How does the gamification of science compound the Replication Crisis from what we’ve already seen?
Prioritization of Quantity Over Quality: Gamified systems that reward researchers for the number of publications may incentivize rushed or less rigorous research, potentially leading to flawed findings that are difficult to replicate.
Focus on Novelty and Impact Over Reproducibility: If rewards are tied to publishing novel or high-impact findings, researchers may be less inclined to pursue replication studies, which are essential for validating existing research and ensuring its reliability.
Competition and "Publish or Perish" Pressure: Gamified leaderboards and rankings can intensify the pressure to publish, potentially leading to unethical practices like data manipulation or selective reporting to achieve desired results.
Neglect of Null Results and Negative Findings: In a gamified system that values positive and groundbreaking results, researchers may be discouraged from publishing null results or negative findings, even though these are crucial for understanding the limitations of existing knowledge.
Misrepresentation of Scientific Progress: A gamified approach that emphasizes individual achievements and competition can obscure the collaborative and incremental nature of scientific progress, where replication and verification are essential steps.
While the scope of this article is not to get into any one discipline or author, what you will find is that researchers in a specific discipline can’t even acknowledge that there are findings which contradict their original research.? Nor can they acknowledge they’ve engaged in any of the known vices or biases listed above.? That isn’t expertise, that’s politics.
Instead, a vicious cycle emerges. Institutional gamification incentivizes flawed research practices, leading scientists to produce misleading narratives that further obscure the issues within their discipline. This, in turn, perpetuates the illusion of a need for "more research," despite the fact that subsequent studies will be inherently flawed due to their reliance on faulty foundations. This flawed research then feeds back into the gamified system, boosting metrics and reputations, while failing to offer genuine solutions or clarify the problems at hand. The result is a self-perpetuating cycle where researchers become gatekeepers of validity, not through genuine expertise, but by manipulating a system designed to reward quantity over quality. This ultimately ensures a steady stream of grant funding, further entrenching the flawed system.
Co-Founder of GoEvidence Forensic Laboratories, LLC / Certified Latent Print Examiner
4 个月Are you familiar with the concept of Focus Research Organizations? Focused research organizations are special purpose nonprofit organizations with a specific mission to benefit the scientific process. These organizations are organized like startups and have a finite duration, typically five to seven years. Once the mission is completed, the organizations may continue through spin-offs or longer-lived nonprofits, but the FRO itself is not meant to be permanent. I would like to hear your thoughts. This is the YouTube link to the podcast Clearer Thinking where they talk about this subject. https://youtu.be/ALheTW4cB4w?si=voXdt6tCMyIN07u7
Latent Print and Crime Scene expert
4 个月Excellent analysis and well articulated conclusion, as always Boyd Baumgartner. "Follow the money" applies in every facet of human endeavor, I guess. And if the money is available, people will rise to the bait. Thanks for the analysis!