Anecdotes are not Evidence

When we hear I know a CEO that uses this method and she's happy with the outcomes, has several core fallacies wrapped into one.

The first is the self-selection problem of statistics. This is the Standish problem. Send out a survey, tally the results from those that were returned. Don't publish how many surveys went out and how many came back. Or most critically, what the population of potential respondents was for the surveys and if the numbers of responses represent a proper statistical sample to assure high enough confidence on the returned surveys to draw any credible conclusions.

An anecdote is a sample of one from an unknown population

Next is the Anecdotal sample. I know a guy that... in support of the suggestion that by knowing someone that supports your conjecture, your conjecture is somehow supported.

These are both members of the cherry-picking process. The result is lots of exchanges of questions to the original conjecture that have not basis in evidence for the conjecture.

When you encounter such a conjecture, apply Carl Sagan's BS detection kit

• Seek independent confirmation of alleged facts.
• Encourage an open debate about the issue and the available evidence.
• In our domain and most other domains, there are no authorities. At most, there are experts.
• Come up with a variety of competing hypotheses explaining a given outcome. Considering many different explanations will lower the risk of confirmation bias.
• Quantify whenever possible, allowing for easier comparisons between hypotheses' relative explanatory power.
• Every step in an argument must be logically sound; a single weak link can doom the entire chain.
• When the evidence is inconclusive, use Occam's Razor to discriminate between hypotheses.
• Pay attention to falsifiability. Science does not concern itself with unfalsifiable propositions.

When there is push back from hard questions, you'll know those making the claims have no evidence and are essentially BS'ing their constituents.

This way when you hear an ancedote that sounds like it conflicts with a principle, you can test that ancedote with Sagan's BS detector.

My favorites of course are

• We can make decisions while spending other people's money without estimating the outcomes of our efforts - the #NoEstimates idea
• We can work on non-de minimis projects while spending other people's in non-trivial domains, with any governance, any contracts, and not to exceed the budget target, mandatory capabilities to accomplish the mission or fulfill the business case. Just spend till told to stop - the #NoProjects idea
• SAFe is not agile, with no domain or context in which to assess that statement.

Anecdotes versus Numbers (Statistics)

In the world of project management and the process improvement efforts needed to increase the Probability of Project Success anecdotes appear to prevail when it comes to suggesting alternatives to observed dysfunction. 

If we were to pile all the statistics for all the data for the effectiveness or not the effectiveness of all the process improvement methods on top of each other they would lack the persuasive power of a single anecdote in most software development domains outside of Software Intensive Systems. 

Why? because most people working in small groups, agile, development projects, compared to Enterprise, Mission Critical can't fail, that must show up on time, on budget, with not just the minimum viable products, the mandatorily viable capability - rely on anecdotes to communicate their messages.

This is evidenced now all the way to the Project Management Institute's Disciplined Agile

I say this not from just personal experience, but from research for government agencies and commercial enterprise firms tasked with Root Cause Analysis, the conference proceedings from this research (some of which I've participated in), refereed journal papers, and guidance from those tasked with the corrective actions of major program failures.

And if you're not using the Project Paradigm, but are just spending away until told to stop or run out of money, you're free to do anything you want. But in a business where there are fiduciary controls, ALL work is a project in some way, recorded on the balance sheet. You may not call it that, but the Finance and Accounting standard in the US (FASB-86) and the UK Accounting standards record costs to produce value for the firm as a project. Otherwise, it's just a level of effort labor cost and recorded as a sunk cost. I really wish Computer Science schools forced students to take a business finance and accounting class.

Anecdotes appeal to emotion. Statistics, numbers, verifiable facts appeal to reason. It's not a fair fight. Emotion always wins without acknowledging that emotion is seriously flawed when making decisions.

Anecdotal evidence is evidence where small numbers of anecdotes are presented. There is a large chance - statistically - this evidence is unreliable due to cherry-picking or self-selection (this is the core issue with the Standish Reports or anyone claiming anything without proper statistical sampling processes). 

Like said above Anecdotal evidence is considered dubious support of any generalized claim. Anecdotal evidence is no more than a type description (i.e., short narrative), and is often confused in discussions with its weight, or other considerations, as to the purpose(s) for which it is used.

We've all heard stories, ? of all IT projects fail. Waterfall is evil, hell even estimates are evil stop doing them cold turkey. They prove the point the speaker is making right? Actually, they don't. I just used an anecdote to prove a point. 

If I said The Garfunkel Institute just released a study showing 68% of all software development projects did not succeed because of a requirements gathering process failed to define what capabilities were needed when done, I have made a fact base point. And you'd become bored reading the 86 pages of statistical analysis and correlation charts between all the causal factors contributing to the success or failure of the sample space of projects. See you are bored.

Instead, if I said every project I've worked on went over budget and was behind schedule because we were very poor at making estimates. That'd be more appealing to your emotions since it is a message you can relate to personally - having likely experienced many of the same failures.

The purveyors of anecdotal evidence to support a position make use of a common approach. Willfully ignoring a fact-based methodology through a simple tactic...

We all know what Mark Twain said about lies, dammed lies, and statistics

People can certainly lie with statistics, done all the time. Start with How to Lie With Statistics But those types of Lies are nothing compared to the ability to script personal anecdotes to support a message. From I never seen that work, to what now you're telling me - the person that actually invented this earth-shattering way of writing software - that it doesn't work outside my personal sphere of experience?

An anecdote is a statistic with a sample size of one. OK, maybe a sample size of a small group of your closest friends and fellow travelers. 

We fall for this all the time. It's easier to accept an anecdote describing a problem and possible solution from someone we have shared experiences with, than to investigate the literature, do the math, even do the homework needed to determine the principles, practices, and processes needed for corrective action.

Don’t fall for manipulative opinion-shapers who use story-telling as a substitute for facts. When we're trying to persuade, use facts, use actual example based on those facts. Use data that can be tested outside personal anecdotes used to support an unsubstantiated claim without suggesting both the rot cause and the testable corrective actions.