Media & Filtering Out All The Noise

Following a conversation at one of our Happen offices on how to judge what public information to trust and what not, I thought it might be nice to list some of the heuristics I've developed over the years. Some are my own, some are inspired by great BS busters like Nassim Taleb, Mark Baker and Ben Goldacre.

My basic starting point: scientific research is the search to disprove, ie prove you're wrong. Most bad science comes from people trying to prove they're right. Or think it's the same type of proof. Below are some of my markers to distinguish publications on value, from likely-true to pure BS. These help me pick up a funny smell from fishy news.

1. Is the author selling something? Yes? Tread carefully as this person isn't independent. Is the author from a credible background, or filling pages from whatever source they can find? Is the publication politically motivated or funded/inspired by a particular agenda? Sadly, almost always. Most blogs and free newspapers fall in this category.
2. Absence of Evidence framed as Evidence of Absence. "If I haven't seen it, it doesn't exist". For example all "proof" that genetically modified food is safe falls in this category.
3. Very small samples being presented as representative for very large populations, simply because they are extreme but sound plausible. Is there a control group to compare and stress test the claim, independently monitored? Of not, can you imagine that creating the same situation again will deliver the same outcome? Social sciences and economics often fall apart here. The incident isn't necessarily the new rule. Download any Consumer Trend Report and feast your eyes on this phenomenon.
4. Is the model in any way truthful to the reality it models? What works in a petri-dish (anti-oxidants) might not work in the real environment (human body). Are you being presented a map, a model or a fair representation of reality?
5. Randomness being mistaken for patterns, or randomness being expected to look uniformly distributed. In reality, random distributions are quite "lumpy", full of clusters. An example of this is three cancer cases in the same flat being accredited to the GSM mast on the roof.
6. Always be cautious when something new is presented that trumps something very old. We all know such transformational innovation almost never happens, and most successful new things you see simply trump something almost as new. Old things adhere to the so-called Lindy principle, which implies that everything that's been around for a long time, will likely stay around for much longer. In a thousand years, people will still read the Bible. But probably not Harry Potter. Fashion presents new dress styles, not new principles for clothing oneself.
7. In the same area: physics truths trump biological truths trump human truths trump societal truths trump personal truths. The lower down the pyramid someone makes bold statements, the bolder their proof and rigour needs to be. Building new habits are easy, introducing new ways of interacting is hard, evolving new organs extremely difficult and ignoring gravity impossible. For example making any claim on changes in human physiology, like puberty hitting at a younger age is very bold.
8. Has the author tested their theory/promise for long enough to notice an actual effect? Media are overflowing with examples of this. Foodies under 30 years old claiming their diet is healthy, unaware that at that age you can eat/smoke just about anything without too much harm. Soylent is the ultimate case, imagine living on that for 25 years. Or running 50km a week: fine for 5 years, not so fine for 25 years. Scalability is often presented on the back of singular evidence. Ie, because it's true for me, it must be true for everyone. I sold 10 widgets, so I will sell 100. Or 10,000.
9. Nature is great. But remember that Nature has more things that can kill you than can heal you. 
10. Snapshot measurements being presented as static reality rather than dynamic. A beautiful example of this is today's narrative on widening wealth gaps. True, but also a very narrow picture. A fuller reality is that at the level of individuals, people's wealth fluctuates quite a lot during their lifetime. Most of us have periods of being relatively poorer, or relatively wealthier. Virtually no one is excluded from at least some economic mobility. For example 50% of Americans will spend some of their life in the top 10% income cohort. Always ask yourself if you're not being presented a tailored selection of the whole picture.

I'm sure there are may more; I'd love to hear yours!