Don't get fooled: The statistical knowledge you need to not get taken advantage of

We’re living in the golden age of (mis)information. New research - from health studies to opinion surveys - is constantly coming out, and increasingly a part of our daily lives in the social media era. Everyday we hear soundbites from politicians, learn about the news from new and traditional media, and see the world through our Facebook feed. 

In this environment, it is critical to be informed - and maybe even a bit cynical - when evaluating the things we hear and the news we see. Not all research is of the same quality, not all “facts” are empirically sound, and not all reporting is honest. Unfortunately, many of us lack the necessary knowledge about statistics and research design to critically assess the information presented to us.

In this article I’ve put together some accessible hints to keep in mind when deciding whether or not the information that you’re hearing is reliable.

“Average” doesn’t always mean average

Ok, flash back to grade school: we’re going to talk about the difference between the mean, median, and the mode. Bear with me, I promise I’ll explain why this is important. 

The mean is what we commonly call the “average,” the median is the value of the observation in the “middle” of the data, and the mode is the most frequent value of a variable. For example, if we have five numbers - 2, 2, 5, 12, 29 - the mean would be 10 ((2 + 2 + 5 + 12 + 29) / 5), the median would be 5 (since there are two observations above and two below), and the mode would be 2 (since 2 occurs most frequently in the data set). 

While the mean is what we’re most familiar with, it’s often not the best way to describe the data. There are even cases where it can be highly misleading, and the median would be a better choice. For example, mean household income in the US was $88,765 annually in 2014, but the median was only $55,657. This is quite a big difference! The reason is that there are many more people who make very little compared to the few that make a lot. In more equal societies, these figures would be closer. 

It’s easy to see how these different ways to say “average” can be chosen to fit a narrative: you’d use the mean to talk about how great the economy is doing, the median to show that it’s not actually doing that hot, or even the difference between them to highlight inequality.

Percentages and percentage points are NOT the same thing

Basically, a percentage is a number representing a fraction of 100, while a percentage point is the unit for the difference between percentages. To illustrate, if 20% of people smoked in 2015, and 15% smoke in 2019, smoking decreased both by 5 percentage points (20 - 15) and 25 percent ((20 - 15) / 20). This is a critical distinction, because it is an incredibly common way to spin facts. For example, when George W. Bush proposed partially privatizing Social Security in 2004, some commentators said that only "2 percent" of the average American's Social Security taxes would be funneled into private accounts. This was a misleading statement, since the average person's income taxes directed toward Social Security would drop from 6.2 to 4.2 percent, which IS 2 percentage points - but more accurately, 32 percent.

Correlation does not imply causation

I know we’ve all heard this a million times, but it bears repeating: just because two things are correlated does not mean one caused the other. I feel like nothing I can say on this topic will illustrate the point more beautifully than this website: Spurious Correlations. While some are clearly silly, like the fact that the number of people who drowned by falling into a pool correlates beautifully with the number of Nicholas Cage films in a given year, some may look causal at first glance. For example, if a study says fast walkers live longer, don’t take that on face value to mean that your brisk gait will ensure your longevity. This erroneous assumption is often caused by the next point:

 Beware of confounders

Say we wanted to study the effect of alcohol on mortality. We could get together a group of heavy drinkers and a group of sober folks, and compare which group dies younger, on average. If we find that the group of heavy drinkers die earlier, we may be inclined to say that alcohol consumption leads to premature death. But this would be misleading because of the impact of something called “confounding variables.” In our example, heavier alcohol consumption may be associated with a higher chance of being a smoker, lifestyle factors, or even gender. What if one of these were the true cause of the lower life expectancy? By not accounting for these confounders, we would incorrectly say that it was alcohol that was lowering life expectancy instead of something completely different. In the example above about walking speed and life expectancy, it’s likely that those who are already healthy walk faster, for instance. 

There are lots of methods that can be used to “control” for confounders. Researchers will typically tell you what they controlled for and how - if you can think of other things that might have an effect on the outcome but weren’t accounted for, it’s best to interpret results with caution. 

Samples matter

Because it’s impossible to survey the political opinions of every single person in a country, or test a new medical practice on every person who might be affected, we need to use samples. It’s important that samples are well-constructed, otherwise they can damage or invalidate findings. There are lots of things to consider, but here are a few main ones to keep in mind:

Is this sample representative? Say there’s a political poll, and the respondents are those that fill out a survey on the Fox News website. These findings cannot be assumed to be representative of the US electorate at large, because those that (1) watch Fox news and (2) are motivated enough to go to their website to fill out a survey, will likely be fundamentally different from the average voter. This poll can ONLY be interpreted to mean that those in this particular sample feel a certain way. Similarly, if a survey is done only by landline, we can assume that the results are different than they would be if those on mobile phones were polled, since few people these days have landlines, and those that do likely share similar characteristics. On a more micro-level, if I ask 10 of my friends their opinion on a new tax, and four are opposed, I cannot assume that 40% of Americans also oppose the tax. Ignoring the fact that this sample is too small, my friends almost certainly differ in systematic ways from the American population at large. 

Is this sample large enough? Speaking of small samples, this is important for generalizability. While something interesting may be learned from studying a small group, below a certain threshold results can’t be inferred to the population at large. This threshold varies, but it is important to keep this in mind when evaluating research on a small number of people. 

Are claims being made “beyond the data?” If a study is done on the effect of some treatment on a specific subset of the total population, the findings are not generalizable beyond those that are similar to that subset. For example, if a study of people between the ages of 50-60 says that drinking red wine is correlated with a reduced risk of heart disease, this finding is not necessarily applicable to those that are 15 or 90 years old. Similarly, while interesting results may be found from animal trials, it is important to keep this in mind before assuming a similar response in humans. 

Surveys are not infallible

If the year 2016 taught us anything, it’s to be cautious with surveys. Not only are they easy to misinterpret, they are also often poorly constructed. In terms of construction, there’s so much involved to be aware of. Some key things to keep in mind are:

Were the right people - and enough - surveyed? Remember our discussion on sampling. If I only ask 20 people at a shopping mall in Boise, ID, I can’t assume I’ve reached a representative sample of US adults. For context, Gallup typically surveys between 1,000 - 1,500 people.

How was the survey itself constructed? The way survey questions are worded, ordered, and asked can all impact the results. A survey could ask a question in a way that essentially prompts a certain response, or put a question deliberately after a series of questions designed to make the respondent feel primed for a particular answer. The way the survey-taker speaks, the method of contact, and whether the respondent feels comfortable answering personal questions can all have an impact. There’s also the possibility that respondents simply lie. For example, when asked questions that are more socially uncomfortable, such as if they hold racist views or if they voted in the last election, they may say whatever they think the survey-taker wants to hear (called “social desirability bias”). They may also inflate their income, etc. 

When interpreting surveys, the margin of error is the key figure to keep in mind. Basically, it’s a range of how “off” a survey result is expected to be. A large margin or error represents less certainty about the results. For example, a survey saying that Candidate A is expected to receive 51% of the vote with a margin of error of +/- 3 given a 95% confidence interval essentially means that 95 times out of 100, we expect candidate A to receive between 48% and 54% of the vote. Importantly, this means that 5 times out of 100, we expect a result that is even more extreme than this range.

Graphics can mislead you

Graphics are a key way to spin data. There are a ton of ways this can be done. This article and this infographic summarize a few key things to look out for (they are really worth a look). Here are just a few examples of how messed up it can get, and how easy it can be to miss:

Flipped y-axis: This one literally blew my mind when I first saw it. It looks like there was a decline in gun deaths in 2005, right? But wait - that y-axis is upside down. It starts at 800, and goes up to 0. There was actually an increase in 2005.

Truncated y-axis: This one is similar, though not quite as egregious. In this case, the y-axis starts far above 0, making a small difference between the two bar graphs look much larger than it would be in context with an accurate graph.

Totals over 100: Often pie charts are used to compare the differences between groups (which really should be a bar chart), rather than parts of a whole. In the example below, the total is clearly far more than 100%. 

Be wary of “proof”

Proof is a very strong word in statistics. If someone tells you a study “proved” something, they probably don’t know what they’re talking about.

Be skeptical about the source

There are so, so many ways to spin data in a misleading way. Paying attention to what the motivations of the source are, whether they be financial, political, or otherwise, is often a good way to establish whether the information may be biased. It’s important to consider whether the study has been peer-reviewed, published in respected journals in the field, and to consider who may have financed it. 

If you’re reading an article about a study, take the time to read the study itself before accepting it as fact. It could be that there are problems with the study, or even that the study’s findings were misinterpreted by the news source - this happens all the time.

-----

This article has barely scratched the surface of all the knowledge there is on this topic. But I hope it’s able to provide some tools that can help make finding the truth a little bit easier.