Dr. Jeff Sheldon discusses various topics from a LSE public policy analysis course: confounding variables

Introduction

Between 31 January and 16 April 2024 I took a ten week, 80+ hour graded course from the London School of Economics and Political science (LSE) in public policy analysis. In this series, presented in the order of the ten LSE course modules, I offer up selections of my writings from across the course that include topics in economics, institutions, policy, statistics, policy evaluation, and politics, etc. Most are an answer to a prompt or question (provided if applicable), and vary between 250 and 800 words in length, short but hopefully insightful. You might not agree or agree only in part with my answers or assessments so feel free to engage me in dialogue, it will be most welcome. Enjoy!

Identifying confounding variables

Here I consider a real-world example of a multivariate policy issue provided by LSE, questions and answers follow. In 2017, coffee fanatics were delighted to read new research on the health benefits of coffee published by a large group of academics from Imperial College London (Gunter et al., 2017). The results of the study are summarised below:

• Major finding: Drinking three cups of coffee a day is associated with a lower risk of death from any cause, especially circulatory and digestive diseases. The research also found an increased risk in women who drink coffee developing ovarian cancer.?

• Sample group: The sample group consisted of 521,330 participants from 10 European countries. Individuals who had experienced strokes, and individuals with diabetes, heart disease, or cancer, were excluded from the study.

• Methodology: At the beginning of the 16-year research project, participants were asked how much coffee they intended to drink each day, and the researchers then analysed the deaths over that period.

(Gunter et al., 2017)

Question 1: What are the possible confounding variables that might have influenced the results of this study?

Answer 1: Before I address confounding variables I have a number of questions that could otherwise lead to confounds and errors. Note, I did not look at the ancillary materials; my impressions/questions are based on my experience as a researcher so are prima fascia for both aspects of this discussion. First, assuming there was a research protocol, did every researcher of this “large group” follow it as designed, were country context and coffee drinking culture taken into account, and how was fidelity assured and corrected? Second, were all the original researchers involved over the course of the sixteen year longitudinal study, if not, how were new researchers recruited and trained? Related, was inter-rater reliability measured if researchers were using the same research tools (e.g., interview or observation protocols, surveys, etc.)? I ask these questions because research design and implementation can add unnecessary confounds into a study, methods notwithstanding (those are addressed below). That said, Gunter et al. (2017) note that drinking three cups of coffee a day is associated with a lower risk of death from any cause, especially circulatory and digestive diseases. Great, as a coffee drinker I applaud the findings, but am skeptical. This then begs questions related to confounding variables. First, how were other known risk-lowering strategies controlled for? How did researchers hold study participants to three cups of coffee per day if they were used to drinking more? If they were only a one cup per day coffee drinker how did they entice them to drink more than they might otherwise? Looking at the coffee itself, as is known, different varietals of coffee bean, roasting methods, and brewing techniques can adjust natural caffeine levels. How did the researchers ensure that each cup of coffee consumed by participants over the study period contained the exact same dosage of caffeine in every cup consumed? Likewise, how did the researchers ensure that the size of each cup of coffee consumed was exactly the same for every participant across all years of the study??????????

Question 2: What are possible methodological issues in this study?

Answer 2: As noted above, this study had a number of likely methodological issues especially with study implementation, fidelity to research protocols, and researcher replacement over the sixteen years of the study. Be that as it may, based on the information provided, participants, according to Gunter et al. (2017) were asked about the quantity of coffee they intended to drink each day over the sixteen years of the study. The first methodological issue has to do with measurement, in that intentions and actual behaviours might not hold constant over a sixteen year period. If the researchers measured intent - subject to social desirability bias inherent in asking people questions about their behaviours, intended or otherwise – without repeatedly measuring actual consumption then as an independent variable it would be challenging to determine whether deaths were attributable to the extent to which coffee was consumed. This begs the question whether death is the best dependent variable in that decreases in circulatory and digestive diseases might be better at determining the effect of coffee consumption. Likewise, using only two variables without measuring other co-variates leaves out a host of other potential causes of death (or life, as the case may be). Last, Gunter et al. (2017) indicate there were 521,330 participants (a very robust sample size that would offer a great deal of analytical power) from 10 European countries over the sixteen years of the study. Questions abound as to whether researchers spoke the languages in which participants were fluent, whether study materials including informed consent were written in multiple languages, and how researchers dealt with language comprehension issues. Further, how was the sampling frame built, which sampling strategy was used (e.g., randomized stratified?), how were participants recruited, and whether other exclusionary criteria beyond potential participants who experienced strokes, diabetes, heart disease, or cancer that might end up as confounding variables were considered. Additionally, in practical terms, keeping any participant in the study over sixteen years is unlikely. How did the researchers address replacing drop-outs? How did they motivate participants to continue on with the study??

Thanks for reading. Please feel to ask questions or offer comments below. Next up: causal inference.