Lesson 4 From My Students: Size Matters – Getting Sample Size Right

The Scenario

A student’s dissertation I recently reviewed aimed to predict a rare medical disease using data from patient records. Unfortunately, the student had not done an upfront power analysis to determine the sample size needed. Given the infrequent occurrence of the event, a large sample was necessary to detect meaningful effects, but collecting such a large sample wasn’t feasible.

The analysis, predictably, had low statistical power, and no significant effects were found. This left the study inconclusive. Was there truly no relationship, or was the sample simply too small to detect one?

The Issue

The student’s scenario highlights the risks of low power: the inability to distinguish between a lack of evidence and a lack of effect. This principle is captured by the phrase: “Absence of evidence is not evidence of absence.”

Sample size directly impacts the reliability of your results. In quantitative research, inadequate sample size results in low statistical power, increasing the likelihood of failing to detect meaningful effects. Conversely, an overly large sample size can result in statistically significant findings for trivial differences, leading to overinterpretation. From an ethical perspective, oversampling can waste participants’ time and resources, which is equally problematic.

In qualitative research, the concern shifts to achieving data saturation, which requires thoughtful analysis rather than arbitrary numbers.

The Solution

Addressing these challenges requires both rigorous planning and a commitment to transparency:

1. Conduct a Power Analysis Upfront: In quantitative research, a power analysis should determine the minimum sample size needed to detect an effect of a given size with a chosen level of confidence (e.g., 0.80 power). This step ensures your study design is robust and efficient.
2. Acknowledge Limitations Transparently: If low power is unavoidable, as in the student’s case, address it openly. As I advised the student, the discussion should emphasise, “Absence of evidence is not evidence of absence.” Highlight descriptive statistics to provide context for the findings and acknowledge that the study’s constraints may have limited its conclusions.
3. Prioritize Data Saturation in Qualitative Research: Focus on iterative sampling and analysis to achieve data saturation for qualitative studies. Avoid setting arbitrary sample sizes upfront; let the data guide your approach.
4. Balance Practicality and Ethics: Always aim for a sample size that aligns with your study’s objectives while respecting ethical considerations and avoiding unnecessary participant burden or resource waste.

The Key Takeaway

Appropriate sample size is essential for credible research. In quantitative studies, perform a power analysis upfront to avoid underpowered or overpowered results. In qualitative research, focus on achieving data saturation through iterative analysis.

Most importantly, when results are inconclusive due to limitations, embrace transparency. As in this case, descriptive statistics and a discussion grounded in “Absence of evidence is not evidence of absence” can add value and maintain the credibility of your work.