Study size matters!

One of the more interesting / vexing issues in the design and implementation of a prospective observational study / registry concerns the study’s scope.? (And already, I'm choosing my words carefully to avoid the use of “sample size”, as that connotes a more traditional statistical power calculation built on a specific hypothesis that then merits procedural controls to minimize noise and maximize homogeneity).? In observational research, we embrace the noise, reflecting the heterogeneity / variability of the real world.? As I’ve often said, the real world is real important, but real messy!? But still, we want to develop meaningful findings…?

So, absent a specific hypothesis, how do we rationalize the number of patients and sites in a research initiative designed to observe / document and, ultimately, to provide meaningful (but not necessarily definitive) findings?? What are the factors that should be considered when, for example, designing a (discretionary) observational registry to document clinical, economic, humanistic (and maybe even safety) outcomes attainable in variable and diverse patient (and physician) populations???

For me, the starting point has always been to establish analytical themes (or maybe even questions that the study can answer, with the caveat that “answering a question” may suggest too controlled of a design).? After the themes, however, and the concept of casting the net broadly so as to embrace variability, things seem to get pretty subjective.? So, I asked one of the best resources out there, my former colleague (and liberated statistician), Dave Miller, of DPM Biostatistics.

Jeff: Dave, help me out here!!!

Dave: Ideally, an observational study should be scoped so that the RWD that’s being collected can be turned into RWE.? Unfortunately, I think the industry has been too casual in lumping together RWD and RWE without asking the critically important question “evidence of what”.? For example, your registry could try to fill some of the common evidence gaps that controlled clinical trials can’t address.? An RCT generally demonstrates that a very specific intervention is clinically efficacious over a relatively short duration in a population that often excludes people with common comorbidities, common concomitant medications, and people who don’t receive care at major academic medical institutions.

Keeping in mind that “evidence” doesn’t necessarily entail “proof”, we should think about what are the most important missing pieces of evidence from the RCT (e.g., breadth and duration of outcomes, effectiveness vs efficacy, the impact of comorbid disease, etc.).? My experience is that this is nearly impossible to do without picking at least one concrete example and doing some math, but the trap there is that people get confused that that one example has the same sort of primacy and importance as an RCT primary objective.? Once you’ve done just a little bit of math it becomes a shiny object that the team can’t look away from, but it doesn’t mean you shouldn’t do a little math!? What’s important is to caveat the hell out of the presentation of the math. You’re good at that Jeff: you have nearly as many parenthetical remarks in your opening statement as you have regular sentences.?

Jeff: Thanks for that, I think.? I certainly concur that the starting point is the right information gap to be filled, but I’m also struck by the thought that the stakeholders for RWE may still have different expectations for what constitutes a truly meaningful / influential finding.? And they tend to be more acceptant of the messier real-world realities.? Still, how can I argue with a little bit of math?? Can you give me an example?

Dave: Sure. In this era of precision medicine, a lot of practicing physicians may look at clinical trial results and say, “this looks pretty good, but I wonder if this treatment is right for me or right for my patients”. This can get pretty granular. You can start by asking how well a particular finding generalizes to, for example, African Americans and then move from there to hypertensive African Americans and then go further into African Americans who have poorly controlled hypertension while using a beta blocker.? The math then considers a measure of outcomes variability along with your desired level of precision to arrive at the number of patients you’ll want to study.? And then taking it one step further, by considering the prevalence of that unique subgroup, you can determine how many total participants you need to include not only that specific subgroup, but also other subgroups (with the assumption that the real world holds many subgroups that you’re likely to care about tomorrow even though they aren’t on your radar today).

The level of precision you need is also going to depend on how you want to use the data.? Is it largely an exploratory analysis for general education about a new treatment or is it a key piece of evidence needed for a potential label expansion.? Is your audience community physicians?? Patient advocacy groups?? Peer reviewers?? Payers?? Regulators?? I said before that people use the term RWE too casually without addressing the “evidence of what” question: an equally important, frequently unstated question is “evidence for whom?”? The level of rigor with the math needs to match the intended audience.

Jeff: OK, I get it.? And I certainly can’t argue with applying statistics to formally justify the size of the study.? Still, if we back the train up and examine the underlying strategic rationale — the fundamental “why” — we may find ourselves back at Subjectiveville Station, in which the goal may be as vague as “informing the medical community of the role of this new product”.? What number of patients will constitute having “informed” (if not “influenced”) prescribers?? Or have we now gone beyond math?

Dave: I believe “informed” can always be enhanced as “informed of what”. The “what” need not be comprehensive, nor does it need to be whatever is most important, but there should always be something that you can imagine as the title of a peer-reviewed article, and you should have a study sized to provide some amount of evidence that supports that title.

You’re probably saying “some amount of evidence” still sounds like Subjectiveville Station and you’re not wrong, but there are degrees of subjectivity.? Making a subjective call about the amount of evidence that’s needed is very different from making a subjective call about how big a study should be with no idea at all of what kind of evidence it could produce.

That’s probably a good enough place to stop for now, but I’d be remiss if I didn’t add in another under-appreciated factor that is, as I see it, essential alongside compelling evidence: engagement.? I’ve often said that RWD will just “sit there” unless coupled with intent: similarly, evidence will not achieve its full potential unless the stakeholders we hope to inform are part of a targeted engagement / communication process so that they can, ultimately, fully appreciate the evidence.