Searching for meaning

[Repost from the book chapter, here]

What do we mean when we say ‘meaningful’?

There may be disagreement about the role of the 0.05 p value in ‘statistical significance’, but it’s easy to ignore that the ‘significant’ part of that phrase does not mean ‘meaningful’, but rather means ‘probably true’.

There are also bookshelves devoted to the scientific definitions of ‘clinically meaningful’. Papers will discuss the thresholds of minimum detectable difference (MDD) and minimum clinically important difference (MCID), and attempt to derive a ‘clinically meaningful’ threshold. Let’s be careful with that term here - it does not mean that it is a difference that matters to a patient, just that it represents a noticeable, and apparent, difference in a measure such as Global Rating of Change Scale (GRS), where a patient feels something as vague as ‘improvement’.

Tests like the 6 minute walk test (6MWT, how far a patient can walk at sub-maximal effort in 6 minutes) have the value of being relatively easy to perform, although with wide variation in daily results for most patients. You or I might manage 500m in 6 minutes, but a patient with heart failure might do 300. It isn’t a heart failure-specific score, of course, as COPD, MS and a lot of other conditions would reduce exercise capacity.

So, it is a thing we can measure. When we’re positioning, we need to consider more: does it matter, and does it differentiate??

For many patients with heart failure, a walk of 300m is a considerable undertaking. Two flights of stairs might be inconceivable. Their adaptation may well mean they cope by not doing either of those things - increasing the downward spiral. So, if a drug increased that 300m to 330m, most physicians would say that was a meaningful change in the patient’s status. But let’s put that onto the stairs. It might be 11 steps instead of 10. We could argue about the utility of that extra step, and it’s entirely possible that an agency could craft all kinds of nice ads about that last step, but we then need to decide whether the 10% is the point. Could it be about the first step, not the last? The willingness to start, rather than the ability not to stop so soon? The anxiety, the sense of improvement vs decline, the walk to the car instead of sitting in front of the TV? So, to the question, ‘does it matter?’, the answer might well be ‘it depends.’

Then, to the thornier issue: does it differentiate? Because it is easy to measure, it is easy to mandate as a regulatory endpoint. It isn’t essential (even for heart failure), but most regulatory groups internally will tell you that ‘the FDA says you have to…’ So, you collect the data, against placebo, and then you find that you’re playing the game of numbers - is my 12% better than your 13%, given that we had different NYHA status patients in our studies? How much, then, is that extra 2 metres worth to a payer, or a physician, or a patient, over a generic, or over standard of care?

If you know this is a problematic conversation for your Commercial colleagues to have, when should you have begun to think differently? It is entirely possible that the endpoint that does measure the value your drug brings uniquely could have been used in your phase III (alongside a 6MWT). But, it should then have been validated in your phase II. Which would mean that you’d considered it before designing that phase II programme. A drug that ‘works’ in heart failure is not the ask, even if it works well enough to be approved. A drug that changes people’s lives meaningfully is the task, and to fulfil that task, we have to search for a shared version of ‘meaningful.’