More or Less or Not at All?

The placebo topic seems to invite sloppy methodological thinking.?[1]

Wireless Wisdom

More or Less is an excellent BBC programme that manages the improbable task of dealing with statistics on the radio: no printed figures, no graphs, just words and all this without even the advantage of Victor Borge's phonetic punctuation. Indeed, its excellence has been recognised many times in the Royal Statistical Society journalism awards.

It is presented by the well-known financial journalist, Tim Harford, author of many readable and informative books on economics and related themes. As the BBC's website introduction to the programme states

Tim Harford explains - and sometimes debunks - the numbers and statistics used in political debate, the news and everyday life

Indeed, in the programme of 26 January 2022 a YouTube claim that very few people died of COVID, they were simply dying with COVID, was carefully examined and shown to be untrue. Other matters that were examined carefully were whether inflation may effect the poor more than the consumer price index might suggest (Yes) and whether it is true that if food that is dropped on the floor is picked up within five seconds it will be germ-free (No).

Great Expectations

However, also included was an interview (see minutes 19 to 23 of the programme) with scientific writer David Robson about his latest book The Expectation Effect which, according to his website

is a journey through the cutting-edge science of how our mindset shapes every facet of our lives, revealing how your brain holds the keys to unlocking a better you.

Well, of course, statisticians are a miserable tribe of unbelievers. It's not positivity we embrace but negativity and as I often explain to clients, "whatever you believe in I don't", so it is hardly surprising that I have no great expectations of an expectation effect.

However, I also have to confess that I am a complete ignoramus in all things psychological, so it could, indeed, be true. It could be that I am completely underestimating the power of positive thinking but my disappointment with More or Less's presentation was not so much with what was claimed (in fact the book sounds intriguing and I have ordered a copy) but rather with what was not examined. What was not examined was, 'how does one measure the placebo-effect?'.

Controlling the control

How do you judge the effect of treatment in a randomised clinical trial? Well, you use concurrent controls. In a parallel group trial, the results obtained for patients given a new treatment are compared to those of patients given a control treatment. If you are trying to show that the new treatment is better than what is currently available, you will use a standard treatment for the control group. On the other hand, if you want to try and estimate the absolute pharmacological effect of a treatment, you will use as your control a placebo, a dummy treatment, identical in appearance to the intervention but without any pharmacologically active ingredients.

Why is this done? One reason, certainly, is because one suspects that there may be a so-called placebo effect. It may be that patients will improve simply because they have an expectation of improvement and this expectation affects the outcome. Since I am a firm believer in the value of placebo controls, is it not hypocritical of me to demand proof of the reality of the placebo effect?

No. There are many reasons why the condition of patients may improve during a trial that are unrelated to any treatment effect. In their critical review[1] of Beecher's influential paper of 1955, The Powerful Placebo[2], Kienle and Kiene identified several. An obvious one is trend effects and a more subtle one regression to the mean. I shall discuss regression to the mean in due course. Trend effects are simply explained by example. Given enough time headaches do spontaneously resolve. Therefore it is highly plausible that some patients who suffered from headache and improved having been given placebo would have improved anyway. In the words of Dr Johnson 'subsequence is not consequence'.

Does this matter? Not usually; the point is that as long as you are controlling for all the factors that might affect patient outcomes apart from the pharmacological one of interest, it is only their joint total effect that matters and this is expressed in the outcomes for the placebo group. As soon, however, as you are interested in the placebo effect itself, where this is taken to be some psychological expectation effect, you need a control group for the placebo group. Effectively, you need a three-armed trial: treatment, placebo to treatment and nothing at all.

Nothing will come of nothing

For understandable reasons, very few pharmaceutical companies are interested in the placebo effect per se. They wish to control for all the various biases that might arise to obtain an estimate of the pure pharmacological estimate of placebos. Hence, they have little interest in adding an extra 'open' arm to a placebo-controlled trial. However, independent researchers have occasionally run such three-armed trials and, inevitably, they have been subject to meta-analysis. In a paper[3] of 2001 in the New England Journal of Medicine, which analysed 114 trials, Hrobjartsson and Gotszche came to the following conclusion

We found little evidence in general that placebos had powerful clinical effects. Although placebos had no significant effects on objective or binary outcomes, they had possible small benefits in studies with continuous subjective outcomes and for the treatment of pain.

To be fair, a subsequent paper by these authors[4], together with Krogsb?ll had a more positive assessment. Taking 37 three-armed trials they found

Spontaneous improvement and effect of placebo contributed importantly to the observed treatment effect in actively treated patients, but the relative importance of these factors differed according to clinical condition and intervention.

However, they also found

...on average, the relative contributions of spontaneous improvement and of placebo to that of the active interventions were 24% and 20%, respectively, but with some uncertainty ...

Thus, the message is that one is at risk of considerably overestimating the placebo effect if one simply compares after to before in a clinical trial.

Regression to the Mean

I have often published on this topic (see, for example, [5]) but rarely (if ever) blogged on it, so I shall take the opportunity to explain it here. If subjects are selected for inclusion in a study because their values are extreme, then, other things being equal, these values may be expected to be less extreme at the outcome of the study.

The figure above illustrates the problem. The left hand panel shows a scatterplot for diastolic blood pressure (DBP) at the end of a study (on the Y axis) against DBP at the start of the study (on the X axis). A value of 95mmHg has been set as the threshold for being hypertensive and is shown by dashed vertical and horizontal lines. The line of equality is shown as a dashed diagonal.

The correlation is less than perfect. Some patients improve and some get worse. Those who are hypertensive at baseline and outcome are shown in red, those who were normotensive on both occasions are shown in blue and those who moved from being normotensive to being hypertensive or vice versa are shown in orange. There is no evidence that the treatment works.

The problem is that we won't see something like the left-hand panel. Why? Because those with a measured DBP at baseline less than 95mmHg won't come into our trial. Instead what we shall see is something like the right-hand panel. We shall still see the orange improvers in the study but we won't see the orange deteriorators.

Does this matter? Not if we have a control group. We can then judge the pattern that we see not by comparing outcome with baseline, which would be profoundly misleading, but by comparing the treated group to a control group.

Now suppose that the diagram is actually for a placebo group. We shall observe a magnificant placebo effect but it has nothing to with psychology and everything to do with statistics.

References

1. Kienle GS, Kiene H. The powerful placebo effect: fact or fiction? Journal of clinical epidemiology. 1997;50(12):1311-8.?

2. Beecher HK. The powerful placebo. J Am Med Assoc. Dec 24 1955;159(17):1602-6. doi:10.1001/jama.1955.02960340022006

3. Hrobjartsson A, Gotzsche PC. Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. New England Journal of Medicine. 2001;344(21):1594-602.?

4. Krogsboll LT, Hrobjartsson A, Gotzsche PC. Spontaneous improvement in randomised clinical trials: meta-analysis of three-armed trials comparing no treatment, placebo and active intervention. BMC Med Res Methodol. Jan 5 2009;9:1. doi:10.1186/1471-2288-9-1

5. Senn SJ. Three things every medical writer should know about statistics. Review. The Write Stuff. September 2009 2009;18(3):159-162.?