Scale Fail

Hype or Gripe?

I consider that the rapid development of vaccines for COVID19 has been an impressive achievement. The trials were planned to prove vaccine efficacy of greater than 30% and delivered estimates that were higher, and sometimes much higher, than the value of 60% assumed for power calculations.

However, there have been some griping voices and, in particular, the argument has been put forward, that the number needed to vaccinate is very high for all these vaccines and that this suggests that the vaccines are not particularly effective. I consider this argument to be not only false but rather stupid for reasons I shall now explain.

The Case for Cases

It seems reasonable to suppose that if a vaccine were highly effective it would help prevent many cases that would otherwise have happened were the risk of infection high. This requires, however, not only that the vaccine is technically effective in invoking a sufficient immune response but also that individuals are at a high risk of infection. If few unvaccinated subjects are being infected, then the vaccine cannot prevent many infections.

One might argue that if few subjects are being infected we don't need a vaccine but this overlooks the fact that it is the nature of infectious diseases for the numbers of cases to vary dramatically over time, it is high numbers of case we wish to prevent and that to the extent that we succeed in this aim, we shall inevitably have to study efficacy before the disease is at an unchecked height.

If it were the case that most of the subjects in the control arm were infected, then most of the population outside of the trial would almost certainly be infected also and the only persons to benefit from any effective vaccine would be the relatively few (of the total population) in the treatment arm. This would render the vaccines pointless.

I apologise for labouring the obvious but it seems that many commentators are failing to grasp the simple point that vaccines will inevitably be studied in circumstances that are not representative of the context in which they will be applied and that this requires us to think about what scale we should use for the results.

Observed and Expected

One way to argue is as follows. Suppose that we have approximately nv subjects on the vaccine arm of the trial and np on the placebo arm. An estimate of how many cases we would have seen in the vaccine arm is provided by the case rate in the placebo arm when applied to the numbers of subjects in the vaccine arm. In other words, by calculating

Expected cases under vaccine = nv x Yp/np

where Yp is the cases on the placebo arm, we estimate the cases we would expect to see in the vaccine arm had the vaccine been useless. (If np and nv are equal this is simply equal to the number of cases under placebo.) However, in fact, we observe a number of cases, Yv, in the vaccine arm. We thus estimate that the number of cases of infection in the the vaccine arm that we have prevented by vaccination as expected minus observed and thus as

Estimated cases prevented = (nv x Yp/np)-Yv.

Divide and conquer

So far so good but in order to express this as a rate, what should we use as a divisor? (Clearly some sort of divisor is necessary, if we wish to understand the implications for the general population.) If we use the number of subjects, in the vaccine arm nv then we get the Risk Difference, RD

RD={(nv x Yp/np)-Yv}/nv=Yp/np-Yv/nv.

The reciprocal of this is the number needed to vaccinate so that those who like this number as an expression of the effectivness, are implicitly using the overall number of subjects as the relevant denominator.

But is this reasonable? We have just divided the reduction in cases by the number vaccinanted but many, and in a typical trial most, of these subject would never have been infected had they been unvaccinated. If instead, we divide by the number of expected infections in the vaccine group (if the placebo rate applies) we get the so called Vaccine Efficacy, VE

VE={(nv x Yp/np)-Yv}/{(nv x Yp/np)}=(Yp/np-Yv/nv)/(Yp/np)=1-(Yv/nv)/(Yp/np).

Does it matter?

It certainly does. The figure below plots the estimates and 95% confidence intervals for five large trials for five vaccines. (Note that four of the vaccines are given in two doses, the exception being that of J&J Janssen, which is give as a single dose.) The vaccine efficacies (as a percentage) have been plotted against the risk difference (as a percentage) and it can be seen that there is little relationship between the two. If you favour risk difference (or equivalently, number needed to vaccinate), then judging by the point estimate you should regard the Pfizer/BioNTech vaccine as being the least effective. On the other hand, if you favour vaccine efficacy, then you ought to regard the Pfizer/BioNTech vaccine as being the most effective. (Of course the confidence intervals show that there is a great deal of uncertainty.)

Vaccine efficacy versus risk difference for five large vaccine trials

(Note that the figures are based on my calculations and not those of the companies. As I have pointed out in various blogposts it is possible to get very close to the figures the companies have calculated for vaccine efficacy just using numbers of cases and the allocation ratio. The companies have very sensibly refrained from doing something so stupid as calculating a risk difference but I have just used cases and numbers treated to do so.)

Portability matters

A mistake, that is made over and over again is to assume that clinical trials are, or ought to be, representative of the target population. They rarely are and should not be required be so. They are experiments not surveys. We need to use these experiments to predict what we would see in the population to which the treatments are applied but this does not require that subjects in the trial be representative of those in the population. It requires the scale to be portable. You may regard the risk difference scale as being portable but in that case I think that you are wrong. I think that the risk difference scale is pretty much irrelevant.

The vaccine efficacy scale is not perfect but it is much more plausible that as infection rates rise and fall it is that which would be more stable. The risk difference scale would vary greatly depending on the state of the epidemic. That's a very good reason not to use it.