Blind Date

The value of ignorance

Many randomised clinical trials are run as double-blind studies. Neither the subject nor the investigator knows which treatment the subject will receive. This is generally regarded as being extremely valuable as a way of guarding against the placebo effect. See Fooling the Patient. No doubt, this is a useful feature of such trials but in fact, there are reasons to believe that the placebo effect has been greatly exaggerated. (See references 2 & 3.) The fault that has been made here is of comparing the result under placebo to the baseline. However, the baseline value is a very inadequate yardstick for many reasons, including trend effects and regression to the mean.

It is not always appreciated that randomisation is valuable for effective blinding. However, as RA Fisher put it:

...if I want to test the capacity of the human race for telepathically perceiving a playing card, I might choose the Queen of Diamonds and get thousands of radio listeners to send in guesses. I should then find that considerably more than one in 52 guessed the card right...this sort of thing arises because we are in the habit of making tacit hypotheses, e.g. 'Good guesses are at random except for a possible telepathic influence...'

(Bennett 1990, P268-269).

So it is unwise in making any deliberate choice to assume that the subject might not guess this choice. To assume that it is safe is what I have referred to as the argument from the stupidity of others.

Of course randomisation may be necessary for effective blinding but is is not sufficient. How one assures that trials are double blind is often a difficult technical matter. I have covered this is a previous post and don't intend to cover it here. Instead, I shall consider what happens when it can't be done or one has chosen not to.

Repeat warning

The reader is reminded that although I have researched and written for many years on the methodology of clinical trials and have been involved in many, I have no experience of trials with vaccines.

Blind bind

It is well-understood that just as randomisation helps blinding, so blinding helps randomisation. If a trial is open, then a physician could delay entry of a subject onto the trial to ensure that the subject gets the treatment the physician thinks is appropriate. This is usually handled by using a central randomisation: the physician must first enter the subject onto the trial (having of course obtained consent to be randomised). Only once this has been done and the fact has been recorded, is the treatment revealed.

However, what is less well-appreciated is that blinding has a much stronger effect on maintaining random allocation than this. It is the purpose of this post to explain how and I shall use the example of trials for vaccines, since this is topical and it in any case provides a strong example.

The value of blinding to which I wish to draw attention is that it strongly limits the freedom of subjects and physicians to depart in any way from the plan and thus subconsciously subvert the randomisation procedure.

Consider a trial of a vaccine which is randomised but not double blind. The simplest thing would be just to have a control arm of subjects who are unvaccinated. Subjects are invited to participate. Those that consent to do so are entered on the trial. The randomisation list is consulted and treatment then procceds accordingly: those who have been randomised are invited to receive the vaccine and those who have not act as controls.

Here are some things that might happen. (I am sure that one could think of others.)

1. Subjects who are chosen to be vaccinated are invited to attend a clinic to receive their vaccine.
2. A team of health workers is assigned for vaccination and another team is assigned for assessing controls.
3. Blood samples are collected by the vaccinating clinic.
4. Subsequent blood samples (say after 28 days) are also collected in the vaccinating clinic.
5. Samples are sent in batches to the laboratory to be analysed.
6. Control subjects are visited by nurses at home to collect blood samples.

All of these have the ability to subvert the randomisation process. With the exception of the first and the last and possibly the second they are not biasing per se. But they make very debatable any assumption of independence that might be naively made in analysis. A biasing factor is one that prevents the estimate from converging on the 'right' answer as the sample size grows but attracts it to some other value. Lack of independence, on the other hand, affects the rate of convergence.

The first and the last are potentially biasing because they offer different possible exposure to infection depending on the group to which a subject is assigned. Gathering subjects together may place them at greater risk but so might a stranger visiting them at home.

The second is somewhat different. There may be worker-effects to account for and it could be that randomisation could take some cluster form to allow for this. This would, however, affect the way that standard errors ought to be calculated. Such clustering is a potential problem for any data-set that is not randomised in a simple way and hidden clustering may be a powerful and unacknowledged factor invalidating big data analyses. See To Infinity and Beyond .

The other three are affected by assay differences. The problem here is not that the assays used will systematically differ from vaccinated subjects to control subjects but that there may be some sort of autocorrelation such as might be induced by batch to batch variability that might invalidate naive calculation of standard errors.

Declaration of Independence

The usual formulae for calculating standard errors and hence for calculating confidence intervals or P-values or, for that matter, conducting Bayesian analyses, assume that the obervations are independent conditional on the model employed.

The underlying process being modelled may be far from independent but randomisation will induce a form of pseudo-independence that renders the usual assumption appropriate.

Figure 1. A randomised clinical trial in which there is strong autocorrelation between subjects as regards a covariate value.

Figure 1 shows data for a supposed covariate simulated for 200 subjects from a simple AR1 (first order autoregressive) process with a mean of zero, a variance of one and a correlation of 0.9. The auto-correlation displays itself as waves over time. However, subject to the constraint that there are 100 patients per arm, treatments have been allocated at random througout the process. In consequence, allocation is 'blind' as to date and this means that values may be treated as if independent and that standard analyses of clinical trials will be valid.

For Figure 2, however, I have supposed that actual reception of treatment in the trial is affected by some autocorrelation process. I have used an independent AR1 process with the same parameters as for the covariate. (However, the two processes are generated independently of each other.) I have compared each random number to their median and if the value is above 1 declared the subject to be vaccinated and if not, a control subject. Again there are 100 patients per arm.

Figure 2. A trial that was designed to be randomised but in which delivery of treatment has been effected by an AR1 process.

The net effect is that treatments are no longer distributed randomly (or at least not simply randomly) across time. This means that clustering in time of treatment delivery may associate it with other factors that are clustered in the trial. This is not necessarily biasing per se, in the sense that as the sample size grows the estimate may converge on the right answer, but the rate of convergence will be much slower and if we are not careful we may calculate our standard errors optimistically and assume that a matter is settled when it is not.

Being led by the blind

How does having the trial double-blind help avoid this? By making it impossible to associate any process of delivery or measurement with treatment. Take the first. Suppose you wish to bring subjects into the clinic to be vaccinated. Fine. But all you can do is bring them into the clinic regardless of assignment to the treatment or control group, since you are blind as to whom is receiving what. Thus, there will be local control for any factors associated with bringing subjects into the clinic. Suppose the clinics send the samples off to be tested. The samples will consist of a random mixture of those in the control group and those in the vaccine group. Batch to batch variability will not invalidate the analysis.

In short, the fact that nobody knows who is getting what maintains the randomisation with respect to all delivery and measurement processes

Moral

I am not suggesting that all trials should be double blind. I am not even suggesting that all vaccine trials should be double blind. What I am suggesting is that randomisation is very valuable and where one is not able to run such trials double-blind, one should be very careful of the 'habit of making tacit hypotheses' that Fisher warned us of.

Good design trumps data-curation. Think, don't assume. Act, don't react. Be very, very careful.

References

1. Bennett JH. Statistical Inference and Analysis Selected Correspondence of R.A. Fisher. Oxford: Oxford University Press; 1990.

2. Hrobjartsson A, Gotzsche PC. Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. New England Journal of Medicine. 2001;344(21):1594-1602.

3. Kienle GS, Kiene H. The powerful placebo effect: fact or fiction? Journal of Clinical Epidemiology. 1997;50(12):1311-1318.

4. Senn SJ. Fisher's game with the devil. Statistics in Medicine. 1994;13(3):217-230.

5. Senn SJ. Seven myths of randomisation in clinical trials. Statistics in Medicine. 2013;32(9):1439-1450.