Random, yes, but sampling or isation?

Two uses of a die

Recent exchanges on Twitter @stephensenn revealed some confusion as regards the difference between randomisation, which is regularly used in the conduct of clinical trials and random sampling, which is not. I speculate that the origin of the confusion is to do with courses on statistical inference, which frequently take the latter (with random sampling from an infinitely large population) as a paradigm to illustrate fundamental concepts. Elsewhere[1] I have put this like this

It is misleading that a standard statistical paradigm, to which textbooks often return, is that of estimating a population mean using simple random sampling. For this purpose, the parameter estimate of the simple model is, indeed, the same as the prediction. However, as soon as we turn to more complex sampling schemes, this is not so. Stratified random sampling, for example, yields estimates of stratum means from which the population mean can be predicted using the sampling fractions if one wishes, but there is no immediate connection between any of the parameters estimated and the target quantity. There is also a very common confusion between samples and experiments: The latter carry with them no necessary implication about any population quantity whatsoever.

(p228)

I thought it would be useful to explain the difference and that is what this blog tries to do.

Disclaimer

The design of experiments, to which randomisation is relevant, and sampling theory, for which random sampling is used are two vast fields. I know a fair amount about a particular corner of the former, the design and analysis of clinical trials, but my knowledge of the rest of that field is limited. My knowledge of sampling is much more limited. I have (very reluctantly) taught the elementary theory on various courses. My personal experience is related to limited work in my first job, 1975-1978 and a little bit of consulting advice from my second job, 1978-1987. The reader is warned.

Randomisation

An important field but not using fields

I shall illustrate randomisation by using the application area I know best: controlled clinical trials. This is not the one in which it was first proposed. That was agricultural research, the scientific field where RA Fisher, who worked for Rothamsted agricultural research station in Harpenden from 1919-1932, first applied the idea. For a discussion of that see The Rule of Three .

The representativeness fallacy

It does not matter which field you look at but scientists do not use representative material when constructing experiments. This is so well understood when physics and chemistry are considered that it is not even discussed. But, when human biology is concerned, commentators frequently assume that representativeness would not only be desirable but essential.

However, experiments are what scientists use to help build and validate models and the models are then used to make predictions in other contexts. There is no guarantee where such predictions are made that they will be correct but they may be better than the alternative.

In my opinion, clinical trials are no exception. They rarely lead to a deep mechanistic understanding but they can nevertheless help in producing working models. A good example is bioequivalence studies. These use healthy volunteers to make judgements of suitability for patients. The healthy volunteers are not taken to be similar to patients, indeed they are known to be not like patients when it comes to the measurement of interest, area under the concentration curve. They are studied because it is assumed (with some justification in terms of theory and experience) that although the absolute AUC will differ from volunteers to patients in important ways, the relative AUC for a generic compared to an innovator product would be the same in healthy volunteers as in patients. In particular, it would be strange if relative AUC could be close to one for healthy volunteers but considerably different for patients. See Invisible Statistics for a discussion.

Randomisation in clinical trials

In clinical trials we have little control over the presenting process. Patients cannot present if they don't fall ill but if they fall ill they may not present and if they present they may not consent.

It may be imagined that inclusion criteria govern who comes into a clinical trial but it would be more accurate to say that they control who does not come into a clinical trial. To see this consider (to anticipate a later section) sampling of voters in an election. Clearly, the essential inclusion criterion is 'eligible to vote' (in the election of interest) and we do not wish to include anyone not eligible to vote but nobody would suppose that each and every possible way of selecting persons eligible to vote in a sample provided a valid basis for assessing voter intentions.

The logic of clinical trials is quite different. It is comparative rather than representative. Although we do not have control over the presenting process we have precise control over the allocation algorithm.

Ways and means

The way that randomisation works is that a random (or pseudo random) mechanism is used to allocate treatment to the experimental units. In a parallel group trial, the units are patients. Thus, in a trial with 100 patients we might assign 50 patients to the intervention and 50 to the control. Note that this is nearly always a dynamic process. Who the patients will be will be discovered as the trial unfolds.

In order to balance patient numbers we might use the technique of permuted blocks. For example using a block of size four we would ensure that for every four patients who agreed to enter the trial, two patients received treatment A (say) and two received treatment B (say), the order being random. There are six such orders:

AABB, ABAB, ABBA, BBAA, BABA, BAAB

So the randomisation process consists of generating at random such sequences, stringing them together and given the next patient to be recruited the treatment indicated. Other approaches are possible. See the classic text by Rosenberger and Lachin[2].

In practice, most trials are multi-centre trials. The sponsor obtains an agreement from a number of clinical centres to (try and) contribute a given number of patients. Usually, for a given centre, different patients will be allocated at random centre by centre to the treatments being compared, using, for example, the method of permuted blocks . Sometimes, however, for practical reasons, all the patients in a given centre will be allocated the same treatment at random. This is what is then referred to as a cluster randomised trial.

Sometimes different strata of interest amongst the potential patients are identified and a form of stratified randomisation is used. Such stratified trials, target balancing numbers of patients within strata and not just for the trial as a whole. The method of permuted blocks might then be applied for each stratum to achieve this. Typically such trials will also be multi-centre.

For some chronic diseases it may be possible to study the treatments being compared in the same occasion on different occasions. Such designs are referred to as cross-over trials. The simplest would allocate patients at random either to A followed by B or to B followed by A, with, perhaps, a suitable wash-out between treatments.. Note that the unit of observation is now an episode of treatment and it is sequences of treatments not treatments per se that are allocated at random. The method of permuted blocks could be used for allocating sequences.

Repeated cross-over trials, for example in which patients are allocated at random to one of four sequences of treatment pairs,

ABAB, ABBA, BABA, BAAB,

may permit identification of individual response[3]. Trials in which patients are randomised to several episodes of treatment in this way are sometime referred to as n-of-1 trials.

Analysis

Trials with different designs such as parallel group trials, multi-centre trials, stratified trials, cluster-randomised trials, cross-over trials, n-of-1 trials, require different methods of analysis but the analysis may also depend on the purpose.[4] For further discussion of randomisation in clinical trials see[5].

Random sampling

...in experienced hands sampling may give unsatisfactory results, owing to the use of faulty methods of selection. The prime requirement of any large-scale sample survey is therefore that the organization of the survey should be carried out by a person who has adequate knowledge of sampling methods and their application. (P5)

The reader is reminded that I am not such a person. The quote is taken from Frank Yates [6] who, having established himself in the 1930s as a giant amongst theoreticians on design of experiments, then went on to make important contributions to the very different field of sampling theory.

Where and why

Random sampling is used where it is desired for economic or practical reasons to make a statement about a population of units but it is economically impractical to measure or record them all. Usually this is because the resources available are insufficient to permit a complete enumeration. This was the case when in 1977 I took a sample of case-record forms from the hospitals of my then employer the Tunbridge Wells Health District to try and establish how many prescriptions were typically recorded on a form. The resource constraints were my time, that of the medical records officer and that of a pharmacist. To take a sample, I had to draw up a sampling frame. The frame consisted of a list of all the case record forms according to the discharge register. I am ashamed to say that I did not take a random sample but just a sample at regular intervals, which I then treated as if it were random. (Hangs head in shame and blames it on his youth.)

Another situation where sampling is required is that of destructive examination of items in batches of industrial product. If you test everything, you have nothing left to sell. Thus the properties of the whole have to be determined using a part. Two such cases where I was involved while working on my second job in Dundee (1978-1987) were strength testing of paper sacks by a local manufacturer and examination of contents of whisky bottles by Customs and Excise.

Simple, isn't it?

If the population is of size N and we require a simple random sample (srs) of size n, the sample is chosen at random from the

(N n) distinct possible samples, in each of which no possible member is chosen twice.[6] (P32)

Simple random sampling can be defined as a process of drawing items from a sampling frame (the complete list of all items that might be drawn) in such a way that every possible set of items has an equal chance of being drawn. This may seem rather complicated but it reflects the fact that when sampling from a finite population, the probability of drawing a particular item is affected by the probability of having drawn another. Thus, independence is too strong a property to require.

The fact that the population is finite means that common formulae used for standard errors in theoretical statistical examples, where the iid (independent identically distributed) assumption is commonly made, are not quite right. For sampling they require finite population correction factors, a topic which, in my opinion, involves a lot of tedious but elementary algebra and is one of the reasons I hated teaching sampling theory. Provided that N (the number in the population) is large compared to n (the number in the sample), these have little effect.

Note that simple random sampling (srs) is an example of an equal probability selection method (epsem), whereby every member of the population has an equal chance of being chosen, but that not all epsem methods involve srs. Consider a case where we have n pupils in each of m schools and wish to say something about the N = mn population of students. If we choose one of the m schools at random and examine all the n pupils, each pupil in the population has an equal 1/ m chance of being chosen. This is an epsem. It is not an srs since many samples (all those involving mixtures of pupils from different schools) have zero chance of being chosen.

But you can make it more complicated

Analogously to clinical trials you can have stratified sampling. You can use a sampling frame for each of a number of strata of interest and then carry out sampling from these. Depending on the purpose of such a procedure (for example to make equally precise statements about each stratum or to make statements about the overall population) different schemes could be appropriate.

Sometimes, for convenience, cluster sampling can be appropriate. If door to door visits are involved, simple random sampling could require a great deal of travel. Choosing streets at random and contacting all those in the street will reduce the cost per person contacted but this will probably come at a price. A larger overall sample size may be necessary to overcome adverse effects of clustering on precision.

Model based versus design based inference

What both methods have in common is that there is a theory to relate the inferences that may be made to the way that allocation (experiments) or selection (surveys) has been conducted. Whether the design should dictate the analysis, whether the intended analysis should guide the design or whether some mixture of the two is appropriate, is a deep and fascinating issue on which I and others more expert than me have various opinions but this blog has gone on long enough, so I shall stop here.

References

1. Senn SJ. Conditional and marginal models: Another view - Comments and rejoinders. Comment. Statistical Science. May 2004;19(2):228-238.?

2. Rosenberger WF, Lachin JM. Randomization in clinical trials: theory and practice. Second ed. John Wiley & Sons; 2016.

3. Senn SJ. Three things every medical writer should know about statistics. The Write Stuff. September 2009 2009;18(3):159-162.?

4. Senn SJ. Added Values: Controversies concerning randomization and additivity in clinical trials. Research paper. Statistics in Medicine. Dec 6 2004;23(24):3729-3753.?

5. Senn SJ. Seven myths of randomisation in clinical trials. Statistics in Medicine. Apr 30 2013;32(9):1439-50. doi:10.1002/sim.5713

6. Yates F. Sampling Methods for Censuses and Surveys. Fourth ed. Charles Griffin & Company Limited; 1981:458.

7. Barnett V. Sample Survey Principles and Methods. Third ed. Arnold; 2002:241.