Lauding Lord: a postscript

AND Joshua cast lots for them in Shiloh before the LORD: and there Joshua divided the land unto the children of Israel according to their divisions. Joshua: 17, 10.

So he brought down the people to the water: and the LORD said unto Gideon, Every one that lappeth of the water with his tongue, as a dog lappeth, him shall thou set aside; likewise every one that boweth down upon his knees to drink. Judges: 7, 5.

An Important Difference

If I had read the useful introductory text Causal Inference in Statistics: A Primer [1] by Pearl, Glymour and Jewell before I had read The Book of Why [2] by Pearl and MacKenzie (also a book I can recommend), I would have discussed Lord's Paradox differently. I shall explain why, in due course and in doing so shall refer to these two books as CIS and BoW. For the moment, suffice it to say that Lord's paradox features in both but the presentation of it differs from one text to another.

As followers of my blogs will know, Lord's Paradox[3] is a topic I have frequently discussed and in particular the solution that is offered in The Book of Why. I first explained why I objected to the solution here: Rothamsted Statistics Meets Lord's Paradox.

The paradox is best illustrated by example and an artificial data set is given in the next section. (It would be much nicer to have a real example, but I don't have one!)

A Fictitious Example

Figure 1. An illustration of Lord's paradox using fictitious data created by a simulation. Students are following one of two diets (A or B) over a period (say a scholastic year beginning in September and ending in June). The points are final weight plotted against initial weights. The black diagonal line is the line of equality whereas the blue and red lines are those that would be used in an analysis of covariance (ANCOVA). The difference between them would be the estimate of the treatment effect.

At its most basic level, Lord's Paradox occurs simply because two different analyses, each of which seems quite reasonable, will give two different results. It is supposed that two groups of individuals are following different diets and their initial weights when the diets are given are recorded, as are their final weights after a suitable period of following the diet.

Figure 1 shows an imagined situation. Note that the two groups are extremely unbalanced at baseline. This is deliberate in order to make the results more obviously different by method of adjustment and is not meant to be realistic. If a statistician (say John) proposes to adjust for initial weight simply by subtracting the initial weight from the final data to produce a change score (sometimes called a gain score) and then performs a two-sample t-test, the result can be illustrated by Figure 2. There is no appreciable difference between the two groups as regards the change score.

Figure 2. The change score (final weight - initial weight) for each of the the two diets

However, if the statistician performs an analysis of covariance, the fitted parallel lines have a slope of 0.46. Thus, the ANCOVA approach adjusts the final weight by subtracting 0.46 times the initial weight. The situation can be illustrated in Figure 3 which now shows that there is a noticeable difference between groups. (Note that as regard judging 'significance', it would not be adequate just to perform a t-test on these adjusted data, since there is a penalty for estimated adjustments that has to be paid in calculating the standard error. See appendix to Siegfried et al[4] for a discussion. Nevertheless such an analysis would give approximately the same answer as ANCOVA.)

Figure 3. Adjusted final weights using ANCOVA.

Tell me a story

Of course, data like these do not, on their own, suggest how they should be analysed. Before analysing a data set, every statistician needs to ask themselves the question.

" How did I get to see what I see?"

We need a story describing how the data arose. In my blog Lauding Lord, I give four different (fictitious) stories as to how the data might have arisen. Here I shall consider the stories given by CIS and BoW.

I start with the former.

At the beginning of the year, a boarding school offers its students a choice between two meal plans for the year. Plan A and Plan B. The student's weights are recorded at the beginning and the end of year. (Pearl, Glymour & Jewell, 2016, P65.)

In the latter book, the story is as follows.

...the students eat in one of two dining halls with different diets. (Pearl & McKenzie, 2018 P216.)

Of course, either of these two stories is problematic as regards analysis. In the first case it is stated that the diet is chosen by the student and not allocated, so that, unlike in a randomised experiment, choice is confounded with diet. This is the point of the opening quotations, where, in the biblical story of Gideon, those that make up Gideon's army are determined by their actions (choices) rather than chosen by lot, a device used by Joshua.

In the second case, it is not clear how diets are assigned but irrespective of how they are assigned, they are delivered by different halls. Thus hall is confounded with diet. As I explained in a previous blog, this may not matter if we are interested in assessing the joint effect of both but if we consider that diet is something of interest in itself and that for any given hall, different diets could be chosen, it does matter.

Twice two solutions

In CIS the following possible analyses are presented

The first statistician calculated the difference between each student's weight in June (W_f) and in September (W_i) ...the second statistician divided the students into several sub-groups, one for each initial weight W_i. (Pearl, Glymour & Jewell, 2016, P65.)

In BoW we have

The first statistician claims...that switching from Diet A to B would have no effect in weight gain (the difference W_f - W_i has the same distribution...). The second statistician compares the final weights under Diet A to those of diet B for students starting with weight W_0... (Pearl & McKenzie, 2028 P216.)

In other words the analysis of statistician 1 is the same in both stories. For statistician 2 a form of "non-parametric" adjustment by stratification is proposed in CIS. In BoW, other discussions suggest that, as in Lord's original story, analysis of covariance (ANCOVA) will be used. For reasonably well-behaved data, these two approaches can be regarded as equivalent. The stratification approach is not so dependent on an assumption of a linear relationship between initial and final weight. However, it eventually runs into the sands as finer and finer strata are defined. Alternatives to both approaches exist and one is illustrated in Figure 4, in which the relationship between final and initial weight has been modelled using a polynomial of order 4. Thus, a more complex adjustment for weight is possible. (In the header for this blog, a fit of order eight is illustrated, a degree of complexity I would never consider in practice.)

Figure 4. Adjustment using a polynomial of order 4 for the relationship between initial and final weight.

However both stratification and regression adjustment are forms of conditioning, so essentially both CIS and BoW have the second statistician doing a form of conditioning and both appear to approve of this approach to analysis.

The Rothamsted Approach

However, the stories in the two books are not the same. The Rothamsted approach pioneered by statisticians such as Fisher, Yates and Nelder during the middle of the last century would see a fundamental difference between the two.

I have illustrated how this works in a previous blog and will not rehearse the argument here except to say, that the distinction between block and treatment factors is fundamental. Here, in the story given in BoW, the factor Hall can be treated in one of two ways. It can either be regarded as a Block factor, something that is part of the fundamental experimental structure and must be regarded as potentially effecting outcomes whether or not there is a treatment effect or as part of the treatment structure. In the former case, the Nelder algebra will indicate that the effect of diet cannot be estimated: it is confounded with something that potentially already affects final weight and cannot be eliminated simply by conditioning on the initial weight. Alternatively, it could be regarded as part of the treatment structure. In that case, the algebra will tell you that the joint effect of diet and hall can be estimated but they cannot be disentangled. In neither case will it accept the solution given in BoW to the story told in BoW although, ironically, the solution would be valid if we could assume that student choice was irrelevant given initial weight and the story was as in CIS.

Further reading

Possible stories and their solutions are given in my blog Lauding Lord. A key paper is that of Holland and Rubin [5]. I have addressed it in [6]. A recent treatment is that of Tennant et al [7]

References

1. PEARL, J., GLYMOUR, M. & JEWELL, N. P. 2016. Causal Inference in Statistics: A Primer, John Wiley & Sons.

2.PEARL, J. & MACKENZIE, D. 2018. The Book of Why, Basic Books.

3. LORD, F. M. 1967. A paradox in the interpretation of group comparisons. Psychological Bulletin, 66, 304-305.?

4. SIEGFRIED, S., SENN, S. & HOTHORN, T. 2022. On the relevance of prognostic information for clinical trials: A theoretical quantification. Biom J.

5. HOLLAND, P. W. & RUBIN, D. B. 1983. On Lord's Paradox. In: WAINER, H. & MESSICK, S. (eds.) Principals of Modern Psychological Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates.

6. SENN, S. J. 2006. Change from baseline and analysis of covariance revisited. Statistics in Medicine, 25, 4334–4344.

7. TENNANT, P. W., TOMOVA, G. D., MURRAY, E. J., ARNOLD, K. F., FOX, M. P. & GILTHORPE, M. S. 2023. Lord's 'paradox' explained: the 50-year warning on the use of 'change scores' in observational data. arXiv preprint arXiv:2302.01822.