Nonspecific Vaccine Effects Journal Club no. 4

Take control of the narrative – a euphemism of ‘fake it till you make it’

In the Nonspecific Vaccine Effect Journal Club no. 1 the article Clin Infect Dis. 2018 May 2;66(10):1573-1580 was assessed. The article reported a randomized controlled trial in 8309 children, and no nonspecific effect (NSE) of the measles vaccine (MV) was found.

In the introduction of this article it says:

The numbers 7-9 refer to observational studies, 10-15 refer to the randomized trials. As the the #NSEvacc JournalClub focuses on articles reporting trials inferring causality we will herein look into the articles that are referred to as “randomized trials” (italics reflect copy paste from the article’s reference list, without author list and titles):

10. BMJ 2010; 341:c6495.

11. Pediatr Infect Dis J 2003; 22:798–805

12. J Infect Dis 2011; 204:245–52.

13. Clin Infect Dis 2017; 65:1183–90.

14. Pediatr Infect Dis J 2012; 31:306–8.

15. Clin Infect Dis 2015; 61:1504–11.

In this fourth issue of the Nonspecific Vaccine Effects Journal Club we will assess the primary outcome of these six individual randomized controlled trials (RCTs) on the measles vaccine (MV), the Bacillus-Calmette-Guerin (BCG) vaccine, and on the oral polio vaccine (OPV), respectively. We will check and see to what extent they support the claims that are made in the Introduction section of Clin Infect Dis. 2018 May 2;66(10):1573-1580, which is the study that I will keep referring back to - as it the statements made in the Introduction of this article that we will take a closer look at.

So the question of this #NSEvaccJournalClub no. 4 is: Do the results of the randomized trials that are being referenced indicate that the MV, BCG, or OPV have beneficial nonspecific effects by lowering all-cause mortality due to non-vaccine-target infections?

In Section 1 - we will start with the two articles reporting RCTs on MV.

Section 1 - the measles vaccine (MV)

For the measles vaccine, two articles are being referenced:

BMJ 2010; 341:c6495 Doi:10.1136/bmj.c6495. Link to full text:? https://www.bmj.com/content/341/bmj.c6495.long

and Pediatr Infect Dis J 2003; 22:798–805?|?DOI:?10.1097/01.inf.0000083821.33187.b5 – full text link unfortunately requires a purchase.

BMJ 2010; 341:c6495

was already assessed in Journal Club no. 2. The article reported an RCT in 6648 children. The primary outcome analysis of the RCT was a mortality rate ratio (MRR) of 0.78 (95% confidence interval (CI) 0.59 to 1.05) implying that the study was negative and did not support the existence of NSE of the measles vaccine measured on all-cause mortality. Readers should keep in mind that a trend towards an effect (CI 0.59 to 1.05) does not justify 36 statistical post hoc analyses, with unaccounted multiplicity.

As a random finding a sub-subgroup analysis eventually yielded a significant p-value, which was then reported as part of the conclusion: “Although it showed no overall effect in the main analysis, this trial suggested a non-specific beneficial effect of MV among girls who have not received neonatal vitamin A.

The conclusion stated in BMJ 2010; 341:c6495 is not in accordance with the trial results. ?

Pediatr Infect Dis J 2003; 22:798–805 is the second article that is reference in support of NSE of MV in the Introuction section of Clin Infect Dis. 2018 May 2;66(10):1573-1580.

The title of the article is "Survival of previously measles-vaccinated and measles-unvaccinated children in an emergency situation: an unplanned study."

In the article it is explained that

“During the recent war in Guinea-Bissau, most children fled from the city of Bissau and immunization services in the country broke down for several months. We were performing a trial in which children were randomized at 6 months of age to receive either measles vaccine or inactivated polio vaccine. Because of the war many children did not receive the dose of measles vaccine planned for 9 months of age. We were able to monitor mortality during the war and after.”

And in the Introduction it says:

“Therefore when the war started, there was a group of infants who had been randomized to receive either measles vaccine or IPV at 6 months of age and who did not receive measles vaccine at 9 months of age. This unplanned experiment is the first randomized study of the effect of measles vaccine on mortality.”

The title” …emergency situation: an unplanned study” rhetorically signals innocence ie. a trial is undertaken when an external emergency factor (the war) takes control of the situation, where 433 children of 6-11 months of age had been included and randomized to measles vaccine (MV) or inactivated polio vaccine (IPV).

The way the data analysis was conducted and presented in the article is, however, not innocent, nor unplanned, and a series of methodological issues raise serious concerns ??

First, if the war was jeopardizing the integrity of the trial, it should have been a) terminated prematurely or b) paused and resumed again after the war. Instead the authors proceed with what is referred to as a “natural experiment.” ?A natural experiment is often used in empirical settings of social sciences – and much less frequently in biomedicine. The reason is that within biomedicine, the strongest experimental designs for assessing causal effects, the randomized controlled trial (RCT) supersedes the evidence provided by natural experiments which per design have an inbuilt risk of bias. (Benton J, 2016; DOI:?10.1186/s12966-016-0433-3.)

Second, the authors make a series of choices on how to analyses the data. They chose to follow only the children who did not receive the MV at 9 months (which is standard of care). This introduces healthy vaccinee bias to the data set, as it is well known that it is associated with better health and socio-economy to get a vaccination. This is a well-known fact (BMJ Open. 2019 Sep 18;9(9):e029794. doi: 10.1136/bmjopen-2019-029794 and BMJ. 2017 Aug 30:358:j3862. doi: 10.1136/bmj.j3862) and was well known way before this article was published (BMJ1993;307:1308-11, where the risk of selection bias was described by the same first author). Hence, the authors made a choice to follow the group of children who expectedly would do poorer during follow-up.

Third, the primary outcome of the study “survival,” was never defined in the 8-page long publication with respect to study group or observation time. Why this is – was never described in the Methods section but instead under Results. ?The argumentation goes like this:

"The main analysis was limited to the period when mortality was particularly high and the children would be unlikely to have received measles vaccine elsewhere because the vaccination program had broken down. Because we have previously emphasized that the effect of measles vaccine is stronger for girls [4, 5] we examined the mortality pattern according to sex."

This implies that the main analysis was a) cut up into several 3-4 months periods and b) girls and boys are analyzed separately. This is not only an odd approach, analytical rigor and stringency is completely absent – which is often the case when p-hacking is undertaken. Several small subsamples plus a multiple statistical tests strategy increase the risk of a Type 1 exponentially, and as a consequence the authors risk confirming the research hypothesis even if it is false. ?

Fourth, as also mentioned above the Law on small numbers (Daniel Kahnemann and Amos Tversky (1937-1996)) applies extensively to this article. The Law on small numbers, or hasty generalization, is the incorrect belief that small samples are likely to be highly representative of the populations from which they are drawn. Table 2 is a clipping from the article, the red boxes are inserted by me to highlight the data that the comparisons in the “main analysis” i.e. girls mortality rate ration in June – August 1997 (which is during the war) are based upon.

The results are reported like this:

“The effect of measles vaccine was marked for girls [mortality rate ratio (MR), 0.00; 95% confidence limits, 0.0 to 0.37], whereas there was no difference for boys (MR _ 1.02; 95% confidence limits, 0.25 to 3.88). In a combined analysis controlling for factors that differed between the two groups, the MR for measles-vaccinated children was 0.30 (95% confidence limits, 0.08 to 0.87).”

So, the first analysis is comparing 0 deaths (MV) with 6 deaths (IPV) among girls from June-August 1998. The second analysis is comparing, 4 deaths (MV) with 5 deaths (IPV) among boys from June-August 1998. And the third analysis compares 4 deaths (MV) with 11 deaths (IPV) among boys and girls (after adjusting for differences between groups) – and again this is from June-August 1998.

Keeping in mind that the total number of events in the whole study population and in the full observation period of 12 months is 27 deaths, are readers then convinced that this is a true finding? I am not convinced. Are you? If it was a true finding, it could be expected that other researchers have been able to replicate the finding subsequently. This is an elaborated fishing expedition, designed to yield a significant outcome by applying the flawed methodology described in all of the four objections described above. Notably, the more elaborated and deep diving the fishing expedition of a statistical test strategy becomes the more remote the chances are for replication.

Multiple large RCTs – two of which derive from the very same site in Guinea-Bissau – have been conducted subsequently with the objective to demonstrate the existence of NSE of the MV vaccine.

• eClinicalMedicine. 2022 Jul; 49: 101467 – reporting an RCT including 6636 children: No NSE of MC was found.
• BMJ 2010; 341:c6495 – reporting an RCT including 6648 children: No NSE of MV was found.
• Clin Infect Dis. 2018 May 2;66(10):1573-1580 reporting an RCT including 8309 children: No NSE of MV was found (which hey - yes, you'ved guessed it - is the very same article we are looking into the Introduction statements right now).
• And more recently, an RCT performed in Denmark:
• BMJ 2023;381:e072724 reporting an RCT including 6542 children: No NSE of MV was found.

This list is far from exhaustive, but it does make an independent point of its own that neither of these large scale RCTs were never able to replicate the small numbers of this “natural experiment” undertaken as an unplanned study.

Conclusively, none of the publications on MV that are being used as references for randomized trials in the Introduction section of Clin Infect Dis. 2018 May 2;66(10):1573-1580 support the existence of NSE.

Reference 10 is a null study.

Reference 11 is not a randomized trial.

This concludes section 1 – which covered the MV-studies.? Section 2 is pertaining to trials on BCG.

Section 2 - Bacillus-Calmette-Guerin (BCG)

For the BCG vaccine three articles were referenced in support of NSE measured on all-cause mortality, they are:

The Journal of Infectious Diseases 2011;204:245–52; DOI: 10.1093/infdis/jir240; full text: https://academic.oup.com/jid/article/204/2/245/833883

Clinical Infectious Diseases 2017;65(7):1183–90; DOI: 10.1093/cid/cix525; full text: https://academic.oup.com/cid/article/65/7/1183/4079383?login=false

The Pediatric Infectious Disease Journal 31(3):p 306-308, March 2012. DOI: 10.1097/INF.0b013e3182458289; full text: https://journals.lww.com/pidj/fulltext/2012/03000/small_randomized_trial_among_low_birth_weight.22.aspx

All three trials were previously assessed in the Nonspecific Journal Club no. 3

In the following we will briefly assess these three trials for the result of the primary analysis and the conclusion as it was stated in the individual article (shown in italics)

The Journal of Infectious Diseases 2011;204:245–52

The primary outcome was infant mortality at 12 months of age, and the result was a mortality rate ratio (MRR) of BCG vs. control 0.83 (95% CI 0.63-1.08). Hence, the study refuted the existence of nonspecific vaccine effects of BCG in a trial conducted in 2320 low-birth-weight infants. The conclusion of the study says:

“Though early BCG did not reduce infant mortality significantly, it may have a beneficial effect in the neonatal period.”

The conclusion stated in The Journal of Infectious Diseases 2011;204:245–52 was not in accordance with the trial results – as the primary analysis was infant mortality at 12 months of age.

Clinical Infectious Diseases 2017;65(7):1183–90

The primary outcome was infant mortality at 28 days and the analysis showed that there was no nonspecific effect of the BCG vaccine measured on mortality after 28 days: MRR 0.70 (95% CI 0.47-1.04). The RCT was negative, no difference between groups were found in 4172 low-birth-weight infants. The title of the article is “Early BCG-Denmark and Neonatal Mortality Among Infants Weighing <2500 g: A Randomized Controlled Trial” implying that it is reporting an RCT. But it in fact is also reporting a post-hoc meta-analysis that makes the negative trial results disappear. The conclusion in the article consequently reads as follows:

?“Early administration of BCG-Denmark in LW infants is associated with major reductions in mortality rate.”

The conclusion stated in Clinical Infectious Diseases 2017;65(7):1183–90 was highly misleading and completely disregarding the results of the RCT.

The Pediatric Infectious Disease Journal 31(3):p 306-308

The primary outcome of this RCT was overall infant mortality (12 months) of BCG administered at birth in 105 low-birth weight infants. The result of the primary analysis was that there was no nonspecific effect of the BCG vaccine measured on mortality at 12 months of age: MRR was 0.41 (95% CI 0.14–1.18). ?Irrespective of this very clear nonsignificant result of the primary analysis, the authors conclude on a post-hoc subgroup analysis. The conclusion of the article says:

“The small trial of BCG at the first health center contact supported the beneficial effect of BCG administration in the first months of life among LBW children.”

The conclusion stated in The Pediatric Infectious Disease Journal 31(3):p 306-308 was not in accordance the result of the main outcome of the study.

Conclusively, none of the three publications on BCG that are being used as references for randomized trials in the Introduction section of Clin Infect Dis. 2018 May 2;66(10):1573-1580 support the existence of NSE of the BCG vaccine.

Reference 12 is a null study.

Reference 13 is a null study.

Reference 14 is a null study.

This concludes section 2. Section 3 will be on oral polio vaccine (OPV).

Section 3 - oral polio vaccine (OPV)

The final publication that is used as a reference in support of the existence of NSE is Clin Infect Dis 2015; 61:1504–11. Doi: 10.1093/cid/civ617. Full text link is available at this link: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4614411/

In the Nonspecific Vaccine Effects Journal Club we have not previously assessed articles reporting trials on OPV. But there is a first time for everything – so here it comes.

Clin Infect Dis 2015; 61:1504–11

The title of the article is “The Effect of Oral Polio Vaccine at Birth on Infant Mortality: A Randomized Trial” and reports an RCT conducted in 7012 infants in Guinea-Bissau. The main outcome was overall mortality (excluding accidents) within the first 12 months of life. The result of the primary analysis was a Hazard Ratio (HR) of 0.83 (95% CI 0.61-1.13), meaning that no nonspecific effect of the OPV was found. See table 2 for reference (a clipping from the article, the red box is inserted by me to highlight the main outcome).

The article further reports on an extensive series of additional statistical tests, all secondary analyses, 12 of them are presented in table 3 of the article (a clipping from the article):

Consequently, the conclusion of the article abstract disregards the main outcome and focuses on a post-hoc subgroup analysis:

“OPV0 may be associated with nonspecific protection against infectious disease mortality, particularly when given early in life. There are reasons to monitor mortality when OPV is being phased out.”

The conclusion of the article is not in accordance with the result of the main outcome of the study.

For Clin Infect Dis. 2018 May 2;66(10):1573-1580 to refer to this study as supportive of NSE of OPV is therefore a falsity as Clin Infect Dis 2015; 61:1504–11 is a null study.

We have now worked our way through all six references that were used by the authors in support of the existence of nonspecific vaccine effects of MV, BCG, and OPV demonstrated in randomized clinical trials. This concludes section 3.

Fake it till you make it

We have come a long way. See the table below for an overview of the six articles that were assessed herein. It includes information on the tested vaccine, the number of participants, the definition of the primary outcome, the author conclusion (copy-pasted from the individual article and shown in italics) and concordance (or lack thereof) between the main result and the conclusion stated in the articles.

This table shows the five trials and the “natural experiment” that were used as references in a peer-reviewed article’s Introduction section as supportive evidence of nonspecific effects of vaccines demonstrated in RCTs. One is not an RCT, and neither of the articles reporting RCTs, supported the existence of nonspecific effects of MV, BCG, or OPV in the primary analyses.

These are the facts that remain after smokescreens from multiplicity in statistical testing of subgroups and sub-subgroups, post-hoc analyses, and p-hacking fishing expeditions were rolled out in the individual articles.

Altogether this implies that the Introduction of Clin Infect Dis. 2018 May 2;66(10):1573-1580 contains a falsity, because the randomized trials that are referenced do in fact not “indicate that the MV, BCG, or OPV may have beneficial nonspecific effects by lowering all-cause mortality due to non-vaccine-target infections.”

In every one of the articles reporting RCTs, the conclusions claim that the results of the trial reported point to the existence of nonspecific vaccine effects, and by doing so the narrative is taken under control. Similarly, the same tactics are employed in the Introduction of Clin Infect Dis. 2018 May 2;66(10):1573-1580 – taking control of the narrative – or in plain English: Fake it till you make it.

From a communication perspective this may be considered a smart move, it is however, strongly misleading. Research communication can be angled, but it must stay compliant to the results.

Furthermore, it is disrespectful to the study participants – and in this case their parents. And finally as a reader of scientific literature it is discouraging that the editors are not scrutinizing this element of the publication process with stronger efforts. I would not blame reviewers as they are after all working for free using their spare time. Vetting references ought to be taken seriously by editors, journals, and publishers.

The assessment herein demonstrates just how misinforming publications end up being. That is not in the best interest of these stakeholders, researchers, let alone the study participants.

Moreover this problem is likely to be found in many other therapeutic areas of biomedical research. Now you know what to look for.

Replication crisis

We face a manifest replication crisis in biomedical science.

The one good piece of news that can be drawn from this assessment of six articles claiming to support nonspecific vaccine effects in >20.000 infants altogether and across three different vaccines, is that they have in fact managed to replicate the null hypothesis, and thereby they have rejected the hypothesis that vaccines have nonspecific effects.

This concludes the fourth issue of the #NSEvaccJournalClub

Stay tuned for more

For previous issues:

Issue no. 1: https://www.dhirubhai.net/pulse/nonspecific-vaccine-effects-journal-club-1-charlotte-str%2525C3%2525B8m%3FtrackingId=jnvz9LqhSuyzZnzLzYj5hg%253D%253D/?trackingId=jnvz9LqhSuyzZnzLzYj5hg%3D%3D

Issue no. 2: https://www.dhirubhai.net/pulse/nonspecific-vaccine-effects-journal-club-2-charlotte-str%2525C3%2525B8m%3FtrackingId=5Aq78EyJRcKzqHr%252F5oiZWg%253D%253D/?trackingId=5Aq78EyJRcKzqHr%2F5oiZWg%3D%3D

Issue no. 3. https://www.dhirubhai.net/pulse/nonspecific-vaccine-effects-journal-club-3-charlotte-str%2525C3%2525B8m%3FtrackingId=wxbZter7U%252FXV%252F%252BxVkVKfkw%253D%253D/?trackingId=wxbZter7U%2FXV%2F%2BxVkVKfkw%3D%3D