Journal Club: The show must go on = Send more money

Nonspecific Vaccine Effects Journal Club no. 7

In the publication “CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials.” BMJ 2010;340:c869, doi: 10.1136/bmj.c869 it says in the very top of the article (text in italics represents quotations from the articles in mention)

“Overwhelming evidence shows the quality of reporting of randomised controlled trials (RCTs) is not optimal. Without transparent reporting, readers cannot judge the reliability and validity of trial findings nor extract information for systematic reviews.”

The NSEvacc JC no. 7 will follow the reporting of a large RCT (NCT01694108 ): Bacille Calmette Guérin Immunisation at Birth and Childhood Morbidity in Danish Children.

In 2017 the article “BCG vaccination at birth and early childhood hospitalisation: a randomised clinical multicentre trial.” Arch Dis Child. Mar 2017;102(3):224-231. doi:10.1136/archdischild-2016-310760 was published. This study was reporting the primary outcome of the RCT.

4262 infants were allocated to BCG or no intervention. The result of the primary outcome analysis was very clear:

“There was no difference in risk of hospitalisation up to 15?months of age; 2129 children randomised to BCG experienced 1047 hospitalisations with a mean of 0.49 hospitalisation per child compared with 1003 hospitalisations among 2133 control children (mean 0.47), resulting in a HR comparing BCG versus no BCG of 1.05 (95% CI 0.93 to 1.18) (intention-to-treat analysis).”

From this follows a conclusion in the article that is in accordance with the results of the RCT: “BCG vaccination at birth did not reduce the risk of hospitalisation for somatic acquired disease until 15?months of age in this Danish study population.”

All is good… or is it?

Table 3 demonstrates why all is not good. It displays eight secondary statistical analyses that were performed to test if any of the listed items influenced the main result.

These eight secondary analyses represent only a fraction of the list of planned secondary analyses according to the protocol (NCT01694108 ): 22 secondary outcomes in total were planned – and yet even more subgroup analyses - it appears (in fact you just wait and see...). In this current article eight of them are presented – and the concerning thing is that none of the secondary analyses are adjusted for multiplicity.

The reporting of the RCT in this article seems to pay very little attention to the CONSORT 2010 updated guidelines publication (BMJ 2010;340:c869) that states:

“Because of the high risk for spurious findings, subgroup analyses are often discouraged.[14, 185] Post hoc subgroup comparisons (analyses done after looking at the data) are especially likely not to be confirmed by further studies. Such analyses do not have great credibility.”

Further,

“Authors should clarify the choice of variables that were adjusted for, indicate how continuous variables were handled, and specify whether the analysis was planned or suggested by the data.[188]”

The point is that irrespective of whether subgroup analyses in an RCT are prespecifed or not – the reporting must clearly describe how the analyses are adjusted for the multiple statistical tests that are employed. And with an increasing number of statistical tests, the credibility of the reported findings decrease steadily.

In the present study the type 1 error is defined by k^nd test alpha: 1 – 0.95^k. This means that at the 8th analysis alpha the type 1 error is 33,7%. It seems like there is a pretty high risk that the singled out significant finding is spurious – pure chance. Irrespective of this evident flaw to the statistical analysis applied to the data, it is stated in the Result section (bolded text by me):

“No statistically significant effect modification by caesarean section, administration of antibiotics to the mother during vaginal delivery, birth weight <2500?g, ≥1 sibling, atopic disposition, age at vaccination ≤2?days, maternal BCG vaccination and BCG vaccine batch was observed (table 3). However, a tendency towards protection against hospitalisation was observed among BCG-vaccinated children whose mother was also BCG vaccinated.”

And it carries on in the Discussion:

“In agreement with this, in a preplanned secondary analysis of hospitalisations for infection within the present trial, a significant beneficial effect of BCG among children of BCG-vaccinated mothers was observed (personal communication). If maternal exposure to BCG or mycobacteria is essential for the development of beneficial non-specific effects of BCG, this may explain the beneficial effect of BCG observed in low-income countries but not overall in the present study.”

But why waste more time (and money) chasing this alleged finding…? This was the 8th statistical tests or more and keeping in mind that according to CONSORT 2010 “An analytical strategy with multiplicity causes high risk of spurious findings, i.e., findings that are most likely chance findings” it makes little or in fact it makes no sense at all to pursue this finding in the assumption that it will show that nonspecific effects of the BCG vaccine exist.

Let’s briefly remind ourselves what the primary analysis showed:

Figure 2 is copy-pasted from the article, illustrating graphically the primary outcome. It is pretty obvious that there is absolutely no difference between the two groups (red curve – control; blue curve BCG). Maternal BCG vaccination is hardly a major game changer. Unless of course THE SHOW MUST GO ON ??

And it appears to be the case. The nonspecific vaccine nonsense for sure goes on.

In 2019 the article “BCG Vaccination at Birth and Rate of Hospitalization for Infection Until 15 Months of Age in Danish Children: A Randomized Clinical Multicenter Trial.” J Pediatric Infect Dis Soc. Jul 1 2019;8(3):213-220 doi:10.1093/jpids/piy029 was published.

In the Introduction it says:

“Because all-cause hospitalization might not be sufficiently specific to detect a potential beneficial protection of the BCG vaccine against infections, a secondary outcome was hospitalization for infection.”

So this is a secondary outcome (one of 22 secondary outcomes!!) and wauw – it appears to deserver its very own publication. Something important must be reported then… ? I don’t think so. It seems odd that a secondary outcome “hospitalization for infection” is reported separately from the primary outcome “all-cause hospitalization.” Especially, because there is a great deal of repetition in this paper compared to the first paper that was assessed above. Anyway – let us look at the results of this secondary outcome analysis:

Table 1 is copy-pasted from from the article doi:10.1093/jpids/piy029 .

… what does the table mean? It means that there is no nonspecific effect of the BCG vaccine on hospitalisations for infections.

And it gets even better – take a close look at figure 1 (copy-pasted from the same article as mentioned above):

The figure shows that there is no difference whatsoever between the red curve – controls and the blue curve – BCG. And by the way it has a remarkable resemblance with figure 2 of the previous article, “BCG vaccination at birth and early childhood hospitalisation: a randomised clinical multicentre trial.” Arch Dis Child. Mar 2017;102(3):224-231. doi:10.1136/archdischild-2016-310760 , which reported a negative primary outcome?analysis. ??

This current article “BCG Vaccination at Birth and Rate of Hospitalization for Infection Until 15 Months of Age in Danish Children: A Randomized Clinical Multicenter Trial.” J Pediatric Infect Dis Soc. Jul 1 2019;8(3):213-220. doi:10.1093/jpids/piy029 to a large extent leaves an overwhelming – but false – impression that this is the report of the primary analysis; it pretends to be the “mothership” of publications on the BCG study – the key article communicating the most important results from the RCT. The truth is that it is just reporting a secondary analysis and a hysterical amount of subgroup analyses. But why this false pretence?

The explanation for this puzzling way of reporting an RCT is disclosed in the section Effect Modification under Results:

“Among prespecified potential effect modifiers, maternal BCG vaccination significantly modified the effect of the BCG vaccine; the HRs for BCG-vaccinated children were 0.65 (95% CI, 0.45–0.94) for those whose mother was BCG vaccinated and 1.10 (95% CI, 0.93–1.29) for those whose mother was not BCG vaccinated (P?=?.01, test of no interaction [ITT-analysis])”

In the Discussion it says:

“Although we found no overall effect, the results of 3 prespecified subgroup analyses indicated that exploring whether the effect of BCG vaccination is modified by gestational age, cesarean delivery, or maternal BCG might be worthwhile.”

So for the record, let’s just remind ourselves here that testing for whether maternal BCG had any influence on the result was a secondary analysis in a subgroup. A test that as shown above had a 33,7% chance of being false. This risk of a chance finding is growing exponentially in this current article. ?Notably, there is no mentioning of adjustment for multiplicity – and no mentioning of how many analyses were performed before the data caved under the torture and confessed to a single significant result.

Table 2 (copy pasted from the article) says it all. Go ahead and knock yourself out in Table 2 – for sure the authors have! It seems like there is no limit to how many statistical tests they have performed here, not only in subgroups, but also in subgroups of subgroups of subgroups. Keep counting….??

For your convenience and in case you cannot be bothered to do the calculation yourself – I leave a table herein where the type 1 error (alpha) has been calculated according to the number of statistical tests:

What do you think, is it sensible to conduct … I count 100+ statistical tests?

I think it is ridiculous that this paper was published in the first place as it reflects a complete lack of respect for the CONSORT guidelines. This paper is a strong example of untransparent reporting of an RCT where “readers cannot judge the reliability and validity of trial findings” (BMJ 2010;340:c869, doi: 10.1136/bmj.c869).

The paper pretends to communicate something important - where in fact it is a massive p-hacking expedition. Reviewers and the editor of the journal should have stopped it at submission. ?

These researchers certainly have no fear of rejecting a true null hypothesis. As long as the show can go on they appear to be happy to do as many tests as it takes basically to confirm their beliefs. Mark – the word: belief. That’s what it is – it has nothing to do with science.

Below you will find an overview of the articles that were published based on the Danish BCG trial (disclaimer – may not be an exhaustive list). A total of 20 publications: one reports the primary outcome, three report safety, four report methodology, protocol, 1 is a corrigendum and eleven articles report secondary analyses and heaps of subgroup analyses. Cutting to the chase: The correct term to describe the conduct that is so vividly displayed here is p-hacking.

The learning from this Journal Club is that some researchers will not stand down to an unlimited number of statistical tests as long as it will keep them in business by yielding just a single significant result. Herein it is obvious that the result of the primary outcome analysis is ignored and an agenda aiming to confirm the hypothesis is put in operation. There is no point is pursuing maternal BCG – it is an explorative and moreover random finding that occurred by chance and therefore it has no scientific bearing.

But why do researchers, from whom you would/ could / should (!) expect a higher level of professionalism and integrity, embark on p-hacking manoeuvres like the ones displayed above?

Well, the answer to that question is evident from the conclusion of the article (bolded text by me):

“The results of our trial rejected the hypothesis that BCG vaccination within 7?days of birth reduced the rate of hospitalization for infection until 15?months of age among Danish children. Our findings call for further studies of the role of gestational age, cesarean delivery, and maternal BCG vaccination in the nonspecific effects of vaccines.”

In other words: ??“Send more money! ?? Help us keep our scientifically inferior hypothesis alive, so that we can stay in business.” ??

We started out with an ABBA song and believe it or not there is another ABBA song released in 2021 (a great comeback single by the way) that you can imagine True Believers are singing to ALL funding bodies now that it is crystal clear from the data that there is absolutely no nonspecific effect of the BCG vaccine: Don’t shut me down ?? ?? ??

You may consider listening to that song too ?? as I don’t want to leave you disillusioned and sad about the (lack of) integrity of the research carried out in fine research institutions, such as Research Center for Vitamins and Vaccines (CVIVA)… Look it up! The center no longer exists. I wonder why ??. Perhaps someone did shut it down?

Nonetheless, the NSE vaccine show goes on and the emperor continues to walk the streets wearing no clothes at all.

This concludes the seventh issue of #NSEvaccJournalClub.

Stay tuned for more

Previous issues of the NSEvacc Journal Club:

Issue no. 6: Special X-mas edition Triangulation of beliefs

Issue no. 5 Misinformation during and after the Covid-pandemic

Issue no. 4 Take control of the narrative – a euphemism of ‘fake it till you make it’

Issue no. 3 Can post-hoc and subgroup analyses make negative trial results go away

Issue no 2 How many RCTs does it take to cast aside a scientific hypothesis?

Issue no. 1 Are medical textbooks subject to AI?

Publications based on the RCT: NCT01694108

Primary outcome:

BCG vaccination at birth and early childhood hospitalisation: a randomised clinical multicentre trial. Arch Dis Child. Mar 2017;102(3):224-231. doi:10.1136/archdischild-2016-310760 (this article was assessed above: no nonspecific effect of BCG)

Secondary outcomes and heaps of subgroup analyses that per definition are exploratory:

Effects of Bacillus Calmette-Guérin (BCG) vaccination at birth on T and B lymphocyte subsets: Results from a clinical randomized trial. Sci Rep. Sep 29 2017;7(1):12398. doi:10.1038/s41598-017-11601-6

Bacillus Calmette-Guérin vaccination, thymic size, and thymic output in healthy newborns. Pediatr Res. Jun 2017;81(6):873-880. doi:10.1038/pr.2017.27

Bacille Calmette-Guérin vaccination at birth and differential white blood cell count in infancy. A randomised clinical trial. Vaccine. Mar 4 2020;38(11):2449-2455. doi:10.1016/j.vaccine.2020.02.006

Nonspecific effect of BCG vaccination at birth on early childhood infections: a randomized, clinical multicenter trial. Pediatr Res. Nov 2016;80(5):681-685. doi:10.1038/pr.2016.142

Bacillus Calmette-Guérin vaccination at birth and in vitro cytokine responses to non-specific stimulation. A randomized clinical trial. Eur J Clin Microbiol Infect Dis. Jan 2018;37(1):29-41. doi:10.1007/s10096-017-3097-2

Bacille Calmette-Guérin (BCG) vaccination at birth and antibody responses to childhood vaccines. A randomised clinical trial. Vaccine. Apr 11 2017;35(16):2084-2091. doi:10.1016/j.vaccine.2017.02.048

BCG Vaccination at Birth and Rate of Hospitalization for Infection Until 15 Months of Age in Danish Children: A Randomized Clinical Multicenter Trial. J Pediatric Infect Dis Soc. Jul 1 2019;8(3):213-220. doi:10.1093/jpids/piy029 (this article was assessed above and found no nonspecific effect of BCG)

Neonatal BCG has no effect on allergic sensitization and suspected food allergy until 13?months. Pediatr Allergy Immunol. Sep 2017;28(6):588-596. doi:10.1111/pai.12748

Neonatal BCG vaccination and atopic dermatitis before 13 months of age: A randomized clinical trial. Allergy. Feb 2018;73(2):498-504. doi:10.1111/all.13314

Neonatal BCG vaccination has no effect on recurrent wheeze in the first year of life: A?randomized clinical trial. J Allergy Clin Immunol. Dec 2017;140(6):1616-1621.e3. doi:10.1016/j.jaci.2016.12.990

Bacillus Calmette-Guérin vaccination at birth: Effects on infant growth. A randomized clinical trial. Early Hum Dev. Sep 2016;100:49-54. doi:10.1016/j.earlhumdev.2016.05.015

Safety reporting:

The association between Bacillus Calmette-Guérin vaccination (1331 SSI) skin reaction and subsequent scar development in infants. BMC Infect Dis. Aug 3 2017;17(1):540. doi:10.1186/s12879-017-2641-0

Lack of a Negative Effect of BCG-Vaccination on Child Psychomotor Development: Results from the Danish Calmette Study - A Randomised Clinical Trial. PLoS One. 2016;11(4):e0154541. doi:10.1371/journal.pone.0154541

Adverse reactions to the Bacillus Calmette-Guérin (BCG) vaccine in new-born infants-an evaluation of the Danish strain 1331 SSI in a randomized clinical trial. Vaccine. May 11 2016;34(22):2477-82. doi:10.1016/j.vaccine.2016.03.100

Methodology:

Recruiting to Clinical Trials on the Telephone - a randomized controlled trial. Trials. Nov 21 2016;17(1):552. doi:10.1186/s13063-016-1680-y

Mothers' informational needs when deciding to have their newborn infant vaccinated with BCG. A Mixed-methods design. Scand J Caring Sci. Sep 2018;32(3):1118-1126. doi:10.1111/scs.12557

Protocol:

[The bacille Calmette-Guérin vaccine may reduce the general child morbidity]. Ugeskr Laeger. Apr 28 2014;176(9)

Bacillus Calmette-Guérin immunisation at birth and morbidity among Danish children: A prospective, randomised, clinical trial. Contemp Clin Trials. May 2015;42:213-8. doi:10.1016/j.cct.2015.04.006

Corrigendum:

Corrigendum: BCG Vaccination at Birth and Rate of Hospitalization for Infection Until 15 Months of Age in Danish Children: A Randomized Clinical Multicenter Trial. J Pediatric Infect Dis Soc. Feb 28 2020;9(1):106. doi:10.1093/jpids/piaa003