Can neuroimaging studies be replicated?
The short answer: yes they can. But you will need thousands of participants to get reliable results. And because such large studies are hardly available, previously reported results are unlikely to be replicated.
Shu Liu investigated the replicability of neuroimaging studies by making use of the largest neuroimaging study available (full text here: https://rdcu.be/dfpLS). The UK Biobank includes neuroimaging data from over 37,000 individuals. We then split the data into a discovery sample and a replication sample, and investigated how large the discovery sample needs to be in order to replicate the results in 75% of replication samples. The results confirm the recent expectation by Marek and colleagues (https://www.nature.com/articles/s41586-022-04492-9): it takes thousands of participants to identify which brain regions are associated with person characteristics such as personality and intelligence. Though it only takes 300 participants to identify the brain regions associated with aging.
The reason for poor replicability is the weak association between person characteristics and particular brain structures and their functioning. The required number of participants can almost perfectly be predicted by the strength of the association (see Figure).
Does this apply to all neuroimaging studies? Luckily not. These results speak to studies that aim to identify which brain regions are related to person-to-person variability in cognition, personality and behavior. The link between individual differences in cognition and all brain regions together is considerably stronger (https://www.nature.com/articles/s41586-023-05745-x). And machine learning analysis of neuroimaging data can even identify individual patients that will or will not respond to antidepressant treatment (https://www.nature.com/articles/s41398-021-01286-x).
So differences in brain structure and function are definitely related to differences in person characteristics. But you can forget about identifying THE brain region for intelligence – it takes your entire brain!
?
?
Thanks Guido! That looks like a robust fit :-)
Very interesting and thorough analysis indeed. Typically sample size is inversely proportional to the square of the sample size and that is pretty much what you find. I wonder though how the analysis (calculating these effect sizes in an embedding space, say using a future "foundational model" in neuroimaging) might affect these calculations?