A better how: notes on developmental meta-research

Edited by Patrick Forscher (Busara) and Mario Schmidt (Busara & Max Planck Institute for Social Anthropology)

Introduction to ?? ???????????? ??????: ?????????? ???? ?????????????????????????? ????????-????????????????

The past decade has raised concerns about how research is conducted, evaluated, and disseminated. Fuelled by the replication crisis in psychology and allied disciplines, these concerns have spawned a movement that unites scholars from across the globe: the open science movement.

The movement has produced and popularized a huge array of innovations to enhance the replicability of research and has even caught the notice of several large institutional actors.

For example, the Biden administration declared 2023 the ‘year of open science’ in the United States. Internationally, UNESCO has collated and issued comprehensive recommendations for how member states can use and incorporate open science into policymaking.

This interest has set in motion a broader movement and a dedicated academic subfield to improve how research is used in society.

This movement, the meta-research movement, is the subject of this book.?

Meta-research, sometimes called meta-science, is research focused on investigating the research process itself, often aiming to make concrete improvements.

These improvements have, to date, primarily focused on improving the fundamental soundness of academic research. Due to its roots in the behavioral sciences, these improvements are also often behaviorally informed.

For example, the meta-research innovation called “preregistration” involves a precommitment to a particular way of analyzing data before seeing it.

This innovation is designed to reduce the risks that the analyst intentionally or unintentionally changes the analysis plan after seeing how the data turns out to suit their preferred interpretation.

Although this innovation focuses on the basic soundness of research, the improvements sought by meta-researchers can, in principle, involve anything – including the problem areas that are the traditional focus of the global development community, such as north-south power differences, building healthy research ecosystems, and the treatment of participants or beneficiaries.

Our team has spent the past two years investigating the potential intersections between meta-research and global development. We discovered that many communities within global development have already been concerned about how to improve how research is conducted and used without necessarily realizing that they have a common cause with other, similar communities. Likewise, many meta-researchers would benefit from learning how research is used outside academic contexts.

We believe this intersection between meta-research and global development is a fruitful one and has the potential to constitute an entirely new subdiscipline, which we call developmental meta-research.

This subdiscipline turns meta-research’s behaviorally-informed critical lens toward topics that have traditionally been the focus of global development practitioners.

We believe the resulting subdiscipline can fruitfully inform intersections between development practitioners and meta-researchers and mobilize a new community around improving how research is done in development.?

This volume brings together various contributors from across development and meta-research.

These contributors span different sectors, institutions, countries, and problem areas. What unites them is a shared focus on improvement: in the distribution of power, how evidence is used to inform policy, and regarding the overall conduct of research.

We have categorized the contributions into three clusters, each focusing on a different dimension of what it means to strive for a ‘better how’ by considering how research is produced, implemented, and disseminated.

Contributions to our first part, How to improve collaboration across the Global North-South divide, discuss ways local research infrastructures can be strengthened through collaborations across the Global North-South divide, and critically interrogate the potentials and pitfalls of science across economic, political, and linguistic barriers.

Contributions in this section ask uncomfortable questions about who benefits from collaborations across these barriers and how the research capacity in low- and middle-income countries can improve through such collaborations.

In our first contribution to this section, Advantages and challenges of Global North/Global South research collaborations: an emphasis on sub-Saharan African research infrastructure, Dana Basnight-Brown, PhD, MS, MA uses her own experiences of working across the Global North/Global South divide to offer advice on how to avoid misunderstandings and mismatched priorities that often emerge in collaborations between high-income countries and low-and middle-income countries.

Shifting the focus away from Africa, our next contribution – Amirun Haqqim and Miguel Silan ’s ‘Should you study abroad?’ The mechanisms and utility of educational emigration from Southeast Asia – sheds light upon the political-economic effects of educational migration on local research infrastructures in Southeast Asia.

Complexifying one-sided answers to the consequences of educational migration, the authors argue for a more nuanced view on the topic, highlighting both potential benefits and dangers for local, underfinanced research infrastructures.

The successive contribution Building local networks for open science: a case study of the Chinese Open Science Network (COSN), was written by a group of authors from the Chinese Open Science Network ( Zhiqi X. , Yue Wang, Liangyou Zhang, Wenqianglong Li , Hu Chuan-Peng , Chenghao Zhou and Xi Chen). It shares crucial insights on how comparatively cost-effective ways could improve local research infrastructures if enough early career researchers team up together and embrace the principles of open science.

Among the initiatives discussed are translating important English-speaking articles into Chinese, tutorials, conference meetings, and increasing the use of social media platforms.

How such efforts toward a more just and open science could look if implemented transnationally is explored in the contribution by Alma Jeftic , Marc Yancy Lucas, Nadia S. Corral-Frias Ph.D. , and Flavio Azevedo . Their paper Bridging the majority and minority worlds: liminal researchers as catalysts for inclusive open and big team science introduces two collaborative efforts – FORRT (The Framework for Open and Reproducible Research Training) and ABRIR (Advancing Open and Big-Team Reproducible Science through Increased Representation) – and illustrates the pivotal, yet underestimated, role of what they call ‘liminal researchers’ who have intimate knowledge of different research environment in the Global South and North.

Early career researchers, however, also depend on more senior colleagues for advice, mentorship, and guidance on navigating science and its relationship to society, development, and their lives.

Sharing her own experience with founding and leading the African-led research mentorship organization Eider Africa in her contribution Contextually grounded research in postgraduate research training in Africa: why and how Aurelia Munene calls for more contextually grounded research that dismantles colonial knowledge hierarchies and helps African scholars to move from the margins to the center of global knowledge production.

Along similar lines but specifically focusing on clinical psychology, Helen Niemeyer and Louis Schiekiera ’s paper How inclusive and equitable is research in clinical psychology that focuses on the Global South? criticizes that clinical studies in the Global South are rarely informed by local knowledge and often only implement superficial adaptations, such as changing names in narratives.

Considering the potentially dramatic consequences of mental health studies, Niemeyer and Schiekiera suggest integrating local researchers into teams and taking their knowledge seriously.

The last contribution of our first part, Maya Ranganath ’s Collaborating to support a more inclusive evidence ecosystem, introduces the Center for Effective Global Action’s initiative Collaboration for Inclusive Development Research that aims to produce knowledge about the inclusion of African scholars in global development along different stages with the ultimate aim to improve research, impact the evidence-to-policy pipeline, and design actionable guidelines to ultimately transform the development space into a more diverse, equitable, and inclusive ecosystem.?

Our second part How to improve the evidence-to-policy pipeline zooms in on the question of how we can ensure that the evidence that informs policy is sound and inclusive, that it addresses issues of concern to the ultimate users of evidence, and that the ecosystem for synthesizing evidence into insights is healthy and robust.

In their contribution to the edited volume Behavioral public policy for global challenges, Dr Sanchayan Banerjee and Matteo Maria Galizzi focus on how behavioral interventions can be scaled up to target the global problems of our time. Suggesting three different ways to accomplish this – expanding the toolkit of behavioral public policy, assessing the heterogeneity in treatment effects of behavioral interventions, and implementing systematic, reproducible, and transparent multi-country experimentations – they urge all of us not to lose sight of the bigger picture.

Focusing on a significant ‘big team science’ effort across several African countries which he led, our next author, Adeyemi Adetula , centers his contribution Building research capacity in Africa via big team science: challenges and lessons learned from the ManyLabs Africa initiative around the challenges his team faced when trying to boost local interest and skills in open science practices and when generalizing from sub-Saharan African scholarship to the West rather than the reverse.

Among these challenges, Adetula highlights political-economic incentives for African researchers to focus on their careers and infrastructural problems, such as poor internet access, insufficiently staffed labs, and expensive ethics approvals.

Tommie Yeo Thompson , Winnie Mughogho , and Anisha Singh ’s The gender data gap in development policy research zooms in on what we still too often forget: changing the behavior of men and women does not just require different interventions.

We must also be careful to do justice to the gendered dimension of human behavior when we choose our sample and measurement tools.

Enlarging our toolbox is something our next two contributions suggest as well. In their paper Assessing deliberative polling methodology for decolonizing and indigenizing research, Dennis Chirawurah and Niagia F. Santuah discuss the decolonizing potential of the Deliberative Polling Method to open up the space for local and contextual knowledge to emerge.

Starting from the premise that whenever policy choices have a consequence for local communities, these communities should be consulted on their opinion about it, the Deliberative Polling Method urges researchers and those implementing policies to embrace the tacit and explicit knowledge of indigenous people if they want interventions that have a lasting impact.

Evaluating interventions: a practical primer for specifying the smallest effect size of interest, written by a large team of authors spearheaded by Hannah Peetz and Maximilian A. Primbs, urges us to move beyond conventionally determined ways of interpreting the effect sizes used to measure the results of quantitatively assessed interventions, such as Cohen’s benchmarks.

As an alternative, they consider a different kind of benchmark that better allows for considering contextual and other kinds of knowledge: the smallest effect size that important stakeholders, such as researchers, consider to be meaningful in a specific context.

Introducing three ways to determine the smallest effect size of interest – person as effect size, cost-benefit analyses, and the minimal important difference – they suggest moving beyond mindless benchmarks when judging the efficacy of interventions.

The last contribution of our second section, Anushka Ghosh ’s Utility of meta-research for Global South policymaking: a reflection on education research, scrutinizes the potential of meta-research to influence decisions made by researchers and policymakers in the wider field of education.

Reflecting upon her own experience as a researcher from the Global South, she urges us to make use of meta-research not only to improve research and policies but also as a tool to open up space for critical distance and reflection upon political hierarchies and colonial legacies still impacting how research and policy is done.?

Papers in our third and last section How to improve how we conduct research take a closer look at how we can improve research epistemologically, ethically, and methodologically.

In his contribution, The democratizing effects of doubting, Adam Moe Fejerskov shares his uneasiness about global development’s fascination with and focus on certainty, suggesting that we would all benefit from embracing a more humble attitude that leaves space for doubt.

One potential way to increase one’s embrace of doubt might be to engage in more intimate ethnographic methods, a form of research advocated for by Ben Jones and Ben Eyre together with their team of ‘citizen ethnographers’ ( Sharon Enon Acio , Dorah Adoch, Vicky Alum, Joel Ekaun Hannington, Jimmy Ezra Okello, Robert Oluka, James Opolo, and Ann Gumkit Parlaker).

Their paper ‘Here, my degree does not matter, you are the teacher’: ethnography, citizen ethnography, and researching research on global development argues for the need to train and mentor local citizens in ethnographic methods to increase the context-sensitivity of development projects.

Mario Schmidt ’s contribution Do the randomized know they are randomized? A critique of the turn towards randomization in development cooperation picks up on Fejerskov’s embrace of doubt and Eyre’s call for more ethnography by highlighting respondents’ (mis)understanding of the randomization process during an unconditional cash transfer program in western Kenya with ethnographic detail.

Also, taking a Kenyan cash transfer program as its empirical starting point, our next contribution Manage relationships when starting and ending research with human participants by Joel Wambua , Anisha Singh , Kelvin K. , Irene Ngina , and Patrick Forscher , discusses ways to improve the relationship between researcher and researched.

The authors particularly focus on the moments of community entry and exit, and caution us that research projects characterized by large gaps between the knowledge and expectations of researchers and those of the researched are particularly susceptible to cause harm if approached naively.

Jason M. Chin’s paper Why Applied Psychologists Should Consider Their Work’s value-laden context also focuses on a specific case, namely a consequential law and psychology study that occurred in the context of an inquiry by the Australian government called the Royal Commission into Institutional Responses to Child Sexual Abuse (RCIRCSA).

Chin questions the neglect of non-epistemic values coupled with an overreliance on scientific conventions when evaluating interventions, such as specific standards of evidence sufficiency (for instance, an alpha level of 0.05), irrespective of the value-laden context these interventions appear in.

Our next two authors, Dr Symen A. Brouwers and Floriza Freire Gennari, DrPH , also argue for more context-sensitivity in global development.

In their contribution Cultural context and ecological validity in global development research, they argue that we should never naively assume that an initiative that worked in one place will also work elsewhere even if the two areas are similar. Rather, they suggest a critical analysis of context to optimize ecological validity through a mixed-methods approach combining qualitative and quantitative approaches.

Our last contribution, Joel Wambua ’s Consent, open-ended questions, and feedback loops: empirical insights into research ethics in the Global South, shares the first results of an experiment in Nairobi testing the effects of minor improvements to research protocols – particularly focusing on changes to consent procedures, the inclusion of open-ended questions, and different types of feedback sent to the participants after the study – and how these change participants’ perceptions of ethical practice.

The experiment conclusively finds no change as a result of any of the interventions, suggesting that, in this context at least, small changes are insufficient to impact participant experience – even though participants consistently ask for improvements in consent, the inclusion of open-ended questions, and feedback, including in this experiment.

Wambua urges us to focus more directly on what participants mean when they ask for improvements to their research experience, as small tweaks to the research experience may be insufficient to fully respond to these requests?

If there is one thing we wish our readers to learn from this edited volume, it is that meta-research and global development are neither opposed nor mere add-ons to each other. Instead, meta-research and global development are integral to one another. It is this integration that forms the developmental meta-research subfield that we have identified over the past two years and in the preparation of this volume.

Today, we not only witness increasing calls for evidence in the development cooperation sector but also observe the production of thousands of scientific studies evaluating interventions. We can thus no longer afford the luxury to either ‘do’ meta-research or ‘do’ development.

If we do not reflect on how we construct robust and supportive research ecosystems that support researchers in all parts of the world, build the tools and workflows so that those researchers can plan and implement sound and useful research, and create the infrastructure and partnerships to ensure that research can be used in the places where it matters the result will be a development sector that relies on hearsay and conventions.

Quality of research, in other words, is not only a necessary part of our way to accomplish development goals but, to a significant degree, also depends on thinking about what constitutes good research and the ecosystems that are necessary to support it. It depends, in other words, on developmental meta-research.?

We have put together this edited volume that zooms in on the relationship between meta-research and global development to reflect upon these urgent questions. The volume tries to catalyze debates about decolonizing development and science in the Global South, boost the open science movement and its principles globally, nurture emergent local research ecosystems, and ask critical questions about methods and ethics.

We hope that the contributions will spark a debate about the potential of developmental meta-research, opening new collaborations across subfields, geographies, and sectors.?

Other examples of the developmental meta-research approach in action

These are the kinds of insights that can be gained from developmental meta-research;

1. Missing Majority Dashboard
2. Behavioral Insights in the Global South
3. African researchers must be full participants in behavioral science research:
4. SOBC measures repository