Untangling the Web: How to Measure Impact in Complex Programmes

Let's be real - trying to demonstrate lasting, transformative impact for a complex social programme or behaviour change initiative can feel like a mind-bending exercise. With so many interrelated factors and stakeholders involved, how can you cut through the tangled web to truly understand your contribution, what worked, what didn't, and why?

As development professionals know, linear logic models and traditional indicators often fall short when it comes to complex, adaptive interventions spanning multiple sectors or layers of society. Ticking boxes on pre-defined outputs and outcomes doesn't necessarily capture the deeper, more nuanced changes happening on the ground.

So how do we rise to this measurement challenge? How can we open the black box to shed light on systemic impacts in a credible, meaningful way?

We use 'complexity-aware methods' that don't rely on linear logic models or predetermined indicators.

Methods that are “complexity-aware” enable us to address the inherent complexity in modern development programmes, where environments are dynamic, multiple stakeholders intervene, and programmes have many sectors of activity.

- MEASURE Evaluation

One method is contribution analysis. Rather than trying to establish direct attribution links, contribution analysis examines the plausible contributions an intervention made towards an observed impact through evidence and logical reasoning.

For example, let's say a multi-year gender empowerment programme worked across communities to shift social norms, update policies, and increase women's access to resources like education, healthcare and financial services. Evaluators could map out the programme's theory of change, activities implemented, and audience responses. They could compile data showing changes in areas like girls' school enrollment rates, women's health outcomes, household decision-making dynamics, and income levels.

Then through contribution analysis, they could test hypothesized causal chains about how and why these positive changes occurred. Perhaps new policies and Budget investments created an enabling environment. Or maybe working with community leaders and local organizations helped spark mindset shifts that were instrumental. By weighing and validating each contributory factor and pathway, a coherent impact story emerges grounded in credible evidence.

Image reproduced from USAID

Of course, effectively measuring impacts in complex systems also requires drawing upon other cutting-edge approaches tailored to the context.

Case studies, outcome mapping, and process tracing can also provide rich insights. Techniques like Qualitative Comparative Analysis (QCA) can rigorously test what combinations of conditions and interventions are necessary or sufficient for an outcome to occur.

Many organisations are also embracing more participatory methods that engage communities and key stakeholders in defining success, exploring changes, and making sense of complex dynamics together. Through tools like Outcome Harvesting, diverse perspectives get elevated to co-create impact narratives.

Ultimately, these approaches share some common strengths - they embrace non-linearity and emergent impacts. They dig into contextual factors and unpack black boxes instead of isolating simple cause-effect links. Perhaps most powerfully, they fuse multiple data streams and viewpoints into holistic stories about sustainable change.

So the next time someone asks what genuine impact your complex programme achieved, you can point to robust evidence and richly validated impact stories. You'll reveal the positive ripples of change radiating out from your interventions, however non-linear and intertwined those pathways may be.

??Join my Monitoring and Evaluation Academy where you learn more about methods to measure impact via our monthly workshops and community of practice. We just had a workshop on outcome harvesting and the recording and handouts are available for members. Sign up today at https://lnkd.in/epqEsMF6