Quantifying Impact: Key methodologies for programme evaluation
Dragos Dragomir
Director of Support Services @ Betknowmore UK | I help organisations to design, deliver and evaluate addiction and mental health recovery programmes | Founder of Architecture of Recovery
In the realm of addiction and mental health treatment, understanding the true impact of your efforts is crucial. It's not just about whether clients show improvement; it's about determining if your programme was the driving force behind those positive changes. To achieve this, a robust impact evaluation is essential. Impact evaluations measure the effectiveness of interventions by examining changes in clients’ lives and attributing those changes to the program. This article outlines a comprehensive list of methodologies designed to provide a thorough assessment of your program's impact, ensuring that your efforts are making a meaningful difference.
Methods presented are grouped into the following sections:
A.????? Methods for synthesis of existing evidence
?
Always a great first step to begin with. The synthesis of existing evidence can be used to answer evaluation questions relating to the impact of an intervention, the process by which it was implemented and the value for money of the intervention.
?
B.????? Theory based methods for impact evaluation
?
Theory based methods can be used for impact evaluation to address questions about whether the intervention caused an impact, how and why it occurred, how context may have influenced outcomes and help understand to what extent results are generalizable.
They allow attribution of causality, but none gives precise estimates of effect sizes.
?
C.???? Experimental and quasi-experimental methods for impact evaluation
?
For all statistics fanatics out there. The methods outlined below are useful for generating an impact estimate; the size and significance of change brought about by an intervention. These methods are less able to answer questions about how and why impacts occurred (sometimes known as the ‘black box’), who was affected, how context played an influence and the extent to which impacts are generalizable.
?
D.???? Methods for value-for-money evaluation
?
These methods are used to answer questions about the net value of an intervention. It compares its benefits relative to its costs. By determining the relative benefit to cost of different interventions, organisations can ensure that their money is used to produce the most value for society. “For every ￡1 spent, we generate ?￡x of benefits to society”.
?
E. Generic research methods used in both process and impact evaluation
?
?
A.????? Methods for synthesis of existing evidence.
?
1.????? Systematic review
2.????? Scoping review
3.????? Rapid evidence assessment (for when the deadline is looming…)
4.????? Meta-analysis
5.????? Meta-ethnography
6.????? Network analysis (such as eco-system mapping)
7.????? Realist synthesis (not that the other are fictional by any means)
?
B.????? Theory based methods for impact evaluation.
?
1.????? Qualitative comparative analysis
2.????? Realist evaluation
3.????? Process tracing
4.????? Contribution analysis
5.????? Bayesian Updating
6.????? Contribution tracing
7.????? Most significant change
领英推荐
8.????? Outcome harvesting
9.????? Simulation modelling (I love to use this when I design new programmes)
?
C.???? Experimental and Quasi experimental methods.
?
1.????? Randomised Control Trial (the godfather of all evaluation methods)
2.????? Propensity score matching
3.????? Pretest – Posttest methods
4.????? Interrupted time series analysis
5.????? Instrumental variables
6.????? Synthetic control methods
7.????? Difference in difference (confusing… I know!)
8.????? Regression discontinuity
?
?
D.???? Methods for value-for-money evaluation.
?
1.????? Cost Benefit Analysis
2.????? Cost effectiveness analysis
3.????? Cost consequence analysis
4.????? Econometric analysis (time series, panel data etc.)
?
E.????? Generic research methods used in both process and impact evaluation
?
1.????? Interviews and focus groups
2.????? Case studies
3.????? Surveys, questionnaires and polling
4.????? Output or performance monitoring
5.????? Data mining
6.????? Content and process analysis
7.????? Documentation analysis
8.????? Concept mapping
9.????? Qualitative observational studies
10.? Participatory (deliberative) methods
?
And this a not? a complete list by any stretch of imagination.
There are many other methods out these as well as sub-methods within the ones listed.
Moreover, to make matters more complicated best practice advises for use of
-??Sequential Exploratory Design (qualitative analysis followed by quantitative analysis, or the other way around) or
-??Triangulation (using multiple methods to study a single issue) in order the most comprehensive and accurate answers to our questions.
?
Let me know if you are curious to find out which of these methods would be most helpful to understand the impact of your programmes and interventions.