What's the best way to compare program evaluations?

1 Define your comparison criteria

Before you can compare program evaluations, you need to have a clear idea of what you want to compare and why. What are the main questions or objectives of your comparison? What are the similarities and differences between the programs and their contexts? What are the relevant dimensions or aspects of program performance and evaluation that you want to focus on? For example, you might want to compare the relevance, efficiency, effectiveness, impact, or sustainability of different programs, or their adherence to certain standards, principles, or values. You might also want to compare the quality, rigor, or usefulness of the evaluation methods, data, or findings. Depending on your comparison criteria, you might need to use different tools or approaches to analyze and synthesize the evaluation information.

2 Choose a suitable comparison method

When it comes to comparing program evaluations, there are various methods to consider. Meta-analysis is a statistical technique that combines the results of multiple evaluations in order to calculate effect sizes and measure the overall impact of a program or intervention. This requires that the evaluations use similar outcome indicators and report quantitative data. Meta-synthesis is a qualitative technique that integrates the findings and insights of multiple evaluations to generate new themes, concepts, or theories. Comparative case studies are also useful for comparing programs; they require sufficient data and information to allow for meaningful comparison. All of these methods can help you compare the magnitude and direction of effects, meanings and implications, strengths and weaknesses, as well as similarities and differences between programs.

3 Apply a consistent and transparent comparison process

Whatever comparison method you choose, you need to apply it in a consistent and transparent way, so that your comparison results are valid and reliable. This means selecting and justifying your comparison sample, assessing and reporting the quality and credibility of the evaluations, analyzing and synthesizing the evaluation information, and presenting and interpreting the comparison results. You must consider criteria for selection, standards or criteria for assessment, tools or techniques for analysis and synthesis, main findings or conclusions of your comparison, level of confidence or uncertainty in your results, and any ethical or political issues or implications of your comparison. Doing so ensures that your comparison results are accurate and trustworthy.

4 Use the comparison results for learning and improvement

The ultimate purpose of comparing program evaluations is to use the results for learning and improvement. This involves sharing and discussing the comparison results with relevant stakeholders, applying the comparison results to inform program decisions and actions, and monitoring and evaluating the effects of the comparison results. When sharing the results, it is important to consider who the intended users or audiences are, and solicit and incorporate their feedback or perspectives. The comparison results should also be aligned with the program goals and strategies, so that any changes or recommendations can be prioritized and implemented. Finally, it is essential to measure and report the changes or outcomes resulting from the comparison results, as well as identify and address any unintended or negative consequences.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?