Reflections: Peer review of evaluation planning

I’ve recently had the opportunity to peer review evaluation documents for a number of public sector clients. While it been very refreshing to not be the one writing these documents, it’s also been fascinating to see how different organisations and evaluators are designing evaluations and bring their documentation together.

I’m noticing a few themes emerging through my peer review work.

• Don’t let brevity impact on functionality - While it’s important to ensure that you’re concise and to the point, evaluation planning documents still need to contain the detail. Ideally any evaluation plan should contain enough information that someone else could use it to deliver the evaluation as you intended – without speaking to you. If relevant information is contained in other documents, include the cross-references so people know where to find the detail.
• Confirm your understanding - Make sure that you have a clear understanding of the scope and purpose of the evaluation before you begin drafting. If this is not clear in your own mind, you won’t be able to create a cohesive plan that can deliver the information you and your stakeholders need.
• Program logics aren’t just a pretty diagram – If you are going to the effort of creating a program logic model (and I highly recommend you do!) make sure you that you are actually using it design your evaluation. Refer back to this model as you develop and refine the evaluation plan – do your KEQs align with program progress? Have you identified measures and data collection approaches that will demonstrate whether your activities, outputs and outcomes are being achieved?
• Think carefully about your data sources – It’s very easy to get excited and create long lists of what data you’ll consider as part of the evaluation, but this is only half the puzzle. When planning your data collection consider not only what information you would ideally like, but also what you will be able to practically access, collect and analyse.
• Be practical about your consultation activities – When planning consultation activities think very carefully about who you need to speak with, and why. We often get minimal time to speak stakeholders during an evaluation so use their good will wisely – focus on questions that you can’t answer through other means.
• Detail your assumptions – When designing an evaluation, you’ve no doubt run through multiple options of how it could be delivered. Make sure you are capturing any assumptions that you have in mind around the delivery approach.?
• Don’t skimp on the research – There is a wealth of knowledge and findings contained in evaluations, reports and other publications that have previously been conducted. Take the time to find out if any similar evaluations have occurred in your organisation, or if there are a published reports (be it evaluations or other report types) that deal with a similar subject. Considering how others have evaluated in similar circumstances, including where they were successful and how they overcame challenges, will help you create a robust evaluation plan.

What are my key takeaways?

There are amazing evaluations happening all over the public sector right now. To make sure that these are successful we need to ensure we come back to the basics and have clear and robust evaluation plans in place.

When planning evaluations, it is important that you:

• take the time to understand the program and define the scope and purpose of each evaluation
• ensure the evaluation is tailored to the program at hand, reflecting the logic model, applying learnings from similar evaluations and fit-for-purpose considering the practicalities
• capture information about the evaluation in the planning documents. This not only ensures other can deliver the evaluation as intended (if required) but also provides a record of how the evaluation was structured others may learn from in the future.