Quick tips for better impact evaluation

I just finished a second impact evaluation project for a client, so I have spent a lot of time thinking about best practices for funders the last few months. Here’s some quick tips on how to approach evaluating the impact of your portfolio to go to the next level:

Refine what you measure, and avoid out-of-the-box “impact measurement” solutions.

Far too many funders simply emphasise growing number of publications (without any additional context on what the publications are about), or big-looking numbers of leveraged funding, that do not stand up to scrutiny.

Instead, think about what you care about and what you are trying to influence through your funding programmes (i.e., develop a theory of change, but that would be the subject of a longer post!).

Are you a translational organisation? Then measure how often your research is influencing patents, or its disruption index. Are you trying to improve people’s health through clinical research? Then measure clinical guidelines inclusions, or implementation and adoption rates.

These things take some thinking, and likely some digging, but you would be surprised about the type of data that is readily available for you to analyse.

I have just completed a piece of work on how to get the most out of a subscription-based scientific database. Part of the work was showing that, when you look at the data in the right way, you can answer very sophisticated questions relevant to your organisation’s strategy, from how often your researchers collaborate with healthcare professionals, to how interdisciplinary their work is.

Start at the portfolio level, and then drill down to the project level

Often funders put off doing impact evaluation, because they think they need to start ploughing through old progress reports, hundreds, maybe thousands of them, to understand their impact.

Instead, looking at the outputs and outcomes from your whole portfolio, side by side with thematic analyses and AI categorisations, can help you quickly uncover key areas of research that performed well, and that in turn leads you to specific researchers and projects to focus on for more extensive analyses.

I am currently doing a piece of work for a client where the concept “Malawi” came back with an 80% policy citation rate (i.e., 80% of the publications that were about “Malawi” had gone on to influence policy). Looking at their portfolio, it was easy to find the single project based in Malawi they had funded.

Doing it the other way around, finding this project would have been like looking for a needle on a research haystack.

Focus on the people

I would bet money that every funder, even very small ones, have been key in the career development of leaders in their respective scientific fields. They probably awarded that junior fellowship that led to the senior fellowship from one of the big funders, or they funded a pilot project that gave the researcher the key data to apply for a big project grant.

Most funders would be able to list 2-3 people who fit the above, but I argue that if you dig a little deeper, you’ll be able to find those additional connections that show the outsized impact your funding can have.

In a previous project I was doing interviews with senior researchers in a field with long-standing relationships with my client. Without a fault, they all talked about the impact my client had had in providing early funding for some of the absolute leaders in the country today (on occasion, themselves included). Several of them started listing names, one after another, of people who had developed a successful career in part thanks to that little grant, or that PhD fellowship, from my client. It was a beautiful thing to see these very senior, very respected researchers expressing that type of gratitude for the role small pots of funding had in their fields.

Think beyond fundraising: impact evaluation is about making better decisions (which then help you fundraise)

Finally, while the need to demonstrate your impact is and will continue to be a key tenet of fundraising organisations, when done correctly, it can be much more than that.

It can help inform your strategic direction. It can help define your next funding call. It can point you towards people to partner with. And once you do all of that, it absolutely can help you fundraise.

In other words, for me fundraising is a downstream effect of good impact evaluation, instead of the sole reason to do it.

So, next time you think that impact evaluation is just about creating pretty charts and catchy headlines, I’d encourage you to think twice!