The Wild West of Social Return on Investment (SROI) in Aotearoa

Would you trust a plumber to do your electrical work?

Both electricians and plumbers have a trade background; both solve practical, hands-on problems with tools. Their broad skill sets have many things in common. Where they differ, however, is in their foundational knowledge and training.

I am sure a plumber can connect some wires – the problem is, without the foundational knowledge, are you confident that they will do it correctly?

I see the same issue in Social Return on Investment (SROI) analysis. SROI came out of work on applying Cost-Benefit Analysis to social enterprises in the early 2000s. Without a firm grounding in the principles of economics or finance, it is difficult to trust that an SROI provider has the knowledge and skills to produce rigorous work.

This may seem disingenuous as I am an economist who has produced SROI reports for several years. I hope you understand why someone might feel slightly protective of their craft when they see its standards being compromised. My intent here isn’t to stake out professional territory but to advocate for higher standards that will benefit everyone in the sector - myself included.

SROI can be a fantastic tool. I have written previously on the importance of SROI for Māori and impact assessment from a Mātauranga perspective. I will write more about what a good SROI looks like, but first, I need to have a ‘little’ rant about the current state of SROI in Aotearoa.

Many of the largest providers of SROI in Aotearoa do not have a foundational background in the theory that underpins non-market valuation. Additionally, most of the lighter SROI reports I have read provide no transparency on the method or data. If you cannot see how the values were determined, how can you be sure they are correct? This is a case where you really do want to see how the sausage is made.

SROI requires a broad set of skills, many of which do not come from economics and finance. Public policy, sociology, social psychology and more can be essential to a great SROI analysis. The challenge arises when it comes to assigning dollar values. Concepts like discount rates, attribution, deadweight, and others can be complex when assessing social impact. Until recently, the Treasury’s CBAx model, a core component of many SROI reports, used a standard discount rate not associated with social impacts. My colleague Assoc. Prof. Peter Tait has completed entire research programmes focused solely on discount rates. As the choice of a discount rate determines the overall SROI figure, it is critical to understand this component.

I have also seen questionable practices in assigning non-market values to an SROI analysis. CBAx contains values such as the monetary value of being able to express your culture, measured on a scale from 1 to 5. Where people go wrong is they then apply this scale directly to a participant group. For example, how much did the service increase your ability to express your culture on a scale from 1-5? The problem here is that this is not the question used in the original research, nor is the context surrounding the question the same. These values cannot be used this way, yet they are.

CBA is complex - SROI is complex. This complexity needs to be thoroughly understood for the outputs of an SROI analysis to be rigorous. The issue is even more pressing in the current political context. The Government has placed significant emphasis on demonstrating the social impact of a service when applying for funding. I have increasingly seen funding guidelines that require an SROI report from an independent third party. What kind of third party is not specified.

Early in my career, I was focused mainly on sustainability and sustainability reporting. I would look at sustainability jobs and their requirements. The main requirement was typically a qualification in health and safety. I never understood why – until later, when I realised it was simply about compliance and not sustainability. SROI is highly valuable and can enormously impact an organisation - it should not be treated as a tick-box exercise.

I have helped organisations raise millions in funding off the back of an SROI report, but even more importantly, they have identified leverage points to significantly increase their impact by going through the process.

I applaud your perseverance if you have made it this far in my rant! I wrote this in about 20 minutes after encountering another problematic SROI report - one of dozens I’ve seen that fail to meet basic standards. While this piece stems from frustration, my next article on SROI will focus on constructive solutions and best practices for conducting rigorous analyses. ??