The Social Value sniff test - five key questions.

Seen any social impact and economic value reports that spray around £millions in value like a Cash Cannon ?

When we do this type of work we make sure we can answer this set of Fab 5 fundamental questions. If the report you are reading cannot say YES to all of these, be careful.

1. Have those reporting ££s of value actually asked the people they work with about the benefits they experience?

Is there any evidence from the people who are meant to benefit from the activity, charity or programme that they have experienced those benefits? Have those claiming the social value asked them? If not, why not? Kids Company claimed to work with 30,000 kids - but there was no evidence they could contact them, ask them anything - and we all know what happened there… 

2. Have key characteristics, like socioeconomic group, of participants been considered and factored into the findings?

Participants who are more affluent are often already happier, healthier or more physically active. If you don’t find out the socioeconomics of the people on the programme you may simply be reporting the impact of wealth rather than the project itself having any impact at all. Conversely, results could downplay the impact and value for participants from lower socioeconomic groups.

3. What would have happened anyway? Otherwise known as the counterfactual...

Have the outcomes of the project group been compared to those who were not on the programme (known as a control group)? e.g. Project A reports helping kids be more active from June to September. But the population is more active from June to September anyway - sunshine, school holidays and in 2020 the easing of lockdown.

If the counterfactual hasn’t been addressed then you can chuck the report out the window, or at least raise your eyebrows and strongly question whether the reported change has anything to do with the project in question.

4. Have pound signs been attached responsibly?

Big numbers can give a social impact report some oomph but be warned - it can be tempting to use these numbers with a freer hand than is credible. If ££s are attached are they linked to a recognised methodology (e.g. from the UK Govt Green Book on Policy Evaluation) and is this explained very clearly and openly up front?

Often projects claim big savings to services like the NHS - but can we trust them? People who feel healthier visit the GP less but any cost saving beyond that is much more complex. When it comes to reduced spend on diabetes care, heart surgery or hip replacements, many other complicated factors come into play - so any claims require a leap of faith. A similar case goes for other claimed savings to public services like police, crime, social care etc.

5. Can the big claims be backed up with a simple, easy to understand, and transparent report?

Is there a clear, transparent data set, explanation and suitable caveats around how the values and cost savings were arrived at? Can you access and understand the language and method used and are the sources and references clear? If it’s not 100% clear, be careful - you may be being blinded by science.

State of Life can help you do social value in a way that is simple and credible. Check out our free Step by Step guide to social value for a solid starting point on all this.