FAQ

+ Why are your tools free?

We believe that the charity and not-for-profit sector will benefit from this set of survey tools more if they are free because:

• Any organisation, large or small, can start to exploit sector-leading methods to understand how effective their work is.
• We want to democratise impact analysis so that it's no longer the preserve of bigger, richer organisations.
• We hope this approach can contribute to more transparency and accountability.
• Trust is important - we want to grow trust in this progressive discipline and only charge clients where we genuinely add expertise and value.

We know from our work to date that we add the most value in analysis, insightful reporting, communication of results, and where we can draw on our expertise and experience to tell your story brilliantly.

+ Are they really free?

Yes, if you use the standard survey it’s free but there are a couple of conditions.

• We would like to speak to you.
• Demographic controls and GDPR implications of collecting data are both important so we want to ensure these are understood.

+ What about areas we're interested in that aren’t in the State of Life Surveys?

We can add in tailored questions to the basic survey and analyse them for you too. The outcome areas you're interested in may already be the in the State of Life Survey Bank, if not we can look in the data sets themselves.

+ What if we don't work in Youth, Sport, or Volunteering - can we still use this approach?

Absolutely.

The State of Life approach works for anyone who engages with people and seeks to improve their lives in some way. The three free surveys are just the beginning. We can develop a survey that fits with what you do using our Survey Bank of over 200 questions from UK open data sets. Get in touch - we would love to hear from you.

+ Why not just use our own bespoke survey?

If you use the State of Life approach you results can reveal so much more.

Using pre-validated existing questions from large national data sets has a number of significant advantages: questions are already thoroughly tested and crucially you can access comparisons in the national data. It is an approach endorsed and adopted by the ‘What Works Centre for Wellbeing ’ in their excellent ‘How to Measure your Impact on Wellbeing’ tool.

We know this because in 2014 while running a charity we used a bespoke wellbeing index. Only to find that the questions and results were not directly comparable to other much bigger data sets, thus making the data less valid, comparable, robust and useful. We learned from mistakes.

+ Why not compare our data to an actual control group?

Randomised control trials are very expensive and can take you on to shaky ethical ground

Doing a randomised control trial is a huge undertaking and is not cheap. You may be able to find a suitable control group in the community you are working with but it is ethically questionable to collect data from people who need your intervention but who are not able to access it. One alternative is to buy responses from an online provider who find respondents from your target group. However, this can also be extremely costly.

Using the national data to access comparison groups is an efficient, robust, and affordable way to compare your data. It also reaches Level 3 in Nesta's Standard of Evidence.

+ What are the common data pitfalls that could leave us with misleading results?

Not including the right demographic questions can leave you with misleading results

We know people who volunteer tend to be richer, and we know that richer people tend to have higher wellbeing. If you don’t control for this in your analysis your results may just be confirming that more affluent people are happier, or similarly that religious people are more trusting, or people with good physical health are less anxious - rather than understanding the difference your activity actually makes. But you won’t know for sure. If we ask about these key characteristics, we can control for them in the analysis (cancelling out their effect) and isolate the real impact of your project or activity.

+ How can you support us?

However you need

We can work with you to include all the demographic controls you need to enable quality analysis, to tailor your survey, do the analysis (there are different techniques we can use to interrogate the data in different ways), interpret outputs into key findings, translate results into valuable insight, and work with you to communicate this in the best way for your audiences. This could be a data visualisation, a slide deck, a brief two-pager, or a detailed technical report.

+ What is the process if you do our analysis?

We will talk it through with you and maintain a dialogue throughout the process

We take time to discuss our approach so you are comfortable with the process. Our expert team will take the data you collect and firstly produce descriptive statistics that reveal who your participants are and how they compare to population averages, or specific control groups within the national data. Are they younger, happier, healthier, more female, more confident that others in the UK? So far, so standard.

Then we will run regression models specifically to reveal the impact of your activity. Our advanced statistical methods look at causation (i.e. does our project lead to higher reported health?) not just describe correlations (i.e. do people on our project report higher than average health?). All of our analysis is compliant with HM Treasury’s Green Book (2018)- this is Government guidance on evaluating policy.

Some of this is dependent on achieving certain sample sizes so we can chat about that. We can also use other analytic approaches to interrogate your data in different ways - give us a call to find out what we can do for you.

+ What if our participants aren't typical? Comparing them to the national average would not make sense.

We know many interventions work with people who are not average. State of Life can reveal the real impact of this

When comparing your participants to national data sets we are able to find a synthetic control or comparison group based on key characteristics. For example, if you target people from lower-socioeconomic groups we can restrict the comparison group to also be people from a lower-socioeconomic group. Or if you work with young people who have low confidence we can find a matched comparison group based on this too. This means the impact of improvements however small will be revealed, which may not be apparent if they were instead compared to a group of more average young people.

+ What if our results show we’re not making the difference that we hoped?

This is absolutely fine and you don’t have to publicise results. The information can still be used to inform internal decision-making to assess and improve what you do.

Wouldn’t you want to know if you’re not achieving aims especially if it can help you to make important changes?

+ Who owns the survey data?

You do. But we hope you agree to share the analysis and reporting with the wider sector and publish on the State of Life website. Long term, State of Life will look to collaborate on a 'data hub' where all the data collected is pooled. The objective would be to help policy makers and funders understand which organisations are open, accountable and confident in their results.

+ What about valuation?

Once analysis has revealed the impact of your activity or programme, there are different ways to monetise that value. This can help to communicate your impact and some funders or decision-makers may have a preference for this step. We are happy to chat if this is something you would like to explore. Some organisations aren’t interested in valuation and this is fine - it's an area that provokes much debate!

+ What sample size do we need? What if we only work with a dozen people?

There are certain numbers of survey responses that you may need to achieve for some types of analysis to be possible. The exact number depends on a range of factors so get in touch to find out more. We can provide descriptive statistics and indicative results that will offer insight and inform your next step regardless of sample size.

+ What did others find out and how have others used their results?

Please take a look at the Our Work section https://www.stateoflife.org/our-work We'd be happy to put you in touch so you can speak to these organisations about their experience directly.

+ Is the survey mobile responsive?

Yes, all the tools we use and recommend are mobile responsive.

+ What will it cost for you to support us?

This will depend on the support you're after. We’re more than happy to discuss what we’re able to do within your budget.

+ What do we have to commit to?

A chat.

+ How do we get involved?

Use the contact box here https://www.stateoflife.org/join if you have any questions or would just like to have a chat to find out more.