Posted 12th June 2017
Ten things you should be looking for when reading an impact report. A blog by Christina Berry-Moorcroft, Social Value UK Communications and Membership Coordinator.
Here at Social Value UK we are incredibly proud of our Reports Database, at last count it had nearly 700 reports relating to social value in there for you to peruse and learn from. The reports database is made up of SROI reports, assured and non-assured, social impact reports and cost benefit analysis reports. So whatever sector you work in, whatever level of riguer you’re working at and whatever you need examples of, you can find it in one place. But, what if you’re not sure what you’re looking for, or at? We know that many people like to read impact reports before setting off on their own, so to make the process easier we’ve compiled a handy list of ten things you should be looking for. We hope it helps!
What is it for?
What is the report you are reading for? Who’s the intended audience? What are they hoping to communicate and/or achieve with this report? If this is immediately clear, then great, keep this in mind throughout the rest of the report. If it isn’t, then this may not be the end of the transparency issues…
What has and has not been included?
Have the authors been completely transparent about what has and hasn’t been included? Have they also explained how they came to their decisions about what was and was not material? At this point, it’s also wise to check that the report uses enough data, that it’s representative and that the authors have considered all possible biases. If a report isn’t inclusive of all of these things then there’s a fear the authors of the report are over-claiming, meaning they may be claiming the value of activities they were not a part of creating.
Is this an output or outcome?
What’s an outcome? What’s an output? What’s the difference? This can all too often be confused when people start measuring their impact but it’s vital that we aren’t claiming outputs as outcomes if we are to avoid over claiming. An output is a way of describing the activity in relation to each stakeholder’s inputs in quantitative terms, for example the staff who work on the project or the hours a volunteer put into the project. Outcomes are the changes resulting from an activity. The main types of change from the perspective of stakeholders are unintended (unexpected) and intended (expected), but more on that later. For now, ask yourself, has the report you’re reading made clear the difference between outputs and outcomes? We have produced some supplementary guidance on well-defined outcomes that will help.
Who decides what?
The Principles of Social Value provide the basic building blocks for anyone who wants to make decisions that take this wider definition of value into account, in order to increase equality, improve wellbeing and increase environmental sustainability. The first principle is the underpinning for all of the following six, and one that you must not forget when reading or writing an impact report! Principle One is Involve Stakeholders and we define stakeholders as people, organisations or entities that experience change, whether positive or negative, as a result of the activity that is being analysed. Were stakeholders involved? And importantly, were they involved in identifying and valuing outcomes? No two outcomes are equal to stakeholders, have the authors shown how they decided which outcomes were more valuable?
What about stakeholder segmentation?
Unless the only people impacted by your activities are the exact same in every way: age, gender, socio-economic status, health, relationship status, family set-up, and so on, and that’s highly unlikely, then not including stakeholder segmentation is a massive flaw. Segmentation is the process of looking at different groups of stakeholders, based on anything that makes them different from another group of stakeholders. Each type of stakeholder will have different outcomes, and indeed different valuations for those outcomes, and if the report you are reading hasn’t taken that into account then they could be over-claiming!
Is it all positive impacts?
The report you’re reading may be full of positive impacts, in fact we hope it is, we fully believe that measuring and maximising social value will lead to a more equal, more just and more sustainable planet for us all. However, every project, despite how great it’s theory of change is, will have negative impacts, it’s unavoidable. So has the report you’re looking at made allowances for that? Have they asked stakeholders what those were? Have they put measures in place to reduce these? Are they improving their work next time based on this? These are all questions you should be looking for the answers to when reading an impact report.
And what about the change we didn’t mean to cause?
Much like with point four, another key thing to look out for is whether the report lists the unintended outcomes of the project. If a report just states the intended outcomes, then they have only looked at, and asked stakeholders about the project objectives. This doesn’t give a full overview of the project, could be neglecting significant impact and is stunting future project improvement based on impact data. Examples of intended outcomes could be increased confidence, a secure job or reduced isolation, whereas as an unintended outcome could be that attendees on a course suffer a reduction in self worth for not doing as well as their classmates. That’s not to say that unintended outcomes are always negative, organisations often create lots of social value that they aren’t accounting for, listing unintended outcomes makes sure this isn’t happening and allows for project design based on all of the information.
How much of this change can they claim?
So the authors of the report have considered all of the outcomes, based on the opinions of their segmented stakeholders, they’ve looked at intended and unintended outcomes and considered both positive and negative change, that’s all right? Well, not quite. In order to not overclaim (Principle 5) and be transparent (Principle 6) then the authors should have also calculated the likelihood of what would have happened to the stakeholders anyway, without their action and what other actors (organisations, people, interventions) may have played a part in the change experienced by a stakeholder. We call this considering the counterfactual and attribution. If 100% of a change is claimed it’s not only not true, it’s a little foolish and means the author is over-claiming, which means their project isn’t as successful as they claim it is, which means they are missing valuable opportunities to make it better!
Is this important?
Have the authors of this report demonstrated that they undertook a process of determining what information and evidence must be included in the accounts to give a true and fair picture? Some information is important is material and some is not, has this been considered? Enough information should be included so that stakeholders can draw reasonable conclusions about impact. On top of this, has the information been ranked with relative importance, and importantly, were stakeholders consulted on this?
Can I trust this logic?
One thing to consistently consider with impact reports is transparency (Principle 6), authors of impact reports should demonstrate the basis on which the analysis may be considered accurate and honest, and show that it will be reported to and discussed with stakeholders. Have they? And finally, but crucially, have they sought independent assurance of their impact data, assumptions and reports? Verifying the result (Principle 7) is crucial, and shows a mark of quality. You can search our Reports Database for only assured reports, to ensure you are reading reports that have met the standards we require, and find information on the assurance and accreditation services we offer on our website.
For further information on conducting social value measurement please see ‘The Guide to SROI’ and supplementary guidance. We offer a range of support services, including mentoring, that can support you further.
What are your top tips for reading, or writing impact reports? Do you have advice for other community members? We would love to hear your thoughts!