Impact measurement

Last week I wrote about measuring value - a concept that at first seems straightforward (££££) but on closer examination turns out to have a deeper, qualitative layer. Which means that “impact” is not dissimilar to value. It also means the challenges of quantifying success for impact-first organisations is not so unique and strange as it might appear.

So how should we go about measuring impact?

The best way I have come across is to have a Theory of Change.

That’s a phrase that I have seen used in various contexts, and so let me define what I think is the core of it. It’s these two propositions:

The ultimate social benefit we care about is X

We believe if we do Y, then we will increase X¹

The first proposition defines outcomes. And as mentioned last week, these might not be possible to measure (“Improving long-term wellbeing for care leavers”, “Reducing chronic ill health in economically deprived regions”). But, if you are a non-profit or social enterprise, these outcomes are why you do what you do. They are the goal.

Since you cannot measure outcomes directly, you are limited to measuring outputs - which are defined in the second proposition. Often, a focus on outputs over outcomes is an error for organisations, because it actually doesn’t matter how much you did, it matters how well you solved the customers problem. But as long as you provide the right leadership around them, outputs can be useful things to measure.

The second proposition not only defines outputs, it claims there is a relationship between outputs and outcomes. This logic (Y —> X) is the ‘theory’ of the title² - but really it’s more like a hypothesis. It’s something that requires continual testing. So as well as measuring outputs, you need to continually examine whether this logic holds. You have to look for evidence that you are having a positive impact in the way you want.

Since impact is qualitative, this evidence is not going to come in the form of numbers. Instead it exists as stories. Stories that reflect the real world experience of your beneficiary customers. Stories that you learn through the sum total of your organisation’s interactions with them. Although your numbers are useful, it’s stories that are the heart of “impact measurement”.

So to summarise, if you want to embed effective impact measurement in your organisation:

1. Define your Theory of Change

2. Measure your outputs - but keep reminding people it’s outcomes that matter

3. Continually gather and review stories to test your theory

Next week, I’ll explore how this interaction between the qualitative and quantitative, as illustrated by both value and impact, has implications for how to approach all kinds of measurement inside organisations (such as KPIs).

1. Of course you can have more than one benefit and more than one thing you do - multiple Xs and Ys - within a single Theory of Change.

2. When writing your Theory of Change down, you can include a brief explanation of this logic.