Will we ever learn?

In this blog, Bethia McNeil, Director of the Centre explores the seeming disconnect between evidence and learning.

The debate between whether proving or improving should be the focus of evaluation is a particularly live one. The underlying suggestion is that we all – obviously - want to improve rather than prove, given half a chance. But improving provision is intentional, and at its heart involves learning: not just the desire to learn but also the capacity and the feasibility.

This blog is a write up of a recent talk I gave at an event where I was asked to talk about why we might not be learning from our evaluation and impact measurement activity.
​
It is perhaps odd, maybe contentious, to suggest that we – those of us involved in the broad range of provision for young people – have arrived as a point where evaluation is disconnected from learning, but I believe this might just be the case. It has become something of a cliché to talk about a shift from proving to improving, but this is overly simplistic. It suggests that there is an either/or choice, and that the two might be mutually exclusive. It also creates a reassuring sense that all we need to do to refocus is to change our language. None of these things are true. Much of our evidence gathering effort is about decision making – this is the case whether we’re seeking to prove or improve – but again, making decisions implies that we are learning from and acting on the evidence that we gather.

There are three particular signs that I think suggest that evidence and learning are disconnected:
Firstly, that after many years’ effort in evidencing the impact of informal and non-formal learning, we are no further forward in coalescing around a shared evidence base that affords us a collective language in identifying, building on and advancing “the difference that makes the difference” – that is, the particular elements of effective practice in informal and non-formal provision, and the difference it creates, or contributes towards, in young people’s lives. This is particularly remarkable given the strong – and justified – complaints about the burden of impact measurement on providers, and the hours of effort expended. It also suggests that if indeed we have all been focused on proving over improving, then our bar has not been set very high.

Secondly, we have no sector-wide quality improvement effort, based on a shared understanding of what constitutes ‘quality’, its relationship with impact and how we know whether quality ‘is present’. We continue to advance an organisation by organisation approach that is designed to highlight and protect the uniqueness of each one’s work. We are nowhere near a scenario where we can say – individually or collectively – that because we do this, we can reliably predict that young people will experience that, and that these changes might be sustained into other contexts in their lives.

Thirdly, and finally, there continues to be a fundamental disconnect between the evidence we are gathering, and the act of ‘knowing’ about our work. We rightly prioritise the voices of young people, but seem to imply that our impact measurement practice is incompatible with this. We talk of the difficult trade-off between relevance and rigour. Why are the two apparently in conflict?

But my complaints come from a place that assumes that evidence gathering and evaluation should be about learning. What if they aren’t?

I think there are several reasons why we collect and collate evidence: PR and fundraising, accountability (to both our funders and to young people), and learning and improvement. There may be a few more too.

In an ideal world, these reasons would come together, rather than divide our efforts. But we are a long way away from that now. And where is the incentive to change? Culturally, we don’t have a clear blueprint or understanding of what it means to be a ‘learning organisation’ – and this goes so much further than the current fashion for talking openly about failure.

This is a blog about complaints, rather than solutions, so I am not offering any particular responses here. That will be the subject of a future piece.

A member of the audience for the talk I gave last week told me afterwards that I had made people ‘bristle’. Good, and let us now mount a defence, but not as individuals – as a collective.
​
Ed. - A version of this talk was originally given at a Generation Change/Step up to Serve roundtable on quality youth social action.