You are here

Is it impossible to measure youth work impact?

"Let us kill this myth now (spoiler alert), there is no holy grail. There is no way of measuring impact on people that is definitive and universally accepted."

As reported in CYP Now, these are the words of Paul Oginsky, chief executive of Personal Development Point and former advisor to David Cameron in a white paper, titled A Way Forward For Character Development: The missing piece of education.

Needless to say, there has been a strong response within the sector.

Read the original article

The following responses were initially featured in CYP Now, 25th October 2016. 

Bethia McNeil, Director, The Centre for Youth Impact:

Paul Oginsky is right that there is no 'definitive and universally accepted way of measuring impact'. A 'definitive approach' implies agreement, a conclusion and a sense of authority. Impact measurement in the youth sector has none of those features. He's also right that finding this definitive and universally accepted approach has at times seemed like the Holy Grail. This is in part because impact measurement has been connected with sustainability, and the suggestion has long been that the ability to 'prove' your impact or worth is associated with turning on the funding taps. A search for the Holy Grail is also motivated by the belief that it exists - that it is possible to find a definitive and universally accepted approach to measuring impact. Again, Paul is right - there isn't one.
Part of the challenge is language - what we seek to measure is change; many would say we can never measure impact, we can only estimate it, because we can never really know what would have happened without our 'intervention' (even though it's absolutely worth trying to work it out). And when we're measuring change, the way we do this must be determined by the question we're trying to answer, so there will never be one universally accepted way to do it. What we consider to be evidence of change varies wildly, and we can't ignore the power dynamics at play here.
The second area of challenge is 'universal acceptance': universally accepted means developing a consensus between practitioners, managers, funders, commissioners, politicians and young people. It means agreement on what is measured, by whom, when and why. This is unlikely and frankly undesirable. I'd prefer to focus on shared or common measures, rather than 'universally accepted'. It is absolutely right that we should look towards shared approaches for understanding impact, and for common measures where there is consistency in the way we work - and we have more in common within the youth sector than we'd like to admit. But we should care more about asking good questions of our work, and answering them reliably, in a way that is relevant to practitioners and young people, and is reasonable as part of frontline practice. We should care about asking these questions of ourselves, and trying to find the answers together. "Measuring impact" is not a path we should each forge alone.
Asking young people to tell us about their experience of programmes, services and relationships, and what changed for them, is vital. This kind of systematic feedback should be part of every organisation's approach to developing its provision and evaluating it. All young people should know how and when they can give feedback, and what happens to their feedback afterwards. This kind of feedback can be both qualitative and quantitative - both are equally important. Feedback is a critical part of understanding both quality and impact.
However, this isn't the only way. Feedback is only one part of understanding impact, and should sit alongside a clear understanding of the key features or activities within provision, an assessment of quality, and insight into data or information drawn from outside your work that can help you reflect on where it might be contributing to change - or outcomes for young people and communities. The development of character brings additional challenges because young people will be influenced by all their interactions and relationships, and it will be very difficult to connect change to any one organisation or project. Debates about character education are still fairly contentious, and there is no agreement on outcomes or even constructs - what we mean by resilience, for example - so it is unsurprising that we cannot universally agree on how to measure it.

However, we should take care not to confuse the fact that something is hard, and that we might not ever find a universally accepted approach, with it being a waste of time. The purpose of understanding impact, whether we seek to measure, assess or estimate it, is to create new insight, to learn and to get better at what we do. And this matters more than ever.  

Phil Kerry, Programmes Director, London Youth:

I read with interest Paul Oginsky’s recent white paper, A Way Forward for Character Development: The missing piece of education. Much of what Personal Development Point says rings true. There is no holy grail to character measurement within our sector and the sooner we all realise this the better. But attempting to condense all of this into one evaluation tool or definition would just be just as much of a fruitless mission. Our sector embraces a brilliantly wide church of practice and thinking and in this lies our strength and not our weakness.
Because our accountabilities lie first and foremost to the young people we are all here to support and challenge, we do need to understand and assess our impact. Whether we call this measurement, or assessment doesn’t really matter. But fundamentally it is about ensuring that our work is making a difference for young people. It should never be about pleasing funders and while Oginsky is right to challenge those funders who misunderstand this, it is also for us as organisations to be clear that Funders are not the bad guys in this situation. They are our friends and our partners and they need us, just as we need them, to deliver our respective missions.
The PDP report makes me even more convinced that within our sector we need to do more to bring a coherent narrative to our collective impact  - and it is therefore encouraging to see such collaboration beginning. But we need to celebrate and reflect our differences as much as our similarities. The risk is that if any of us claims that somehow our model or definition provides all the answers, then it will be much harder for the rest of us, who see things slightly differently, to move forward. If there is a holy grail, then genuine collaboration which acknowledges different approaches to get to the same outcomes, is the one I would commit my time to finding.