When treasuring is measuring, and why we might need a rethink
In this blog jointly written by Bethia McNeil, Pippa Knott and Matt Hill, the Centre’s core team responds to some of the key issues raised by Tony Taylor’s article regarding measurement in personal and social development. The Centre addresses the challenges found in the current dominant measurement framework and propose a rethink of the value of measurement in youth work.
Back in March this year, we hosted an event focused on measurement in personal and social development. We were really pleased to see Tony Taylor’s recent article in Youth and Policy, following up on the discussion, and agree that it would have been most beneficial had there been more time and space to explore the themes. Indeed, these themes are so vital that we felt moved to add our voice to Tony’s in this blog. Overall, we were struck at the many points where we agree with Tony’s forthright critique of the dominant paradigm in impact measurement, but there also remain some areas of fundamental disagreement – perhaps as might be expected in such a complex and contested area.
No measurement framework is ideologically neutral
We agree wholeheartedly that the theory and practice of measurement is never neutral - and arguably, neither should it be. Accepting this latter point would create more space for critical reflection on the inevitable ‘positioning’ of existing practices and frameworks, rather than an illusory search for independence and objectivity. Equally, focusing measurement efforts on the potentially less contested area of ‘skills’ as opposed to character, awareness or consciousness does nothing to make neutrality any more likely.
Any measurement framework involves collating, distilling and selecting information. This process is influenced by the background, experience and disposition of the individual or team carrying out the task, and the broader political and relational context in which they are acting. We are learning from other sectors - even those more historically predisposed towards the 'scientific method' - where people are equally questioning whether we can really rely on scientific robustness to neutralise values and context. Yet we disagree that failing the objectivity test is fatal. Instead, the best research and measurement acknowledges context and bias and embeds a constant reflection into research practice, including consideration of the influence on relationships ‘in the moment’. This has the potential to result in an enhanced, not a diminished science and practice.
The current dominant measurement framework systematically undervalues certain forms of activity, and privileges others
Open youth work, a term usefully endorsed by Tania de St Croix, sits at odds with many dominant forms of measurement, not least due to a (laudable) resistance among practitioners to impose pre-determined measurable ‘outcomes’ upon young people. We believe that we absolutely need to find ways of gathering, interpreting and sharing data about this approach to engaging with young people that both enhances practice and generates meaningful evidence and insight. We do not believe that this should be led by the pursuit of funding (particularly given that, as Tony and others have noted, the relationship between evidence and funding is nowhere near as straightforward as we might think/hope), or capitulation to broader forces that can seem overwhelming. We believe that this is about learning, improving, developing and advocating.
At the Centre we do not define the youth sector tightly. In fact, we sometimes deliberately don’t define it at all, preferring to work with and build shared understanding and learning among as many organisations engaging with and supporting young people as possible. But this is dependent at least in part on the ability to talk meaningfully and collectively about our practice. We agree that it might be harder to ‘measure’ the impact of youth work than other more targeted or narrowly defined forms of work with young people – but, for us, this demands that we develop how we measure and understand what really counts about youth work, and via a process that enriches rather than undermines practice. We should also pause to reflect on why we feel it’s ‘harder’ to measure impact in (particularly open) youth work: harder fundamentally, or harder to fit within the dominant paradigm?
There are broader conceptions of measurement out there
We agree the current dominant paradigm of measurement in the social sector has emerged within the neo-liberal landscape and shares (and actively perpetuates) many of its key features - monetisation, marketisation and competition to name a few. But neo-liberalism doesn’t have a monopoly on measurement and indeed there are many participatory and emancipatory methodologies of ‘measurement’ that are fundamentally opposed to it. There is a risk that we throw out the ‘measurement’ baby with the neo-liberal bathwater. Instead we believe that effective measurement of open youth work is a crucial bulwark against narrow conceptions of value. ‘Measurement’ in the current parlance has become short-hand for a particular approach that tends to focus on outcomes, objectivity, attribution and individual change. It is perhaps inevitable that most practitioners experience this form of measurement as externally imposed, though we should not overlook or discount those who feel it has brought a helpful challenge to their practice.
Our stance is that measurement is a fundamentally human activity that is woven into every aspect of our lives, and which helps us make sense of the world around us. We need to reclaim this broader understanding, alongside questioning the current drivers of impact measurement.
The specific challenges of measuring open youth provision are a call to arms not an excuse to down measurement tools
Do fluctuations in levels make something hard to measure? Not necessarily, and it depends entirely on what one is trying to measure and why. Do we care most about the ‘amount’ of change between point A and point B, or the journey along the way? How much do we care about understanding what influences the fluctuations? And is it possible to measure one thing that necessarily fluctuates according to context (such as confidence) alongside another that may slowly develop (like self-awareness)?
When one talks of measurement in relation to personal and social development, one is necessarily talking of perception. It is inevitably young people’s (individually and collectively) perception of themselves, and of the world around them. Simply asking about their perception has the potential to change it – and this process sits at the heart of most relational work with young people, and is to be welcomed and celebrated. It also demands measurement tools and approaches that are fit for purpose, and which go with rather than distort this process.
Some measurement practice is poor and meaningless
Much as assessing quality and process without one eye on outcomes (whether intended or unintended) can become a bureaucratic or meaningless exercise, too heavy-handed a focus on ‘delivering’ outcomes, particularly one driven by funding and PR pressures, too often detracts from and even distorts the quality of provision. This is a central concern at the Centre and we would see our role as supporting organisations and funders to move away from such narrow practice. Poor or meaningless measurement practice is not simply a waste of time: its effects reach much more widely than that. Understanding the interplay and relationship between process, conditions and outcomes matters enormously, but this importance is drowned out in performative impact measurement activity.
How we are taking these perspectives forward in the Centre’s work
Our forthcoming conference will address many of these issues explicitly, with a particular focus on what we do as a result. Tony’s critique is timely and important – we need to talk more about these perspectives – but we also need to shape practical responses.
We’ll be following up the conference with a series of blogs on where our work will focus in the next 12 months and beyond.
Pursuing any agenda related to impact measurement perhaps hangs on the question of whether we expect to see change as a result of our work. If yes, we remain as committed as ever to being clear about what we (the young person, the practitioner, and the external observer) hope that change will be, how we will know if it is happening, and what we can do to create the best conditions in which change might occur. We see a clear value in a focus on measurement but understand that these issues are value laden, highly contested and the source of much debate. A debate we are always happy to have.
In this blog, Bethia McNeil, Director of the Centre, reflects on how we can collectively shape a new path for evaluation and impact measurement. She argues that we need to do more than just think differently, we need to behave differently.
Social sector organisations would be forgiven for giving a weary eye roll at yet another invitation to ‘look to the future’. The promise of opportunities that might be on the horizon, just hidden from view, is a well-used trope within the third sector. Especially within reports and conferences. I can’t be the only person that mentally sings a Disney tune when they read about a ‘whole new world’ just around the corner.
So, why did we decide that it was a good idea to focus our forthcoming conference on ‘shaping the future’? Because, on this occasion, I believe it’s true. And it may well also be an actual opportunity.
All too often, the debate about impact measurement and evaluation is reduced to one of either technical skills (all you need to know is how to produce a theory of change, and which standardised questionnaire to use) or capacity (and I’m never entirely sure whether we all mean the same thing here). But it’s so much more than this. Not only are evaluation and impact more about culture and enquiry, but to focus primarily on building capacity and skills suggests that all we need to know is sitting somewhere, waiting to be passed on. And I just don’t think that’s how it is, nor how it should be.
Developing our collective understanding about how and why our work with young people contributes towards changes in their lives is not going to be achieved through doing more of what we’ve done so far. Will we reach nirvana when every youth charity has a theory of change? I doubt it.
What if we need to do things really quite differently to see things differently? What if we needed to try some approaches that have never been tried before, and stop doing some things that we’ve been doing for some time? And what if now was as good a time as any to do this? This is what our conference is about, and it marks a new phase of our work through which we want to make this a reality.
At our conference, and in our forthcoming work, we’ll be exploring the evolution of some ideas you will be familiar with, and also what happens when we bring different disciplines together. We will also look at a range of other conditions and contexts that shape evaluation and impact measurement: leadership, cultures of learning, truth claims and power.
I believe we need to set a new path for evaluation and impact measurement. Sometimes, it can be hard to turn back from a path we’ve travelled for some time (especially if lots of other people are on it too), and we must acknowledge this. But we also need to have the collective courage and openness to explore some different steps.
So whose responsibility is it to shape this future path? I think about this a lot, and I’m not sure. I think it’s our responsibility at the Centre for Youth Impact to create space to actively explore it, and to share ideas and resources that help us understand what the journey might involve, and where it might lead. But – to end as I began: on a cliché – this is your journey.
I look forward to seeing many of you at the conference on 11 September, and to working with you in the coming months.
The Centre's essay collection informs, challenges and encourages you to take your thinking one step further.
It is several years since the Select Committee Report on Services for Young People pushed the debate about evidence and impact into the mainstream.
For some, this was a new conversation; for others, it was a repeat of many that had gone before. The conversation shows no signs of going away, and has arguably only picked up pace. But where has the debate got to?
In 2016, The Centre for Youth Impact launched a new collection of essays to help us to take stock, understand the differing perspectives, and explore where the conversation has taken us. The essays brought together leading thinkers, practitioners and policy makers, along with young people who were reflecting on their role in this agenda. We hope that the essays will continue to inform, challenge and encourage you, taking your thinking one step further.
The first two essays, from Dr Nick Axford and Dr Genevieve Maitland Hudson, were published in April 2016.
Two further essays were published on 6 July 2016 at the Centre for Youth Impact Gathering 2016, from Jenny North and Tania De St Croix.
Since September 2016, five additional essays have been published via our monthly newsletter, from Kai Hopkins, Kaz Stuart and Steve Hillman, Emma Taylor, Asha Ali, Beca Sandu and Michael Little, and Jane Melvin.
You can read the essays along with many other useful tools in our Resource Hub.
Our monthly newsletter collects news, events, research and blogs from the Centre, our networks and practitioners and organisations around the world. Sign-up here