Standing up for being counted

When treasuring is measuring, and why we might need a rethink
In this blog jointly written by Bethia McNeil, Pippa Knott and Matt Hill, the Centre’s core team responds to some of the key issues raised by Tony Taylor’s article regarding measurement in personal and social development. The Centre addresses the challenges found in the current dominant measurement framework and propose a rethink of the value of measurement in youth work. 

Back in March this year, we hosted an event focused on measurement in personal and social development. We were really pleased to see Tony Taylor’s recent article in Youth and Policy, following up on the discussion, and agree that it would have been most beneficial had there been more time and space to explore the themes. Indeed, these themes are so vital that we felt moved to add our voice to Tony’s in this blog. Overall, we were struck at the many points where we agree with Tony’s forthright critique of the dominant paradigm in impact measurement, but there also remain some areas of fundamental disagreement – perhaps as might be expected in such a complex and contested area.

No measurement framework is ideologically neutral
We agree wholeheartedly that the theory and practice of measurement is never neutral - and arguably, neither should it be. Accepting this latter point would create more space for critical reflection on the inevitable ‘positioning’ of existing practices and frameworks, rather than an illusory search for independence and objectivity. Equally, focusing measurement efforts on the potentially less contested area of ‘skills’ as opposed to character, awareness or consciousness does nothing to make neutrality any more likely.

Any measurement framework involves collating, distilling and selecting information. This process is influenced by the background, experience and disposition of the individual or team carrying out the task, and the broader political and relational context in which they are acting. We are learning from other sectors - even those more historically predisposed towards the 'scientific method' - where people are equally questioning whether we can really rely on scientific robustness to neutralise values and context. Yet we disagree that failing the objectivity test is fatal. Instead, the best research and measurement acknowledges context and bias and embeds a constant reflection into research practice, including consideration of the influence on relationships ‘in the moment’. This has the potential to result in an enhanced, not a diminished science and practice.

The current dominant measurement framework systematically undervalues certain forms of activity, and privileges others
Open youth work, a term usefully endorsed by Tania de St Croix, sits at odds with many dominant forms of measurement, not least due to a (laudable) resistance among practitioners to impose pre-determined measurable ‘outcomes’ upon young people. We believe that we absolutely need to find ways of gathering, interpreting and sharing data about this approach to engaging with young people that both enhances practice and generates meaningful evidence and insight. We do not believe that this should be led by the pursuit of funding (particularly given that, as Tony and others have noted, the relationship between evidence and funding is nowhere near as straightforward as we might think/hope), or capitulation to broader forces that can seem overwhelming. We believe that this is about learning, improving, developing and advocating.

At the Centre we do not define the youth sector tightly. In fact, we sometimes deliberately don’t define it at all, preferring to work with and build shared understanding and learning among as many organisations engaging with and supporting young people as possible. But this is dependent at least in part on the ability to talk meaningfully and collectively about our practice. We agree that it might be harder to ‘measure’ the impact of youth work than other more targeted or narrowly defined forms of work with young people – but, for us, this demands that we develop how we measure and understand what really counts about youth work, and via a process that enriches rather than undermines practice. We should also pause to reflect on why we feel it’s ‘harder’ to measure impact in (particularly open) youth work: harder fundamentally, or harder to fit within the dominant paradigm?

There are broader conceptions of measurement out there 
We agree the current dominant paradigm of measurement in the social sector has emerged within the neo-liberal landscape and shares (and actively perpetuates) many of its key features - monetisation, marketisation and competition to name a few. But neo-liberalism doesn’t have a monopoly on measurement and indeed there are many participatory and emancipatory methodologies of ‘measurement’ that are fundamentally opposed to it. There is a risk that we throw out the ‘measurement’ baby with the neo-liberal bathwater. Instead we believe that effective measurement of open youth work is a crucial bulwark against narrow conceptions of value. ‘Measurement’ in the current parlance has become short-hand for a particular approach that tends to focus on outcomes, objectivity, attribution and individual change. It is perhaps inevitable that most practitioners experience this form of measurement as externally imposed, though we should not overlook or discount those who feel it has brought a helpful challenge to their practice.

Our stance is that measurement is a fundamentally human activity that is woven into every aspect of our lives, and which helps us make sense of the world around us. We need to reclaim this broader understanding, alongside questioning the current drivers of impact measurement.

The specific challenges of measuring open youth provision are a call to arms not an excuse to down measurement tools
Do fluctuations in levels make something hard to measure? Not necessarily, and it depends entirely on what one is trying to measure and why. Do we care most about the ‘amount’ of change between point A and point B, or the journey along the way? How much do we care about understanding what influences the fluctuations? And is it possible to measure one thing that necessarily fluctuates according to context (such as confidence) alongside another that may slowly develop (like self-awareness)?
​
When one talks of measurement in relation to personal and social development, one is necessarily talking of perception. It is inevitably young people’s (individually and collectively) perception of themselves, and of the world around them. Simply asking about their perception has the potential to change it – and this process sits at the heart of most relational work with young people, and is to be welcomed and celebrated. It also demands measurement tools and approaches that are fit for purpose, and which go with rather than distort this process.

Some measurement practice is poor and meaningless
Much as assessing quality and process without one eye on outcomes (whether intended or unintended) can become a bureaucratic or meaningless exercise, too heavy-handed a focus on ‘delivering’ outcomes, particularly one driven by funding and PR pressures, too often detracts from and even distorts the quality of provision. This is a central concern at the Centre and we would see our role as supporting organisations and funders to move away from such narrow practice. Poor or meaningless measurement practice is not simply a waste of time: its effects reach much more widely than that. Understanding the interplay and relationship between process, conditions and outcomes matters enormously, but this importance is drowned out in performative impact measurement activity.

How we are taking these perspectives forward in the Centre’s work
Our forthcoming conference will address many of these issues explicitly, with a particular focus on what we do as a result. Tony’s critique is timely and important – we need to talk more about these perspectives – but we also need to shape practical responses.
​
We’ll be following up the conference with a series of blogs on where our work will focus in the next 12 months and beyond.

Pursuing any agenda related to impact measurement perhaps hangs on the question of whether we expect to see change as a result of our work. If yes, we remain as committed as ever to being clear about what we (the young person, the practitioner, and the external observer) hope that change will be, how we will know if it is happening, and what we can do to create the best conditions in which change might occur. We see a clear value in a focus on measurement but understand that these issues are value laden, highly contested and the source of much debate. A debate we are always happy to have.