“Cracking the impact nut” or How the Youth Investment Fund learning and impact strand responds to the challenges of evaluation in open access provision
This blog has been written Matthew Hill, Head of Research and Learning at the Centre for Youth Impact.
Our YIF approach to data collection
NPC and the Centre for Youth Impact are leading the learning and impact strand of the £40 million Youth Investment Fund (YIF), which is a joint programme supported by government funding from DCMS and National Lottery funding from Big Lottery Fund. Eighty-six youth providers are being supported for three-years (2017-2020) to develop and expand their open access youth provision, and we are currently working them to design an evaluation approach that captures the value of their work, and supports their learning and improvement. Our overarching approach is an attempt to crack some of the perpetual practical and methodological nuts (my favourite bar snack) in measuring the impact of open access provision. This blog outlines five ways in which we are confronting these challenges within our work.
Moving away from blanket outcomes measurement
The past decade or so has seen a concerted push to be more outcomes focused. We continue to support an outcomes focused approach to service design and delivery (that is why we all do what we do after all) but our YIF-work represents a shift away from blanket outcome measurement (i.e. trying to capture every outcome for every young person for every organisation). This shift is a direct response to many of the perpetual challenges of outcome measurement in open access settings including fleeting or irregular engagement, defining generalised outcomes for individualised provision, developing robust metrics for broad personal change and the issue of measuring long term impacts. Far from abandoning outcome measurement, we are focusing on high quality targeted measurement with a sub-sample of the YIF cohort, which will ultimately provide us with more robust and meaningful data.
As well as this targeted approach to outcomes, our YIF-work places increased emphasis on the experience of young people and the quality of the provision they receive. Crucially, we are aiming to link the data on outputs, user feedback and quality to the targeted outcome data so we can understand not only whether the provision is having an impact on young people – but why.
Focusing on the user experience
Another challenge is that young people often feel overburdened with rather obscure and meaningless (to them at least) surveys. In response, our YIF-approach focuses on the elements of delivery that are most relevant and meaningful to young people – namely their experience of services (e.g. feelings of safety, respect and positive challenge). We are working with Keystone accountability to develop a set of standardised feedback questions around this experience. Instead of large annual surveys this feedback process uses regular light touch feedback - perhaps 3-5 questions once a month. This ensures that user feedback is embedded in ongoing reflective practice, and crucially, means that organisations can respond more immediately to the findings. Critically, we are also working with providers to process and act on this feedback, and tell young people what has changed as a result.
Improving as well as proving
Another nut that needs cracking is practitioners’ sense of dislocation between a lot of impact measurement and their everyday work. Part of our commitment to ‘going with the grain’ of provision is a focus on the quality of youth work practice. This data absolutely has to be linked to outcomes data – as ultimately this dictates what is and what isn’t quality provision – but by emphasising considerations of quality we are focusing on those elements of provision that are most relevant and meaningful to youth workers themselves. Our YIF work is drawing on an established quality improvement framework from the US – the Youth Program Quality Assessment – which relies on peer observation with youth workers identifying ‘markers’ of quality in the delivery of their colleagues. This framework is not a critique of existing quality assessment frameworks but is, in fact, a complement to them – ensuring quality is also monitored and increased as part of ongoing practice improvement rather than just assessed against an existing standard.
Understanding young people’s journey through services
Although most providers collect detailed attendance data, many tell us that they use this for monitoring overall service demand rather than truly understanding the way that individuals engage with their services. By utilising existing data and trialing new digital methods such as Yoti we aim to build a much more nuanced picture of what young people do with their feet i.e. how often they attend, for how long, and how they move through provision – as a proxy for their levels of engagement and ‘exposure’ to interactions.
Arguably the greatest opportunity presented by the YIF is the potential to collect shared data across 86 grantees for three years. This offers a rare (probably unique) opportunity to build an evidence base across a huge diversity of open access provision (detached/ building-based; structured/ unstructured; universal/ targeted) and, by comparing the results across different types of provision, we will be able to really understand the strengths and weaknesses of different services.
We recognise the many challenges that open access providers face and believe that the dominant paradigm of measurement is not fit for such settings. Our YIF work has the potential to overcome some of these challenges, and develop approaches that are applicable across the wider sector. It is certainly ambitious and we are trying new things out – some of which will work but some of which will no doubt fail. We will confront this uncertainty with a pioneering spirit and a humbleness to admit when things don’t work. As well as grantees we are committed to working with the wider sector and we would greatly value your input in testing, refining and reflecting upon the tools and evidence that emerges. Our dedicated YIF learning and impact website will be live soon… so please stay posted… or get in touch with Matthew.Hill@youthimpact.uk or Anoushka.Kenley@thinkNPC.org if you want to find out more in the meantime.
When treasuring is measuring, and why we might need a rethink
In this blog jointly written by Bethia McNeil, Pippa Knott and Matt Hill, the Centre’s core team responds to some of the key issues raised by Tony Taylor’s article regarding measurement in personal and social development. The Centre addresses the challenges found in the current dominant measurement framework and propose a rethink of the value of measurement in youth work.
Back in March this year, we hosted an event focused on measurement in personal and social development. We were really pleased to see Tony Taylor’s recent article in Youth and Policy, following up on the discussion, and agree that it would have been most beneficial had there been more time and space to explore the themes. Indeed, these themes are so vital that we felt moved to add our voice to Tony’s in this blog. Overall, we were struck at the many points where we agree with Tony’s forthright critique of the dominant paradigm in impact measurement, but there also remain some areas of fundamental disagreement – perhaps as might be expected in such a complex and contested area.
No measurement framework is ideologically neutral
We agree wholeheartedly that the theory and practice of measurement is never neutral - and arguably, neither should it be. Accepting this latter point would create more space for critical reflection on the inevitable ‘positioning’ of existing practices and frameworks, rather than an illusory search for independence and objectivity. Equally, focusing measurement efforts on the potentially less contested area of ‘skills’ as opposed to character, awareness or consciousness does nothing to make neutrality any more likely.
Any measurement framework involves collating, distilling and selecting information. This process is influenced by the background, experience and disposition of the individual or team carrying out the task, and the broader political and relational context in which they are acting. We are learning from other sectors - even those more historically predisposed towards the 'scientific method' - where people are equally questioning whether we can really rely on scientific robustness to neutralise values and context. Yet we disagree that failing the objectivity test is fatal. Instead, the best research and measurement acknowledges context and bias and embeds a constant reflection into research practice, including consideration of the influence on relationships ‘in the moment’. This has the potential to result in an enhanced, not a diminished science and practice.
The current dominant measurement framework systematically undervalues certain forms of activity, and privileges others
Open youth work, a term usefully endorsed by Tania de St Croix, sits at odds with many dominant forms of measurement, not least due to a (laudable) resistance among practitioners to impose pre-determined measurable ‘outcomes’ upon young people. We believe that we absolutely need to find ways of gathering, interpreting and sharing data about this approach to engaging with young people that both enhances practice and generates meaningful evidence and insight. We do not believe that this should be led by the pursuit of funding (particularly given that, as Tony and others have noted, the relationship between evidence and funding is nowhere near as straightforward as we might think/hope), or capitulation to broader forces that can seem overwhelming. We believe that this is about learning, improving, developing and advocating.
At the Centre we do not define the youth sector tightly. In fact, we sometimes deliberately don’t define it at all, preferring to work with and build shared understanding and learning among as many organisations engaging with and supporting young people as possible. But this is dependent at least in part on the ability to talk meaningfully and collectively about our practice. We agree that it might be harder to ‘measure’ the impact of youth work than other more targeted or narrowly defined forms of work with young people – but, for us, this demands that we develop how we measure and understand what really counts about youth work, and via a process that enriches rather than undermines practice. We should also pause to reflect on why we feel it’s ‘harder’ to measure impact in (particularly open) youth work: harder fundamentally, or harder to fit within the dominant paradigm?
There are broader conceptions of measurement out there
We agree the current dominant paradigm of measurement in the social sector has emerged within the neo-liberal landscape and shares (and actively perpetuates) many of its key features - monetisation, marketisation and competition to name a few. But neo-liberalism doesn’t have a monopoly on measurement and indeed there are many participatory and emancipatory methodologies of ‘measurement’ that are fundamentally opposed to it. There is a risk that we throw out the ‘measurement’ baby with the neo-liberal bathwater. Instead we believe that effective measurement of open youth work is a crucial bulwark against narrow conceptions of value. ‘Measurement’ in the current parlance has become short-hand for a particular approach that tends to focus on outcomes, objectivity, attribution and individual change. It is perhaps inevitable that most practitioners experience this form of measurement as externally imposed, though we should not overlook or discount those who feel it has brought a helpful challenge to their practice.
Our stance is that measurement is a fundamentally human activity that is woven into every aspect of our lives, and which helps us make sense of the world around us. We need to reclaim this broader understanding, alongside questioning the current drivers of impact measurement.
The specific challenges of measuring open youth provision are a call to arms not an excuse to down measurement tools
Do fluctuations in levels make something hard to measure? Not necessarily, and it depends entirely on what one is trying to measure and why. Do we care most about the ‘amount’ of change between point A and point B, or the journey along the way? How much do we care about understanding what influences the fluctuations? And is it possible to measure one thing that necessarily fluctuates according to context (such as confidence) alongside another that may slowly develop (like self-awareness)?
When one talks of measurement in relation to personal and social development, one is necessarily talking of perception. It is inevitably young people’s (individually and collectively) perception of themselves, and of the world around them. Simply asking about their perception has the potential to change it – and this process sits at the heart of most relational work with young people, and is to be welcomed and celebrated. It also demands measurement tools and approaches that are fit for purpose, and which go with rather than distort this process.
Some measurement practice is poor and meaningless
Much as assessing quality and process without one eye on outcomes (whether intended or unintended) can become a bureaucratic or meaningless exercise, too heavy-handed a focus on ‘delivering’ outcomes, particularly one driven by funding and PR pressures, too often detracts from and even distorts the quality of provision. This is a central concern at the Centre and we would see our role as supporting organisations and funders to move away from such narrow practice. Poor or meaningless measurement practice is not simply a waste of time: its effects reach much more widely than that. Understanding the interplay and relationship between process, conditions and outcomes matters enormously, but this importance is drowned out in performative impact measurement activity.
How we are taking these perspectives forward in the Centre’s work
Our forthcoming conference will address many of these issues explicitly, with a particular focus on what we do as a result. Tony’s critique is timely and important – we need to talk more about these perspectives – but we also need to shape practical responses.
We’ll be following up the conference with a series of blogs on where our work will focus in the next 12 months and beyond.
Pursuing any agenda related to impact measurement perhaps hangs on the question of whether we expect to see change as a result of our work. If yes, we remain as committed as ever to being clear about what we (the young person, the practitioner, and the external observer) hope that change will be, how we will know if it is happening, and what we can do to create the best conditions in which change might occur. We see a clear value in a focus on measurement but understand that these issues are value laden, highly contested and the source of much debate. A debate we are always happy to have.
Our monthly newsletter collects news, events, research and blogs from the Centre, our networks and practitioners and organisations around the world. Sign-up here