This blog was written by Pippa Knott, Head of Networks at the Centre for Youth Impact
Working with the Talent Match partnerships has given me an amazing opportunity to reflect on the power of relationships in the context of an employability programme. We’re now focusing on what the Centre’s role might be in strengthening and promoting those that sit at the heart of youth work and other provision for young people.
It’s an issue that has woven through much of our previous work: Robin Bannerjee gave a very well-received presentation at our first event of 2017, discussing approaches to measuring personal development in the context of relationships, and the team at Dartington Social Research Unit (now Policy Lab) wrote for us on the place of relationships in social provision.
Through Talent Match, I’m reminded again of how supportive relationships at their best are hugely powerful – sometimes transformative – and often at the heart of programmes that are ‘working well’. They’re also as complicated as the combination of the individuals who make them up. Many people are happy to ‘feel their way’ through relationships, drawing on past experience and what seems right in the moment. This applies as much to relationships within services and other provision as relationships elsewhere. So devising a framework capturing how to ‘do them well’ is difficult: it can quickly feel an academic, even unhelpful exercise, unlikely to be valued and used by practitioners. Watch this space for how I’m working with the partnerships to try and progres some of these issues! Full findings from the project will be launched in March.
The way in which relationships recur in our work at the Centre also suggests it might not make sense to think of them as a ‘topic’. Instead, they could be a crucial piece in the puzzle of how we can support others to flourish, while also reflecting on ourselves and what we’re bringing to any situation.
Thinking only about the relationship between adult professional or volunteer and young person might also be limiting. I’m also thinking about how we can make the best of the relationships upon which the Centre exists: within the central team, between us and our networks, and across the web of organisations, connections and friendships within which we work. Relationships are a key mechanism for development and support in the overwhelmingly complex systems we live within, and something that any individual can learn about and use to affect change in their own lives, and the lives of others.
What might it look like if we invested in relationships for social good, rather than in organisations or programmes? Are our current methods and frameworks for measuring impact in work with youth people sufficient to take account for the nuances, complexities and potential impact of positive relationships? Do they tell us enough about how relationships can be improved? How can we develop them if so? We’ll be learning from youth work principles and practice [for example, Relationship, Learning and Education; Benefits of Youth Work; Grassroots Youth Work] and the work of the Search Institute, the R-Word, and Lankelly Chase as we develop our approach in this area.
Pippa Knott is Head of Networks at the Centre for Youth Impact. She is part of a small team that has established and developed the Centre, and has talked increasingly about creating the conditions for meaningful impact measurement. This requires shifts in culture and behaviour as much as developments in methodological design and research tools.
Will Millard is a Senior Associate at the education and youth think and action tank, LKMco. LKMco works across the education, youth and policy sectors, helping organisations develop and evaluate projects for young people, and carrying out academic and policy research. LKMco seeks to help practitioners and organisations working with young people develop and use stronger evidence to help enhance their impact.
The Centre for Youth Impact and LKMco are both part of (overlapping) evidence movements, and want to play our part in helping people across our respective sectors engage with evidence and promote its place in improving service design, quality, and sustainability.
What is the challenge?
Why does talk of evidence and impact excite some people while alienating others? We (Pippa and Will) met to talk about the similarities and differences between the demand for and use of evidence in formal (school- and classroom-based), informal and non-formal educational and youth settings. Specifically, we discussed the conditions under which ResearchEd, a teacher-led ‘evidence movement’, was established and has since grown, and parallels with the evolution of the Centre for Youth Impact, which supports networks of practitioners across England to develop their use of evidence to improve out-of-school provision for young people.
We identified three influences on the culture, growth and ‘successes’ of both ResearchEd and the work of the Centre for Youth Impact, and engagement with evidence more generally:
What is ResearchEd?
ResearchEd seeks to help teachers share educational research and raise evidence ‘literacy’ through events run across the UK and internationally. It was originally intended to be a one-off event in 2013 but has since become an international movement. Describing ResearchEd as a “grass-roots” organisation, founder and director Tom Bennett says, “it wanted to be built. It built itself”. Some, though, are sceptical whether the term ‘grass-roots’ accurately describes ResearchEd, yet as Debra Kidd argues: “What matters is that it’s here. It’s an opportunity. And it’s there for teachers to make of it what they will.”
What is the Centre for Youth Impact?
The Centre for Youth Impact exists to support organisations working with young people in informal/non-formal settings to improve how they generate and use evidence. It was set up with funding from central government, but has always aspired to be owned and led by the sector. The Centre is perceived by some sector stakeholders as an arms-length and external influence on practice, and so cannot be said to be a grassroots movement. However, a focus on building networks across the country and relationships with practitioners is now beginning to generate a community and momentum that goes beyond the core team.
In many ways, ResearchEd has historically been the opposite of the Centre: the former having a tiny core with a large following, and the latter starting life as a slightly larger ‘institution’ but attempting to build a following once it came into existence.
Why do we ‘do’ impact measurement, and why do those working in formal and informal and non-formal settings sometimes view this differently? We think that part of the answer lies in the relationship between evidence and accountability.
Teachers associate accountability with Ofsted, the Department for Education, line managers, and tests and exams. LKMco’s recent report on assessment found that teachers often feel accountability is something done to them, rather than with them (similar language to that used to describe the experience of evaluation in the youth sector). Consequently, ResearchEd and other events and networks including Northern Rocks, TeachMeet, and #BAMEed – precisely because they are seen as grass-roots movements organised by and for teachers – represent a means by which to reassert professional identity and authority.
Furthermore, there is a widespread sense that evidence on which decisions about teachers’ and schools’ performance has been based has at times been dubious, as was explored here, here and here. ResearchEd tapped into teachers’ desire to raise the quality of discussion at all levels, not just about what learning is for and how it can be best supported, but also how it can be evaluated. ResearchEd is by no means the only expression of this desire: nearly one third of schools in England have taken part in one or more Education Endowment Foundation-funded trials, and the Chartered College of Teaching has made sharing evidence one of its core aims.
By way of contrast, the growth of the evidence and impact agenda as it is currently constituted has been associated in parts of the youth sector with a certain form of ‘high stakes’ accountability, and ‘proving’ worth or value to others. Our experience is that the current impact agenda is too often divorced from the processes of reflection, learning and practice improvement that to many are – or should be – inherent in youth work practice, and other forms of informal and non-formal education with young people.
An Education Select Committee report published in 2011 was a notable point at which evidence of effectiveness was posited as the key to sustainability of provision. In subsequent years, funding has remained the primary pressure facing youth workers and managers, and continues to be associated with a pressure to ‘prove’ impact to funders, commissioners and policy makers. So it’s hardly surprising that for many managers and practitioners in youth organisations, the concept of evidence generation and data gathering remains firmly coupled with the pressures of securing and managing funding, and externally-imposed accountability processes. A widespread scepticism that funding and commissioning decisions are not as ‘evidence informed’ as one might wish has done little to increase the value of impact measurement in the minds of many practitioners.
There is a critical eye on the nature and use of evidence in the youth sector, and many would say that it bears little relevance to either practice or the lives of young people. Some have referred to the increasing desire to focus on ‘what works’ in developing young people’s social and emotional capabilities as ‘pseudo-science’. But the corresponding drive of the youth sector - though it might also be seen to be about raising the quality of discussion – has been more ambivalent about the status of research and evidence in its work. The youth sector has certainly not seen such polarization amongst practitioners as in the formal education world, where debates between the ‘trads’ and the ‘progressives’ can get fairly spicy. It is unclear how many youth providers would sign up to be part of research trials, but one suspects it would not be many – certainly not with parameters similar to those conducted by EEF.
There’s no denying ResearchEd’s growing influence. Senior members of the inspectorate, Civil Service, and government appear to be queuing up to present at the events. This might be a mixed blessing, though. The presence of senior politicians reflects how important it now is to associate with the evidence agenda in education. It also runs the risk of ResearchEd being seen as a conduit through which ministers can exercise their personal ideologies or, worse, an event that actively ostracises those whose views or approaches differ. Bennett, as the founder of ResearchEd, is aware of this challenge, and has publicly talked about steps ResearchEd has taken to remain inclusive (of both people and their ideas). However, ResearchEd’s ongoing success, and more importantly its impact on classroom practice, will depend on it striking the right balance between working with those at the top of the education system while championing the experiences of everyday teachers.
The Centre’s influence is also growing, though it is probably some way off the momentum and international profile of ResearchEd! Meanwhile, the Centre’s team is continuing to challenge and interrogate what is the best trajectory for the Centre. Currently, it feels more meaningful to create space for inclusive debate, with practitioners supported to lead debate, than to provide a platform for politicians.
So what, then, can we do to make evidence and research synonymous with empowerment for all professionals working with young people? We are finding that:
Professional learning and development
ResearchEd, like the Chartered College of Teaching, Teacher Development Trust, Institute for Teaching, and EEF, in part reflects a fundamental belief that teaching is something that can be deconstructed, honed and improved over time. Teachers are made, not born. In their respective ways, these organisations reflect teachers’ desire to share their practice, and learn from others. Testament to this have been the high turnouts at ResearchEd events, and teachers moving heaven and earth to attend.
Alongside a deep-rooted belief that coming together to deconstruct and evaluate practice is an inherently good thing to do, teachers have also felt the need to increase their evidence ‘literacy’ to guard against the false promises of ‘snake oil salesmen’ (as Bennett explores, here and here). Sharing research and evidence can therefore generate and bolster good ideas, and reduce the prevalence of bad ones.
At the heart of traditional youth work is the relationship between the youth worker and the young person – a relationship that is voluntary, led by the young person, starts where the young person is, centres on their individuality, networks and community and cultural identities, and focuses on how they feel as well as what they know and can do. The ‘what works’ debate tends to lead to a focus on interventions – programmes or ‘methods’ of working with young people. Arguably the importance of the relationship is missed in some of this thinking. Could we use the evidence agenda help us better understand and communicate what is at the heart of effective work with young people – the relationship between youth worker and young person as a ‘mechanism of change’ in that young person’s life, and their broader community? How can we reboot and inform learning and development – something increasingly neglected in an underfunded and fragmented youth sector? Perhaps in contrast to the parts of the teaching professional body, the youth sector as a whole is not harnessing evidence to challenge ‘fads’ and attempt to strengthen itself and rise up. It has effectively rejected or resisted it instead..
We are learning that:
Networks (and particularly social media)
In addition to the role that evidence does and has historically played in professional experience, it’s possible that the structures and processes of networks themselves have had an influence in the evolution of our evidence movement.
Networks – and particularly those on Twitter – have been fundamental in ResearchEd’s success. As Bennett has explained, ResearchEd grew out of a single tweet, posted late one evening in 2013. The first conference in September that same year attracted over 500 people. Ever since, ResearchEd has used the combination of events and social media to expand its reach. Likewise, grassroots events such as Beyond Levels and TeachMeet also quickly gained traction through a budding online community.
Of course, it would be naïve to pretend social media is hassle free. Alongside its many benefits, social media can sometimes be any or all of reactive, aggressive, cliquey, and, consequently, intimidating for new (and sometimes even veteran) users. This is an inherent challenge associated with using social media platforms, but has practical ramifications for ResearchEd, which must probe at deeply held ideas while remaining constructive.
The Centre for Youth Impact’s networks have evolved primarily offline – through a small core team, and regional networks, whose leaders received a small fee to contribute towards administration, and help them widen their reach. We have reached out through and invested in existing networks where possible, seeking to reflect the views of practitioners on the ground while drawing together a diverse range of experiences and perspectives from across a fragmented sector. Building trust and perceptions of time well spent, face to face and in small groups, has been critical to the sustainability of the Centre.
There’s clearly no ‘right’ or ‘wrong’ way to establish and grow networks. Funding to support the development of regional networks supports the sharing of ideas locally, but perhaps lacks an overarching, galvanising force. On the other hand, engagement through social media is energetic and organic, but could isolate teachers who do not have, or feel under-confident using online accounts or ‘speaking out’ in this forum.
The emergence of ResearchEd (and other events and networks) indicates:
Where do we go from here?
Our conversation highlighted to us that when promoting evidence and impact to our respective audiences we need to more explicitly acknowledge how systems of accountability affect practitioners; how and where practitioners look for inspiration; and how networks support empowerment.
The evidence movements surrounding both the Centre for Youth Impact and ResearchEd differ significantly. This may not be due to a fundamental lack of interest or engagement in the youth sector, but because ‘evidence’ is perhaps seen as another stick with which to beat an already weakened profession. In addition, there are arguably philosophical and pedagogical differences between teachers and youth practitioners, influencing how each group engages with research and evidence. In theory, we recognise huge potential in the youth sector for a grassroots-led movement that allows practitioners to ‘reclaim’ the evidence agenda, but in reality, we must recognise the impact of political pressures and resource limitations on people’s willingness to engage.
Both the Centre for Youth Impact and LKMco intend to support practitioners to embrace evidence as a means by which to celebrate their achievements, while honing and refining their craft. We will continue to acknowledge that research and use of evidence will always be a political activity, but strive for openness and clarity about why generating evidence is important, and who stands to benefit and learn from it.
Furthermore, there is potential for more widespread voluntary online communities to develop in the informal and non-formal educational sectors, giving a wider range of practitioners a say, and engaging a wider range of people. However, we see roles for both online and offline discussion spaces in helping build these movements.
When treasuring is measuring, and why we might need a rethink
In this blog jointly written by Bethia McNeil, Pippa Knott and Matt Hill, the Centre’s core team responds to some of the key issues raised by Tony Taylor’s article regarding measurement in personal and social development. The Centre addresses the challenges found in the current dominant measurement framework and propose a rethink of the value of measurement in youth work.
Back in March this year, we hosted an event focused on measurement in personal and social development. We were really pleased to see Tony Taylor’s recent article in Youth and Policy, following up on the discussion, and agree that it would have been most beneficial had there been more time and space to explore the themes. Indeed, these themes are so vital that we felt moved to add our voice to Tony’s in this blog. Overall, we were struck at the many points where we agree with Tony’s forthright critique of the dominant paradigm in impact measurement, but there also remain some areas of fundamental disagreement – perhaps as might be expected in such a complex and contested area.
No measurement framework is ideologically neutral
We agree wholeheartedly that the theory and practice of measurement is never neutral - and arguably, neither should it be. Accepting this latter point would create more space for critical reflection on the inevitable ‘positioning’ of existing practices and frameworks, rather than an illusory search for independence and objectivity. Equally, focusing measurement efforts on the potentially less contested area of ‘skills’ as opposed to character, awareness or consciousness does nothing to make neutrality any more likely.
Any measurement framework involves collating, distilling and selecting information. This process is influenced by the background, experience and disposition of the individual or team carrying out the task, and the broader political and relational context in which they are acting. We are learning from other sectors - even those more historically predisposed towards the 'scientific method' - where people are equally questioning whether we can really rely on scientific robustness to neutralise values and context. Yet we disagree that failing the objectivity test is fatal. Instead, the best research and measurement acknowledges context and bias and embeds a constant reflection into research practice, including consideration of the influence on relationships ‘in the moment’. This has the potential to result in an enhanced, not a diminished science and practice.
The current dominant measurement framework systematically undervalues certain forms of activity, and privileges others
Open youth work, a term usefully endorsed by Tania de St Croix, sits at odds with many dominant forms of measurement, not least due to a (laudable) resistance among practitioners to impose pre-determined measurable ‘outcomes’ upon young people. We believe that we absolutely need to find ways of gathering, interpreting and sharing data about this approach to engaging with young people that both enhances practice and generates meaningful evidence and insight. We do not believe that this should be led by the pursuit of funding (particularly given that, as Tony and others have noted, the relationship between evidence and funding is nowhere near as straightforward as we might think/hope), or capitulation to broader forces that can seem overwhelming. We believe that this is about learning, improving, developing and advocating.
At the Centre we do not define the youth sector tightly. In fact, we sometimes deliberately don’t define it at all, preferring to work with and build shared understanding and learning among as many organisations engaging with and supporting young people as possible. But this is dependent at least in part on the ability to talk meaningfully and collectively about our practice. We agree that it might be harder to ‘measure’ the impact of youth work than other more targeted or narrowly defined forms of work with young people – but, for us, this demands that we develop how we measure and understand what really counts about youth work, and via a process that enriches rather than undermines practice. We should also pause to reflect on why we feel it’s ‘harder’ to measure impact in (particularly open) youth work: harder fundamentally, or harder to fit within the dominant paradigm?
There are broader conceptions of measurement out there
We agree the current dominant paradigm of measurement in the social sector has emerged within the neo-liberal landscape and shares (and actively perpetuates) many of its key features - monetisation, marketisation and competition to name a few. But neo-liberalism doesn’t have a monopoly on measurement and indeed there are many participatory and emancipatory methodologies of ‘measurement’ that are fundamentally opposed to it. There is a risk that we throw out the ‘measurement’ baby with the neo-liberal bathwater. Instead we believe that effective measurement of open youth work is a crucial bulwark against narrow conceptions of value. ‘Measurement’ in the current parlance has become short-hand for a particular approach that tends to focus on outcomes, objectivity, attribution and individual change. It is perhaps inevitable that most practitioners experience this form of measurement as externally imposed, though we should not overlook or discount those who feel it has brought a helpful challenge to their practice.
Our stance is that measurement is a fundamentally human activity that is woven into every aspect of our lives, and which helps us make sense of the world around us. We need to reclaim this broader understanding, alongside questioning the current drivers of impact measurement.
The specific challenges of measuring open youth provision are a call to arms not an excuse to down measurement tools
Do fluctuations in levels make something hard to measure? Not necessarily, and it depends entirely on what one is trying to measure and why. Do we care most about the ‘amount’ of change between point A and point B, or the journey along the way? How much do we care about understanding what influences the fluctuations? And is it possible to measure one thing that necessarily fluctuates according to context (such as confidence) alongside another that may slowly develop (like self-awareness)?
When one talks of measurement in relation to personal and social development, one is necessarily talking of perception. It is inevitably young people’s (individually and collectively) perception of themselves, and of the world around them. Simply asking about their perception has the potential to change it – and this process sits at the heart of most relational work with young people, and is to be welcomed and celebrated. It also demands measurement tools and approaches that are fit for purpose, and which go with rather than distort this process.
Some measurement practice is poor and meaningless
Much as assessing quality and process without one eye on outcomes (whether intended or unintended) can become a bureaucratic or meaningless exercise, too heavy-handed a focus on ‘delivering’ outcomes, particularly one driven by funding and PR pressures, too often detracts from and even distorts the quality of provision. This is a central concern at the Centre and we would see our role as supporting organisations and funders to move away from such narrow practice. Poor or meaningless measurement practice is not simply a waste of time: its effects reach much more widely than that. Understanding the interplay and relationship between process, conditions and outcomes matters enormously, but this importance is drowned out in performative impact measurement activity.
How we are taking these perspectives forward in the Centre’s work
Our forthcoming conference will address many of these issues explicitly, with a particular focus on what we do as a result. Tony’s critique is timely and important – we need to talk more about these perspectives – but we also need to shape practical responses.
We’ll be following up the conference with a series of blogs on where our work will focus in the next 12 months and beyond.
Pursuing any agenda related to impact measurement perhaps hangs on the question of whether we expect to see change as a result of our work. If yes, we remain as committed as ever to being clear about what we (the young person, the practitioner, and the external observer) hope that change will be, how we will know if it is happening, and what we can do to create the best conditions in which change might occur. We see a clear value in a focus on measurement but understand that these issues are value laden, highly contested and the source of much debate. A debate we are always happy to have.
Our monthly newsletter collects news, events, research and blogs from the Centre, our networks and practitioners and organisations around the world. Sign-up here