What's in a word, part 2: what do we mean when we talk about outcomes?

This blog is written by Mary McKaskill, Practice Development Manager, and Bethia McNeil, CEO, as a follow up to their recent blog on ‘the impact lexicon’.

One of the questions we are most commonly asked at the Centre is “I just need to demonstrate my outcomes. Can you recommend a tool?”. There are many reasons why this question is problematic, but a big issue is that - like the term ‘impact’ - ‘outcomes’ too have taken on various meanings and intentions through widespread use (and misuse). This influences how ‘outcomes’ as a concept is understood and how it functions, or is put into practice.
 
So what do we mean when we talk about outcomes? Our sense is that we’re mainly referring to two things: firstly, the change that we hope we’re (co-)creating with and for people and communities (the theoretical framing) and secondly, the terms in which we communicate and ‘capture’ this (the functional framing).
 
Let’s start with the theoretical framing of outcomes. The change that organisations hope to achieve for people and communities is an enormously varied collection of aims and intentions, which will occur or develop over very different time frames with a range of different influences. In our experience, people and communities create outcomes for themselves. In our pre-Centre direct work with young people, we’ve never ‘got someone’ a job, or ‘made them’ more confident. We’ve never sustained anyone’s tenancy or ensured that they were financially literate. More to the point, even if we had done – or more likely, contributed to - any of those things, we certainly didn’t do it on our own. A range of other people, organisations and agencies were involved. These ‘outcomes’ are cumulative changes that are subject to the agency of people and communities.
 
‘Outcomes’, in social sector parlance, have come to mean so much more. Outcomes are desirable, positive things that happen to people, and sometimes communities (though we collectively find that a bit more challenging to talk about with confidence). Outcomes are usually intended, and can be created and ‘delivered’ by organisations. They can be counted and measured, and credited to one programme, project or organisation. They can sometimes be valued, or given a financial proxy. They can be recorded, evidenced, tracked, monitored and demonstrated. They can be felt, although ironically not always by the people experiencing them: we often ‘just know’ when we’re ‘creating positive outcomes’.
 
People and communities ‘have’ outcomes, though they might not always know it. Young people in particular may experience many, many outcomes on a monthly or quarterly basis – sometimes the same outcome multiple times, claimed by lots of different organisations. Organisations can get paid on the strength of these outcomes.
 
Outcomes can be short-term, medium-term or long-term. They can be intermediate, proximal or distal. They can be soft or hard. They can be skills, attitudes, behaviours, feelings, the act of doing a thing (like getting or keeping a job) or not doing a thing (like committing a crime).
 
Outcomes might be in the interests of individuals, communities, funders and commissioners or the state, though rarely all at the same time. No one is quite sure who outcomes belong to.
 
So many, many words. It makes little sense to refer to or frame all of this as an ‘outcome’. This theoretical framing of outcomes simultaneously overstates and understates our own role in bringing outcomes ‘into being’ and creates an artificial (and easily contested) premise for measurement.
 
The ‘outcome’ as a catch-all term both encourages a lack of specificity in debate and practice, and eclipses lots of other useful concepts, like understanding with whom you’re engaging and how – with diligence and curiosity – and interrogating the quality of your practice for continuous improvement. Ironically, these are the things that give outcomes data real meaning. Without a focus on what you’re doing and with whom, ‘outcomes’ become more and more meaningless. ‘Outcomes’ have become less a concept that helps us to advance the evidence and impact debate in the youth sector, and more a seductively imprecise idea that undermines and stalls progress.
 
All social programmes are doing something ­– and that ‘something’ will be affecting something else. This is sound theory, though that doesn’t make it any less complicated in the real world. But not knowing exactly what that something has affected is not an excuse for a) not being curious about finding out and b) not being intentional about your practice in the first place. Sadly, the catch all nature of the ‘outcome’ crowds both curiosity and intentionality out. By maintaining that outcomes are the direct result of our work, and often our work alone, we collude with an over-simplified fudge in how we explain and understand what we do with and for individuals and communities, whilst simultaneously paying less attention to what we actually are doing. By the same token, claiming that the complexity and messiness of the real world means we can’t say anything meaningful about outcomes is also deeply unhelpful.  
 
So let’s think about outcomes functionally. The current theoretical framing of outcomes (an ill-defined mix of aims and intentions, usually with some social and emotional constructs alongside) drives an entirely dysfunctional and ultimately pretty meaningless approach to ‘measuring outcomes’, which is easily undermined. Part of an organisation claiming that it ‘has positive outcomes’ is a fairly simplistic approach to counting them, like units. Ironically, given that it’s usually the organisation that wants to claim the outcomes, most often it’s young people who are asked to complete the surveys to count them.
 
And yet, ‘outcomes’ are regarded as powerful communication mechanisms and an impetus for funding. Often the request for a “tool to demonstrate outcomes” appears to come from the position that “we know that outcomes are being achieved, and so we need a tool that will show others in a convincing way that we’re achieving outcomes too”. ‘Outcomes’ as a communication mechanism has fed defensiveness and a drive to ‘prove impact’, as well as clouded collective understanding of practice and how it is experienced by young people. This desire to ‘demonstrate’ outcomes implies that the data’s primary audience is external rather than internal and it places ‘ownership’ over outcomes on the organisations and practitioners (i.e. adults) rather than young people. It also tends to determine the approach to measurement: one that is the least burdensome and disruptive to organisations, and which produces the nicest and most compelling findings.
 
The language used to describe youth provision often sounds like a sales pitch. “People buy benefits, not products” is hammered home in any Sales 101 course. We have shifted our collective focus entirely to the ‘benefits’ of provision at the expense of understanding the ‘product’: what high quality practice looks and feels like that can help to realise those benefits. It was right that the focus moved away from outputs, or just counting sessions or participants, but we’re still missing a critical part of the picture. “We help young people improve their confidence and fulfil their potential” or “we steer young people away from crime” says nothing about what is actually done in the moment, or why, and what a young person experiences when they engage with provision. And in fact, any evidence of ‘benefits’ or ‘outcomes’ of provision is arguably meaningless without having a clear understanding of who is engaging with what activities, how frequently, and if they are experiencing low, moderate, or high-quality practice. The term ‘outcome’ as a catch-all is simply insufficient here.
 
The woolly theoretical framing also means that the ‘outcomes’ concept can be put into practice and experienced very differently by different people, under an apparent consensus on the importance of positive change for young people. Outcomes can function as high-stakes accountability mechanisms, for example, or enabling frames for enquiry. They may act as reassuring confirmation that we’ve always been doing the right thing, or perform as powerful communication narratives to persuade donors.
 
At the Centre, we are trying to untangle the messy outcomes situation in which we’ve found ourselves as a sector. We are excited to have released the Outcomes Framework 2.0 and, having worked on our thinking for several years, we are going to commit to its approach for both understanding and measuring outcomes, and the language we use to talk about them.
 
This means recognising that, in the youth sector, outcomes of interest (to both youth organisations and young people) are likely to be social and emotional, and that they are both malleable (through quality provision) and measurable (with nuanced and thoughtful approaches). 
 
Getting more specific about social and emotional skills also means getting more specific – and intentional ­- about everything else:  activities, experiences, mechanisms of change, and long-term impacts. Outcomes are the positive changes that we hope young people will experience for themselves, and which will ‘transfer’ out of the settings in which we engage with them and into other domains of their lives. They are not the same as the broad aims and intentions that we have when we design and ‘deliver’ provision. Some organisations can name outcomes they hope young people will experience in advance; some can’t.
 
We need to be more comfortable accepting this, and also accepting that all provision will have aims and intentions that can and should be interrogated. So, when it comes to outcomes, let’s talk, but let’s be more honest, specific, curious and thoughtful.