During a recent meeting with one of our regional impact networks, I found myself - not unusually - in the midst of a conversation about the place and value of impact measurement in work with young people.
“…and that’s when I find impact measurement really exciting – when I remember; really feel that it’s a search for the truth”, I said, in a fit of fervour – only to be met with a wave of laughter from the assembled group of youth workers and managers.
“No-one cares about that anymore” came the response and, absolutely correctly, I was challenged on whose “truth” I thought I was searching for.
I left slightly shamefaced, but versions of this conversation are being repeated across the country in all sorts of settings. Plenty of souls are being searched - here, for instance, and here - as to how those in the facts, evidence and impact industry are responding to challenges to their ability to search, and even the nobility of searching, for truth in the form of evidence.
I was also struck by a recent paper from the LSE and its suggestion that an impact measurement ‘elite’, led by people from a particular, homogenous background, may have its own truths, which are not necessarily an easy or automatic fit with those of the existing social sector organisations that they exist to support. Striving for a rational, transparent and comparable understanding of effectiveness may be at the heart of the doctrine of this group, but it is a doctrine like any other, its evolution and implementation determined by the background and world views of those that espouse it.
Personally, I’ve found much of the result of this soul-searching reassuring, and many of the recommended practical responses make sense. Certainly, embracing mixed methods and the associated range of perspectives and information sources should surely now be givens in what we think of as ‘good practice’ in generating and using evidence, as should the value of those in the research world ‘getting their feet dirty’.
Current conversations about truth and facts are presenting an opportunity to reassess our own certainties and world views: how they relate to those of others who think differently: our willingness to interrogate our own world views as much as those of others, and whether and how we communicate what we find when we undertake that interrogation.
So where does this leave us at the Centre for Youth Impact?
We are certainly examining where we can usefully place ourselves within the range of perspectives on these issues held by the practitioners, impact ‘experts’, sector leaders/managers, policy makers and funders with whom we work.
We’re finding the question of ‘whose truth?’ a helpful one for framing our work. We, and many of our partners in the youth sector, are increasingly conscious of the relationship between impact measurement and our own, individual truths. Do we expect to confirm what we already ‘know’ when we strive to measure impact? Are we looking for subjective proof of our own hypotheses? Are we seeking information that might be challenging, but will help us learn and develop? Or are we looking for information that might help us make difficult decisions about funding and prioritisation?
These questions come from different perspectives on what counts as truth and how it might manifest in evidence. The questions being posed also have significant implications for how we go about attempting to answer them.
We want to get far more explicit about the role of truth, and different people’s understanding of how truth might be understood, constructed and deconstructed by activities related to impact measurement. We working with colleagues at Renaisi to develop materials to help us to do so. These materials, which we have just started testing with our network leads, are firmly anchored in the questions: who we are measuring impact for; and whose questions we are trying to answer?
On a more micro level, we also need to consider the extent to which we are seeking to uncover the truth for each individual, by exploring an issue from all angles, and being constantly innovative as to how new perspectives can be unearthed and expressed honestly.
Finally, these conversations have increased our certainty that coaching, peer-to-peer knowledge sharing, and collaborative and collective solutions are the way forward in responding to the impact measurement ‘problem’. We believe the coaching philosophy works for us and our networks because it ensures that we meet individuals where they are – not where we think they should be - and focus on asking questions, not providing answers, in supporting them to develop an approach that works for them, when it comes to understanding their impact.
Current conversations about truth are a reminder that evaluation and efforts to understand impact must a) be explicit about in whose interests they are being carried out, and b) acknowledge the multiple truths that exist and interact when it comes to understanding change within individuals and communities.
Shifts in the public narrative, and political realities, seem to have made so many of us turn back to our own truths, and challenge them and our behaviour accordingly. This is an opportunity, if it makes us reflect on how we and our work are seen by others. It is also a reminder that we should never get too comfortable in our assumptions that we are doing good work because we’re trying hard and our intentions are noble.
Our starting point is that the output of impact measurement will never represent one objective truth – only ever a proxy thereof – and that how that proxy is constructed depends significantly the question being asked, who’s asking it, and how. This is not necessarily a problem, but becomes one where it is not acknowledged: perhaps a significant contributor to some of the divides between youth work practitioners and those from the impact industry that can characterise our work.
While writing this piece, I spent a fascinating couple of weeks planning sessions with the diverse and passionate line up of speakers that took part in "The Measure and The Treasure" conference on16 March – follow #treasurePSD and sign up to our mailing list to see how this all unfolded.