Are you measuring what matters?
Posted on 07 May 2018
By Chris Borthwick, thinker-in-residence, Our Community
This is a new data-driven era, and if you claim that your grants programs have had an impact, you have to back those claims with data. Anecdotes aren’t good enough. You need to set out your aims; provide a persuasive logic model suggesting the steps that should be taken to achieve those aims; map out the milestones that would tell you whether you were on track along the way; and compare actual performance with expected at every point.
You have to set out your criteria in advance, too, to avoid the temptation simply to draw bullseyes around wherever the arrows hit. You can’t argue with that.
But. There are more things in heaven and earth, Horatio, than are dreamt of in your criteria. The problem with measuring the outcomes you expect is that this doesn’t account for the unexpected – and it’s surprising how much of what happens in the course of work is surprising.
Let’s hop into our time machine and go back to 2003, when we’re a not-for-profit foundation listening to Mark Zuckerberg set out his metrics for the performance of his proposed new website, The Facebook. We’ll pretend for the moment that we’re not interested in the money (except insofar as we’re slightly worried about sustainability); as a foundation, we want to know what it will do to bring the community together.
Mr Zuckerberg sets out a number of hard targets. The new offering will allow people over the world to connect easily with their friends, and extend their friendship network by reaching out to their friends’ friends, and keep abreast of their family connections, creating a net of connectivity. He attaches numbers to these predictions – five million users by the end of five years. We give him the money he needs.
And 15 years later we come back and see whether that was money well spent. According to our initial criteria, it certainly was. The number of users has gone up to 2.2. billion, and everything Zuckerberg said he would do he has done. No argument there.
However, it now also appears that in doing that he’s facilitated Russian interference in the American electoral process that may well have resulted in the election of an unfit leader to control the American nuclear arsenal. Among other things, Facebook has apparently facilitated genocide in Myanmar. And Zuckerberg has become one of the most powerful people in the world.
The initial criteria, it seems, didn’t capture everything important about the project.
As medical researchers say, there is no such thing as a side-effect. There are only effects – some that you expected, some that you didn’t; some that you want, some that you don’t. And the same applies to impact.
It’s not all bad. Things can go better than expected (don’t laugh, it can happen). Your work may throw off new discoveries or serendipitous breakthroughs. If you’re funding research, it’s supposed to – even, perhaps, expected to. If it’s research, though, it is by definition looking for things we don’t already know, and that we can’t in advance know are knowable. And even where what we do isn’t defined as research, everything we do should include a research element, seeking to diminish the sphere of ignorance, shedding light into dark corners.
So how do we square the circle? How do we assess the impact of what we didn’t think was going to happen? That has to involve moving beyond the numbers. Qualitative measurements can be as important as quantitative. Canadian health promoter Ron Labonte has sketched out an assessment procedure based on group dialogue around case studies. It’s resource-intensive, but it offers a pathway to what we actually want – not simply facts, but also meaning.
Whether or not you follow Labonte’s methods, your assessment plan should still include an opportunity for reflection on what actually happened and what in that was important. Assessment should provide you with a torch, not with blinkers. We need the wisdom of hindsight.