When you're asking for evaluation, put yourself in 'Sonia's' shoes
Posted on 07 Oct 2019
By Matthew Schulz, journalist, Our Community
Impact specialist Andrew Callaghan from the Australian Social Value Bank (ASVB) urges grantmakers to consider the challenges not-for-profit and charity managers face amid funders' increased expectations of impact evaluation.
Mr Callaghan - speaking at Grantmaking in Australia 2019 - says that when you request an evaluation, you should picture "Sonia", who looks something like this:
- A senior manager
- Works for a medium-sized not-for-profit
- Is conducting activities across Australia
- Doesn't have evaluation knowledge or skills
- Has limited capacity to undertake evaluation work within the organisation
- Is getting mixed messages about what she should be doing from various funders
On top of all that, her organisation is battling to create its own evaluation framework and measurement methods, and hoping to safeguard its funding and promote its social impact.
"Think about it from her angle if you're a grant provider, [and especially] the challenges she faces in proving impact," Mr Callaghan said.
Part of that challenge is the inconsistency Sonia faces in dealing with different government agencies across Australia, each expecting her to adopt a different outcomes framework and different indicators.
AVSB's analysis of the different states suggests New South Wales is leading the way in terms of its outcomes frameworks, now embedded into contracts, such as those for social housing and homelessness. Other states, such as Western Australia, are still in "draft" mode when it comes to outcomes, but are rolling out their frameworks nonetheless.
Sonia is likely to face further hurdles in "aligning" her organisation with those expectations, such as attempting to address "high level" indicators such as "harm reduction" or "preventing suicide" that are commonly adopted by governments with big-picture agendas.
As Mr Callaghan points out, it can be difficult for smaller organisations to demonstrate they've done work that meet those big-picture aims.
He suggests that grantmakers designing or working with those frameworks have a clear understanding of what they're expecting from grant recipients.
This means, for instance, understanding the differences between research and evaluation, even if some of the methods used to achieve results are similar.
And he says grants evaluation should follow principles of "social value" that are already widely accepted, including:
- Stakeholder involvement: Including the design of measurement approaches and questions
- Understanding what's changed, and the effects that are valuable to your organisation and stakeholders
- Valuing only things material to your program, and avoiding taking credit for the work of other partners or over-claiming the effect of what you're doing
- Being transparent about how you got your findings, including your methodology - particularly if other organisations are considering employing your approach on the basis of your projects.
Based on the generally accepted principles of Social Return on Investment, commonly referred to as SROI, "I see them as a broader set of principles about evaluation and impact measurement", Mr Callaghan says.
He suggests grantmakers need to ask themselves these questions:
- What are my desired outcomes?
- What am I going to invest to evaluate the program against those outcomes?
- What is the support mechanism to ensure recipients are able to collect the data needed?
- What is my process for auditing the data to ensure it is sound?
- What are we going to use the evaluation data for?
They're easy questions to ask, but as grantmakers will attest, much harder to answer.
And never forgetting "Sonia", Mr Callaghan believes many funders need to focus more on point three, keeping an eye on how evaluations are progressing and stepping in to help when required to ensure things get back on track.