Understanding ‘real measurement’ in grantmaking

During the session I conducted a simple experiment to highlight the importance of measurement. I asked three volunteers to measure a sign using various tools: a tape measure, a piece of string, a pair of scissors, and empty hands.

The result was both entertaining and eye-opening. While each volunteer produced quantifiable results, only one tool (the tape measure) gave results that were reliable, precise, replicable and valid. This small experiment highlighted what constitutes “real measurement.”

And yet measurement has proven notoriously difficult to pin down throughout human history.

Shortly before my session, SmartyGrants data scientist Nathan Mifsud discussed the value of standardising measurements with reference to the creation of the standard metre.

Prior to the metric system, French markets used over 40 different measurement methods for fruits and vegetables, but the French revolution spurred scientists to adopt a standard. Over the next century, France played a pivotal role in establishing the metric system, which continues to guide scientific endeavour and everyday life for many of us.

The value of a measure that we can agree on can’t be underestimated, so here are a few pointers from me.

Criteria for best-practice measurement

Quantifiable: The ability to count the thing being measured is fundamental. Grantmakers often deal with intangible outcomes such as increased community wellbeing, improved educational outcomes, or deeper cultural enrichment. In the world of outcome measurement, it is necessary to provide alongside the outcome statements metrics or indicators to measure these outcomes. Whether it's the number of children attending an after-school program, the number of community interactions, or the number of artistic performances, being able to quantify these aspects is essential to gauging progress.

Reliable: Imagine conducting the same measurement multiple times. If the results vary greatly, the measurement lacks reliability. Reliability means that the same measurement produces the same result each time – it is consistent. Reliable measurements across grantees are essential if grantmakers want to aggregate these results across a portfolio to learn what worked and what didn’t.

Precise: Precision refers to the accuracy of a measurement. It's about being able to distinguish small differences between measurements. For instance, consider head counts versus registration form counts. While head counts may provide a rough estimate of the number of people who attended an event, registration forms can offer more precise data (if everyone attends), including demographic information about participants and details of their interactions with the program.

Replicable: Replicability means that someone else, using the same methods and tools, can come along and achieve the same measurement. This is vital for transparency and accountability. Grantmakers need to be able to demonstrate that their measurements can be independently verified, reinforcing the authenticity of their data.

Valid: Validity is about using measurement tools and methodologies that have been tested and proven to measure what they intend to measure. Valid measurements accurately represent the concept or attribute they are meant to assess. In the grantmaking context, using validated tools and approaches ensures the integrity of the measurement process.

The principle of proportionality

Measuring the impact of grants isn't a one-size-fits-all endeavour. Grantmakers must take into account the scale and purpose of their grants when setting measures, metrics or indicators for grantees to report against. This is where the principle of proportionality comes into play.

The principle dictates that the effort and resources invested in measurement and evaluation should be proportionate to the size and purpose of the grants being distributed. Smaller grants may not justify complex and resource-intensive measurement processes, while larger grants should involve more in-depth measurement efforts. We recommend the 10% rule: 10% of grant funding should be spent on measurement and evaluation of a grant. This ensures the expenditure is proportionate to the grant.

Just as the effort and resources expended need to be proportional, so does the standard of measurement. Different scales of grants may require different measurement standards. A micro-grant for a neighbourhood project may not demand the same level of rigor as a substantial grant for a large-scale, multi-year initiative.

In other words, the path to “real measurement” varies depending on the size and objectives of the grant.

It is imperative that grantmakers clearly communicate their measurement expectations to grantees at the time of advertising the grant. This could include defining what is expected in terms of quantifiability, reliability, precision, replicability and validity.

Recommendations to grantmakers

The next time you ask your grantees how many participants increased their physical activity or improved social connections or reduced waste, have a think about how you want those numbers provided – that is, what unit of measurement you want them to use – so you can be confident in the data they provide and you can aggregate that data across your grantees.

Here are some recommendations on how to enhance the effectiveness of your measurement and evaluation efforts:

1. Don’t recreate wheels: Find out if there are standard measures for the outcomes you have set. For example, optimum physical exercise is measured as 150 minutes a week, according to the Australian Medical Association; waste is often measured in kilograms; and so on.
2. Clearly define data collection methods, specifying what is acceptable. If a survey is the chosen method, provide the survey questions and the possible multi-choice answers (for example, the Australian Social Value Bank and the Community Services Outcome Tree both provide excellent standardised surveys linked to outcomes). If you prefer a headcount, state that explicitly or ask for a sign-in registration sheet.
3. Provide data collection guidance: Offer guidance on how grant recipients should collect and report data. Encourage them to report data in raw numbers, such as the number of people who reported “high” and “very high” on specific questions. Also, request the denominator, which may differ from the number of people engaged in the activity or completing the survey. This denominator provides context and a basis for comparison.
4. Adapt requirements accordingly: Acknowledge that different grant programs may require varying levels of measurement rigor. Adapt your requirements to the specific needs and resources of each grant.
5. Offer support and resources: Recognise the challenges that grant recipients may face in collecting and analysing data. Provide resources or training, particularly for small organisations with limited capacity.
6. Support continuous improvement: Encourage grant recipients to view measurement as an ongoing process of learning and improvement. Emphasise the value of feedback and adaptation to enhance the impact of their programs.
7. Reward and incentivise measurement: Consider implementing incentives and rewards for grant recipients who demonstrate a commitment to robust measurement and evaluation. This could include additional funding, recognition, or other tangible benefits.

By following these recommendations, grantmakers can strengthen their grantees’ measurement practices and ensure that the data that is collected is meaningful, reliable, and aligned with the goals of their grants.

In the realm of grantmaking, “real measurement” is an essential aspect of ensuring accountability, impact and value. The experiment with a tape measure, string, and hands is a reminder of the importance of meeting certain criteria for meaningful measurement. Quantifiability, reliability, precision, replicability and validity are the cornerstones of real measurement. However, we must strive for the Goldilocks measure, i.e. what is just right within the practical realities of our projects, all with the ultimate goal of making a positive difference in the world.