Ideally, we would like to follow every behaviour change intervention with a robust, long-term evaluation. One that spans short, medium, and long-term outcomes, capturing shifts in mindsets, behaviours, and systems with the clarity and confidence of a well-rounded evaluation framework.
But oftentimes, we are in a situation where resources simply don’t allow for it. Many of our clients in the public and third sector face tighter budgets, limited timeframes, and pressure to demonstrate value quickly. In these contexts, a well-rounded evaluation that covers all necessary and pertinent aspects may not be feasible. Through multiple such cases, Magpie’s behavioural science and insight team has learnt to develop a clear rationale and method to navigate this tension pragmatically and strategically.
Ways of evaluating behaviour change strategies
When we set out to measure latent, complex outcomes like shifting norms, reducing stigma, or building new habits, changes aren’t always directly observable in the short term. By convention, we often rely on one or more of the following evaluation methods:
- Engagement and impression metrics
These act as proxy indicators, revealing how many people saw the campaign, how long they watched a video, how many clicked through etcetera. While useful and relatively inexpensive, these metrics often tell us more about attention than impact. Suitable for awareness related campaigns. - RCTs and pre-post testing of concepts
These methods involve showing campaign materials to the audience and measuring shifts in attitudes or intentions. This can provide stronger causal inference than pure engagement data, but it still doesn’t guarantee actual behaviour change in the real world. And even this level of testing may not be achievable on a shoestring budget. Good for single-strategy interventions that aim for attitudinal change. - Three-phase evaluation: short, medium and long-term outcomes
This is the gold standard. We break the behaviour change journey into stages, define operationalised indicators, and track them over time. But this approach requires significant investment – financially and in terms of time, staffing, and ongoing access to participants. For many public sector campaigns, this option becomes less feasible and sometimes impractical. Ideal for a multi-strategy intervention that aims for behaviour change.
In the world of behaviour change, evaluation generally falls into two categories:
- Summative Evaluation looks at final outcomes. It tells us if the intervention worked and if the behaviours changed over time. This type of evaluation is typically carried out after implementation, often requiring longitudinal data, larger budgets, and sustained involvement. It is essential for validating large-scale impact but can be out of reach for many public or third-sector projects.
- Formative Evaluation, on the other hand, focuses on developing and improving an intervention before it is fully rolled out. It includes relevance testing, concept testing, piloting, and gathering feedback from the target audience. It’s about ensuring that what we build is fit for purpose and context before investing heavily in delivery.
When resources are limited, putting time and budget into formative evaluation is often the smarter investment. It helps sharpen the strategy, deepen audience insight, and increase the chances of genuine impact, all without the cost and complexity of a full-scale summative evaluation.
Focusing on formative approaches gives us the confidence that we’re not just delivering something but delivering the right thing.
When budgets are tight, relevance testing is a smart investment
In these resource-constrained scenarios, we believe there’s a more meaningful way to use limited budgets: invest in relevance testing of the strategy before rollout.
Relevance testing involves taking early-stage concepts, messaging, or prototypes and testing them, mostly qualitatively and sometimes even quantitatively, with a subset of the target audience. The aim isn’t just to measure interest or preference. It’s to uncover how the target audience feels and thinks about the strategy, how well it resonates, how it fits into their lives, and whether the framing or approach needs to be adapted.
This isn’t just validation or pilot testing. It’s where breakthrough insights often happen. Time and again in our work, we’ve found that relevance testing surfaces something we hadn’t anticipated – an assumption we need to challenge, a nuance we’d missed, or an opportunity to frame things more meaningfully for a specific audience. And the impact of that learning dramatically increases the chances of success.
Crucially, this testing also ensures that we’re not just building strategies for people, but with them.
Why relevance testing works
Our work across complex and layered behaviour change projects in alcohol moderation, gambling harm reduction, creating an intuitive guide to healthcare pathways, and responsible waste management has repeatedly shown that the most effective strategies are those that are deeply rooted in the realities of the people we aim to support.
When we engage communities meaningfully through co-creation workshops and concept testing focus groups, we gain more than data points. We build trust, refine relevance, and generate strategies more likely to land, resonate, and bring about lasting change.
That’s why, when budgets are tight, we recommend clients prioritise strategy refinement and hyper-local adaptation through relevance testing over small-scale or superficial evaluations. In many cases, it offers a better return on investment and lays a stronger foundation for future evaluation for when more funding becomes available.
Evaluation should never be an afterthought
At Magpie, we build evaluation into our behaviour change frameworks from the start. Even when a full three-phase evaluation isn’t immediately viable, we create adaptable frameworks that can be scaled up when circumstances change. That means the work we do now still supports deeper measurement and insight later.
If you’re navigating challenges like tight budgets, complex behaviour change goals, and the need for meaningful evaluation, we’d love to talk. Whether you already have a behaviour change aim in mind or are just exploring your options, our team can help you build effective, evidence-based strategies and shape smart, pragmatic evaluation approaches around them.
Get in touch to see how we can work together to co-create impactful, relevant interventions that make the most of your resources without compromising on insight.