Ideally, we would like to follow every behaviour change intervention with a robust, long-term evaluation. One that spans short, medium, and long-term outcomes, capturing shifts in mindsets, behaviours, and systems with the clarity and confidence of a well-rounded evaluation framework.

But oftentimes, we are in a situation where resources simply don’t allow for it. Many of our clients in the public and third sector face tighter budgets, limited timeframes, and pressure to demonstrate value quickly. In these contexts, a well-rounded evaluation that covers all necessary and pertinent aspects may not be feasible. Through multiple such cases, Magpie’s behavioural science and insight team has learnt to develop a clear rationale and method to navigate this tension pragmatically and strategically.

Ways of evaluating behaviour change strategies 

When we set out to measure latent, complex outcomes like shifting norms, reducing stigma, or building new habits, changes aren’t always directly observable in the short term. By convention, we often rely on one or more of the following evaluation methods:

1. Engagement and impression metrics
   These act as proxy indicators, revealing how many people saw the campaign, how long they watched a video, how many clicked through etcetera. While useful and relatively inexpensive, these metrics often tell us more about attention than impact. Suitable for awareness related campaigns. 
2. RCTs and pre-post testing of concepts
   These methods involve showing campaign materials to the audience and measuring shifts in attitudes or intentions. This can provide stronger causal inference than pure engagement data, but it still doesn’t guarantee actual behaviour change in the real world. And even this level of testing may not be achievable on a shoestring budget. Good for single-strategy interventions that aim for attitudinal change.
3. Three-phase evaluation: short, medium and long-term outcomes
   This is the gold standard. We break the behaviour change journey into stages, define operationalised indicators, and track them over time. But this approach requires significant investment – financially and in terms of time, staffing, and ongoing access to participants. For many public sector campaigns, this option becomes less feasible and sometimes impractical. Ideal for a multi-strategy intervention that aims for behaviour change.

In the world of behaviour change, evaluation generally falls into two categories:

• Summative Evaluation looks at final outcomes. It tells us if the intervention worked and if the behaviours changed over time. This type of evaluation is typically carried out after implementation, often requiring longitudinal data, larger budgets, and sustained involvement. It is essential for validating large-scale impact but can be out of reach for many public or third-sector projects.
  
• Formative Evaluation, on the other hand, focuses on developing and improving an intervention before it is fully rolled out. It includes relevance testing, concept testing, piloting, and gathering feedback from the target audience. It’s about ensuring that what we build is fit for purpose and context before investing heavily in delivery.
  

When resources are limited, putting time and budget into formative evaluation is often the smarter investment. It helps sharpen the strategy, deepen audience insight, and increase the chances of genuine impact, all without the cost and complexity of a full-scale summative evaluation.

Focusing on formative approaches gives us the confidence that we’re not just delivering something but delivering the right thing.

When budgets are tight, relevance testing is a smart investment

In these resource-constrained scenarios, we believe there’s a more meaningful way to use limited budgets: invest in relevance testing of the strategy before rollout.

Relevance testing involves taking early-stage concepts, messaging, or prototypes and testing them, mostly qualitatively and sometimes even quantitatively, with a subset of the target audience. The aim isn’t just to measure interest or preference. It’s to uncover how the target audience feels and thinks about the strategy, how well it resonates, how it fits into their lives, and whether the framing or approach needs to be adapted.

This isn’t just validation or pilot testing. It’s where breakthrough insights often happen. Time and again in our work, we’ve found that relevance testing surfaces something we hadn’t anticipated – an assumption we need to challenge, a nuance we’d missed, or an opportunity to frame things more meaningfully for a specific audience. And the impact of that learning dramatically increases the chances of success.

Crucially, this testing also ensures that we’re not just building strategies for people, but with them.

Why relevance testing works

Our work across complex and layered behaviour change projects in alcohol moderation, gambling harm reduction, creating an intuitive guide to healthcare pathways, and responsible waste management has repeatedly shown that the most effective strategies are those that are deeply rooted in the realities of the people we aim to support.

When we engage communities meaningfully through co-creation workshops and concept testing focus groups, we gain more than data points. We build trust, refine relevance, and generate strategies more likely to land, resonate, and bring about lasting change.

That’s why, when budgets are tight, we recommend clients prioritise strategy refinement and hyper-local adaptation through relevance testing over small-scale or superficial evaluations. In many cases, it offers a better return on investment and lays a stronger foundation for future evaluation for when more funding becomes available.

Evaluation should never be an afterthought

At Magpie, we build evaluation into our behaviour change frameworks from the start. Even when a full three-phase evaluation isn’t immediately viable, we create adaptable frameworks that can be scaled up when circumstances change. That means the work we do now still supports deeper measurement and insight later.

If you’re navigating challenges like tight budgets, complex behaviour change goals, and the need for meaningful evaluation, we’d love to talk. Whether you already have a behaviour change aim in mind or are just exploring your options, our team can help you build effective, evidence-based strategies and shape smart, pragmatic evaluation approaches around them.

Get in touch to see how we can work together to co-create impactful, relevant interventions that make the most of your resources without compromising on insight.

At Magpie, we understand the power of learning from our audiences to create meaningful and impactful campaigns that create behaviour change. In a digital age, it’s crucial to approach campaigns with a test-and-learn mindset. Piloting creative behaviour change marketing campaigns (or any marketing campaign) digitally as an initial test and learn phase, is an invaluable approach for creating insight-driven, personalised and cost-effective full-scale marketing campaigns. 

Here’s why: 

• A/B Testing: In A/B testing, two versions of content are compared to see which one performs better. A control version (A) is tested against a variant version (B) to find out which one is most successful. Paid digital campaigns provide the perfect environment for A/B testing that cannot be replicated in more traditional marketing approaches. By experimenting with different creative elements, messaging strategies, and audience targeting, we can gain valuable insights into what resonates best with the target demographic. A/B testing during pilot campaigns allows for data-driven decisions, ensuring that the full campaign is optimised for maximum impact.

• Small Budgets and Big Insights: We often execute pilot campaigns with smaller budgets, making them a cost-effective way to test hypotheses and refine strategies before committing to larger budgets. Despite a limited budget, the insights gleaned from pilots can be invaluable in shaping the direction of the full campaign, leading to better use of budgets on the elements that have proven most effective as well as higher returns on investment.

Message and Creative Testing: Understanding which messages and creative resonate most with the audience is key to crafting compelling marketing campaigns. Paid digital pilots enable us to test different messaging and creative concepts in real time, allowing for optimisations and changes based on performance metrics. This approach ensures that the final campaign is optimised and tailored to effectively communicate key messages and calls to action.

We have done this across a number of our campaigns.

Hertfordshire County Council: Let’s Clear the Air

‘Let’s Clear the Air’ is a creative multi-channel campaign co-created with Hertfordshire County Council to address the impact of air quality on residents. The digital aspect of the campaign aimed to drive residents living in Air Quality Management Areas (AQMAs) to sign up for air pollution alerts via the Hertfordshire County Council website.  

KPIs for this campaign were heavily reliant on the digital strategy, therefore we carried out an initial pilot campaign to A/B test creative behavioural messaging concepts to determine the most effective messaging for the campaign. We utilised Facebook, Google Search and Google Display. Hertfordshire County Council council gained valuable insights into audience preferences and we optimised the strategy for maximum impact. The pilot campaign informed the rollout of a comprehensive, 12-month digital strategy, resulting in increased awareness and engagement with air quality initiatives. Find out more.

• Understanding of KPIs and Manageable Expectations: Pilot campaigns provide an opportunity to define key performance indicators (KPIs) and establish realistic expectations for the full campaign. By setting measurable goals and tracking performance metrics during the pilot phase, we can gain a deeper understanding of what success looks like and adjust strategies accordingly.

• Platform Insights: Each digital platform (ie – social media sites, Google Search etc) can offer different advantages in terms of potential reach/engagement and targeting capabilities. Pilot campaigns allow us to test different platforms and determine which platforms are most successful in terms of the overall goals of the campaign. This knowledge helps us to determine which platforms should be selected for the full campaign, ensuring maximum reach and impact.

• Comments and Community Engagement: Engaging with the audience through comments and feedback is integral to ensuring that concerns are heard and questions are answered. Pilot campaigns provide an opportunity to anticipate and address common themes that may arise in comments, allowing us to refine messaging and response strategies for the full campaign. Advanced preparation based on insights from pilot campaigns ensures a proactive approach to community engagement.
  
• More Effective Use of a Bigger Budget: By leveraging the insights gained from pilot campaigns, we can make informed decisions when allocating resources for the full campaign. A deeper understanding of audience preferences, platform performance, and messaging effectiveness allows for a more strategic use of resources, maximising the impact of the campaign.

RSPCA: Healthier Breeds

Through a targeted digital campaign, we addressed the demand for flat-faced (brachycephalic) dogs, leveraging audience insights and dynamic creative to raise awareness about the health risks associated with these breeds. The pilot campaign allowed for precise audience targeting and message testing, leading to significant engagement and a shift in the desire to purchase a brachycephalic dog. From this, we gained a deeper understanding of the motivations behind purchasing these breeds and which messages resonate the most with the audience. This allows us to create an insight-driven and cost-effective strategy for a potential national extension. Find out more.

In conclusion, digital pilots play a vital role in shaping insight-driven campaigns. By embracing A/B testing, small budgets, and iterative improvements, we can optimise their strategies for maximum impact. Through platform insights, community engagement, and effective resource allocation, pilot campaigns serve as invaluable stepping stones towards successful full-scale marketing campaigns. 

At Magpie, we recognise the importance of leveraging digital pilots to create campaigns that resonate with audiences and drive meaningful change. See below to get in touch and find out more.

Magpie Meets is a unique exchange programme which exists to fulfil Magpie’s volunteering time and professional development needs through exchanges with a diverse range of change-making organisations.

Participants of the programme are invited to join Magpie to solve a behaviour change communications challenge using behavioural science, creativity and collaboration. The process involves an in-depth briefing, a cultural immersion activity and an exchange workshop which encourages Magpie to transfer knowledge, understand change-making leadership at multiple levels and bring creative and academic practices to problem solving.

Between January and February, Magpie worked with Women’s Health Matters, a Leeds based charity that exists to support women and girls to live safe and healthy lives. Since 1987, the organisation has provided holistic, trauma-informed services for women and girls who are socially, economically, or politically marginalised. This includes, but is not limited to, those affected by domestic abuse, those whose children live elsewhere, those accessing maternity services, women seeking asylum, women with disabilities, and women and their children experiencing trauma.

Last year, the charity worked with 2517 women & girls across 27 projects. Provided intensive support to 1214 women & girls, lighter touch support to 1303 women & girls, and supported 303 children. They delivered 869 group sessions and 1728 one-to-one sessions. 

The team are a positive force in Leeds and Yorkshire, they work tirelessly to increase confidence, wellbeing and provide transformational support. For a relatively small team, their impact is huge. Access to more funding and new partnerships are vital to their work so do get in touch if you can offer your support.

Magpie Meets brief

CEO, Rachel Kelly, set Magpie the following creative behaviour change challenge:

How, in a time-poor and resource-stretched organisation, can we grow our reputation?

Following this, a behaviour change exchange programme was designed to co-create an approach for cultural change with those responsible for delivery, resulting in a behaviour change communications blueprint and journey map to guide the organisation on this programme of work.

Meeting the team

‘Magpie Meets… Women’s Health Matters’ began with an initial immersion session to understand the challenge from different perspectives and to dig deeper. During this seesion, Magpie was able to gain a more rounded understanding of Women’s Health Matters vision for change and understand the practicalities of the challenge.

For every Magpie Meets programme, a hand-picked project team is selected with relevant skills matched to the challenge set by the beneficiary. The team for this exchange included Co-founder Ged Savva, Campaign Consultant – Louise Hallworth and Researcher – Mierla Neto.

Exchange day

In January, the Magpie Meets project team arrived at Women’s Health Matters HQ in Leeds for a visionary day of co-creation.

In attendance were five representatives from across the charity, each representing different areas of the organisation. The passion and belief alignment of the team at Women’s Health Matters was truly awe-inspiring, it was clear the vision and mission of the organisation was being lived and realised by all. 

The primary resource was the ‘co-creation canvas’ which would act as a one-page strategy for taking the desired change forward beyond the session, enabling the team to focus the outcomes of the sessions as well as actionable phases of work.

Outcome

Together, Magpie and Women’s Health Matters defined the behavioural outcomes the charity aims to achieve through their project. 

12 possible outcomes were explored and whittled down to one clear behaviour and culture shift change goal. This is:

Making space for proactive thinking to help grow reputation and increase funds.

Using a COM-B analysis for the challenge, we uncovered the capabilities, opportunities and motivations for change. The most prominent barriers for this culture shift included:

• Lack of awareness around the possibility of proactive thinking time
• Finding time in a busy schedule 
• Having inspiration to make thinking time possible and effective

The most prominent facilitators for this culture shift included:

• The importance of framing – how it is positioned and communicated
• Making the solution flexible to fit around individual roles
• Creating a framework to guide and inspire a culture shift

The sessions resulted in a blueprint strategic approach, giving the organisation focus and an outline plan for taking change forward. This included three potential frameworks for change including an open framework, a cumulative change framework and a more creative framework to align with the creative and tenacious culture at Women’s Health Matters.

Next steps

A creative behaviour change report has been provided to Women’s Health Matters and a follow-up consultation is taking place in March to further the organisation with advice around the practicalities of adopting the recommendations. 

Interested?

Magpie is currently planning the calendar of Magpie Meets activity. If you would like to be considered as a beneficiary for our next Magpie Meets programme, please get in touch and a member of the Magpie team will be in contact.

Our Director of Behavioural Insights, Dr Grainne Dickerson, discusses the rise of Behavioural Science and how Magpie uses it to create happier and healthier communities.

What is behavioural science?

Behavioural science refers to the disciplines of psychology, behavioural economics, sociology and anthropology to understand, explain and predict behaviour. These fields of science are driven by academia and the rigorous research produced by behavioural scientists who are trained to the highest standards in research methods in their fields. Whilst the research conducted has potential to improve health outcomes for the population, there have been many missed opportunities to apply the findings of robust research in real world settings, however, this is changing.

What is the history of behavioural science?

Behavioural science started gaining popularity when the Cameron-Clegg coalition government in 2010 set up a nudge unit, inspired by the book Nudge by Thaler and Sustein (2009).

It was in the same year that I achieved my dream of becoming a behavioural scientist when I qualified as a Health Psychologist. Reading the book, Nudge and hearing about the nudge unit I realised I had been trained in all the same stuff! I then found myself determined to utilise my learning and skills within Public Health where I had been working.

“My mantra at the time was … if they can have a nudge unit in the government, they could and should have something similar in the NHS and local authorities, for they are also in the business of changing public behaviour.”

For several years after that I was actively championing behavioural science within Public Health in order to meld together the science and art of protecting the health of the public via Public Health work with the psychological processes in health, illness and healthcare through Health Psychology work. I had fantastic support at times in the various behavioural science and public health roles I worked in, and at other times I faced challenges, and this was because behavioural science wasn’t as widely valued back then. Fast forward to now, 12 years later in a world blighted by COVID-19, there’s been a proliferation in behavioural science…

Where is behavioural science now?

There has been a steady increase in the number of behavioural science teams and roles within the public sector who can apply rigorous scientific approaches to real world settings. This development reflects the recognition that behavioural sciences provide the opportunity to improve outcomes in the many areas of the public sector that are seeking to influence and change behaviour. Organisations with behavioural science teams include the Office for Health Improvement and Disparities, the UK Health Security Agency, the NHS, the Department for Transport and some local authorities such as Hertfordshire County Council.

There is a collective moving away from using ‘good ideas’ and a reliance on assumptions to utilising science based approaches to understanding target audiences, target behaviours and barriers and enablers that interventions need to address in order to enhance the health and wellbeing of population groups.

Here is a selection of highlights that have contributed to the growth of behavioural science during the last 12 years:

• The British Psychological Society’s Division of Health Psychology has worked behind the scenes to try and join up Health Psychology with policy and practice, enabling research to be translated into practice, alongside other social sciences making similar moves.
• The publication of the Behaviour Change Wheel in 2011 has made previously inaccessible theories more accessible, followed by the publication of the taxonomy of behaviour change techniques in 2013.
• More recently in 2019 an updated and practical version of the behaviour change wheel and the taxonomy of behaviour change techniques was published for local authorities in Achieving Behaviour Change: A guide for local government.
• The Behavioural Science and Public Health Network have been bringing together behavioural scientists with the Public Health workforce to develop capacity and capability for behavioural science and developing regional behavioural science hubs.
• An important strategy was published in 2018: Improving people’s health: Applying behavioural and social sciences to improve population health and wellbeing in England which marked a coming together of key national organisations to articulate what the behavioural and social sciences can offer Public Health and the theories and frameworks that can be described under the banner of behavioural science. A key message was the importance of transdisciplinary approaches, and they visualised this with a  re-invigorated version of the Dalgren and Whitehead Model (1991) of the social determinants of health.
• The biggest factor has been the pandemic which has further amplified and highlighted the importance of behavioural sciences for changing public behaviour and essentially reducing health inequalities.

Behavioural science at Magpie:

At Magpie we also adopt a transdisciplinary approach to bring our behaviour change campaigns and interventions to fruition whilst championing unheard voices.  We bring together a combination of different skills and perspectives that combine to add the value that no single approach would achieve on its own. Working in this way creates more impact, but is also more rewarding and fun!  We combine subject matter expertise from our clients and our team with behavioural sciences, community engagement and creativity.

What next for behavioural science?

You could say there is a democratisation of behavioural science; moving away from the mysterious  ‘nudge unit’ in central government and the purely academic field to a more accessible and transparent approach.

Behavioural science is now a respected part of many teams, however there is more that could be done to further enhance the contribution of behavioural science across the country, including:

• Sharing learning: Behavioural scientists working in applied settings share their learnings, and specifically their scientific approaches and impacts made. This will enable the continued learning about what does and doesn’t work for different issues within different populations and communities. Conferences like the Behavioural Science and Public Health Network annual conference and the regional behavioural science hub events are great avenues for this.
• Developing behavioural science skills: People working in practice can develop the understanding and skills they need to apply a behavioural science lens to thinking about problems and how to solve them.  There are now podcasts and several books to dive into that make the learning more accessible to people in applied settings.  For anyone wanting to make a start with a behavioural science podcast, I can recommend the real world behavioural science podcast for the perspectives of real experts in the field and the Behaviour Change Marketing Bootcamp podcast aimed at marketing and communications professionals. Happy listening! (Watch this space for learning opportunities from Magpie later this year…)
• Developing capacity for behavioural science work: For organisations who would like to recruit behavioural scientists to do in depth behavioural science work, my top tip is to use these recently published good practice guidelines for employers who wish to employ appropriately qualified and experienced people.

Written by Dr Grainne Dickerson

(Director of Behavioural Insights and Chair Elect Behavioural Science and Public Health Network)