VaxSocial Evaluation Framework Draft - Guidance and Review

VaxSocial: Evaluation Framework Draft 1 for Guidance and Review 5thJune 2024

Introduction This document provides (i) an overview of the scope of the evaluation; (ii) and initial typology of VaxSocial interventions; and (iii) first draft of the Global VaxSocial Evaluation Framework. It is intended to be a living document, and will be refined in discussion with local grantees as they finalise their Monitoring, Evaluation and Learning plans. A final version of the document will be included in the Inception Report, which will report once formative research has completed and grantees have finalised their own delivery plans.

Scope of evaluation

Evaluation Questions 1. Whether social media-based interventions can translate online engagement to offline action (i.e., increasing vaccine uptake by change perceptions, attitudes, and behaviors among target populations). Improve understanding of how, when, and why to leverage social media and online platforms, as part of integrated SBC multichannel campaigns, to drive behavior change, and the critical success factors required to achieve this. Help generate learnings on the impact, cost-effectiveness, complementarity, and sustainability of diverse social media- based approaches, Programme 2. 3. A. Can we attribute changes in perceptions, attitudes, and behaviors towards vaccination to social media interventions? B. In which situations (incl. specific challenges, population segments, health system context, etc.) is social media an effective modality for building vaccine confidence and driving vaccine uptake and how can it complement or augment social and behavior change programs as part of an integrated social behavior change strategy? C. How do different social media interventions compare to each other: which messages, messengers, channels and intervention designs have the greatest health impact; and which are the most cost-effective? D. What are the requirements/enablers that need to be in place to enable success of social media interventions? What cost and level of complexity in implementation are associated with these requirements/enablers? E. What are the methodologies and best practices to scale, replicate or tailor agile social media interventions in different contexts? F. (Stretch objective) What reach and impact do social media interventions have beyond the direct users? Evaluation

VaxSocial Programme Overview

Introduction The following two slides are an initial attempt to map the range of interventions planned through VaxSocial. This mapping is important to help identify the similarities and differences between programmes, and to help design monitoring, evaluation and learning requirements (including includes the final Evaluation Framework). These have been drafted based on the initial information seen by Ipsos and M&C Saatchi, and will be revised in conversation with grantees over the next few months. Slide 7 is a high-level overview of each programme. Slide 8 aims to capture the different ways in which social media is being harnessed through VaxSocial. In part inspired by WHO guidance on Behavioural and Social Drivers of Vaccination (found here). The current mapping includes the way in which social media is being activated (i.e. as a broadcast message, or a two-way conversation) and the voice of the messenger (is this centralised or via an intermediary). Over time, as programmes are developed further, this will include a mapping of where the content of social media activity is intended to support demand generation, demand fulfilment, or both. Our initial conversations with grantees will explore how their activity maps to this framework, and how we can improve this mapping framework further.

Programme Overview Work in Progress Red = to be confirmed / clarified India Group M Indonesia Nigeria Upswell GHS CISDI HelpMum Nivi Topic Routine TBC and campaign Parents 0-2 HCPs Routine TBC Routine TBC Routine TBC and HPV Nursing mothers TBC HPV Routine TBC and HPV Target Population Parents TBC Parents (0-1) Agents of change Caregivers of girls 9- 14 Girls 9-14 Abuja (FCT), Nasarawa, and in Adamawa Zero-dose & underimmunized HPV eligible Lagos (with focus on Alimosho and Ikorodu LGA) Regions covered TBC Sumatera Barat, Riau, Kalimantan Utara, Papua Barat Kabupaten Bekasi and Kota Depok Kano, Katsina, Jigawa, Osun, Oyo, Akwa Ibom, Imo states Engagement metrics Chatbot users Target Outputs Engagement metrics Knowledge, literacy, confidence Vaccine uptake Engagement metrics Awareness, hesitancy, trust, norms Engagement metrics Perceptions Engagement metrics Sign up for health insurance Engagement metrics Nivi chatbot use Target Outcomes Role of social media Vaccine uptake Vaccine uptake Vaccine uptake HPV uptake Vaccine uptake Broadcast promotion of vaccine uptake, mix of central and intermediary voices Whatsapp, Youtube, Facebook, Instagram Broadcast promotion of vaccine uptake, mix of central and intermediary voices Online communities Whatsapp, Facebook, X, TikTok, Instagram, YouTube Govt website? Broadcast promotion of vaccine uptake, mix of central and intermediary voices Broadcast promotion of conversation with Chatbot Broadcast promotion of conversation with Pharmacist Broadcast promotion of Nivi Chatbot Social media platforms Whatsapp, Instagram Whatsapp, Facebook, Instagram Facebook, Instagram, TikTok Facebook

Initial Intervention Typology work in progress We don t yet know nature of social media content, some tools (eg Chatbot) are likely to be useful for both Demand Generation and Demand Fulfilment To be clarified further (1) understanding how best to categorise where social media is being used to promote a further conversation with a pharmacist / HCP / chatbot / another service; where the resulting conversation could either be demand generation or demand fulfilment). To be clarified further (2) how best to categorise where VaxSocial activity supports a point of reference for individuals to check information (e.g. hotlines / microsites), and to support content generation (e.g. monitoring misinformation). Broadcast communication (vaccine promotion) Central voice Two-way communication Organic communication Intermediary voice Reference Direct message Social media campaign Local leaders (not health) Social media influencers Healthcare professionals Chatbot* Online communities Hotline / microsites?* Misinformation monitoring Layer 1 Layer 2 Layer 3 Perceived disease risk Vaccine confidence Social norms Health worker recommendation Gender equality Intention to get recommended vaccine Promotion of a service / conversation Affordability Ease of access Service quality Respect from health worker Pharmacist* Thinking and feeling Social processes Demand generation Motivation Practical issues Demand fulfilment Vaccination Uptake of recommended vaccines *Here role of social media is to promote this service

Draft Evaluation Framework

Introduction The evaluation framework is intended to be an umbrella through which local evaluations can join together to form a global evaluation of the VaxSocial programme. It is noted that no two local evaluations will have the same design, and that each will have its own strengths. It is not expected that local evaluations will have data on every metric/source proposed in the framework. Instead, the framework provides a common map and language through which we can compare across the programme and draw conclusions about the role of social media overall. Where MEL plans include RCTs and Quasi-Experimental Designs, these will be a core part of the evidence for attributing impact of VaxSocial activity. However, we note local MEL plans include a mix of evaluation designs, the framework is therefore intended to facilitate a contribution analysis, building a comprehensive picture of all the available relevant evidence to understand impact, attribution and cost effectiveness. At the core of the framework sits a classic evaluation Logic Model which that maps inputs, outputs, outtakes, and outcomes of VaxSocial activity. This is supported by wider evidence which will help establish the attribution of VaxSocial social media activity, monitoring the wider context and environment in which VaxSocial operates locally. The triangulation between these points of evidence can help identify the impact of VaxSocial, what works, and how cost effective this activity is. We anticipate that each programme will be underpinned by a robust Theory of Change. This will help identify desired outtakes and outcomes for each grantee, from which we can build a global Theory of Change.

Common risks and challenges to consider Risk (1) Most programme designs are using distinct geographies to demonstrate impact of social media activity between (i) different interventions, and/or (ii) test and control areas. Key considerations / possible mitigations Is vaccine uptake data available at this level of geography? How confident can we be that social media can be effectively targeted at that level of geography including use of intermediaries where planned? What is the likely spill over and contamination (eg sharing by friends and families)? Are geographical areas with equal characteristics being chosen for robust comparison? Do we have effective control groups? Where a comparative service is delivered as normal without additional integration of social media. What are the measures of success early-on in the programme? Are digital engagement metrics the best indicator of long-term behaviour change (i.e. will some messages have mass appeal but yield little impact)? What other forms of testing / research is planned to demonstrate the effectiveness of content shared on social media? What demographic information will be collected throughout the MEL process that can help identify which groups are most positively impacted by VaxSocial activity? Is it possible to obtain outcome data by different subgroups? (2) Early optimisation of content likely to be based on in-flight / AB testing; however, at this stage the long term impact of any engagement will unknown. (3) An ambition of the evaluation is to understand what works for whom i.e. how impact varies by demographics. (4) VaxSocial does not operate in a vacuum, other enablers and barriers also impact the same goals. (5) Evaluations pay little attention to sustainability of how VaxSocial can be integrated with SBC programmes (6) Routine vaccinations cover a large range of vaccines, and differ between countries What are the other major initiatives in play within your chosen geography? How do they evolve over the course of VaxSocial? What are the main barriers out of the control of VaxSocial? Do they improve over time? VaxSocial is not a Process Evaluation ; however, it will be useful to gain wider feedback on what works in delivery to help maximise outcomes once VaxSocial programme ends. How are we defining routine vaccination? Are all vaccines included / to be targeted by VaxSocial? How will this be managed within the social media content/messaging (i.e. will some vaccine be prioritised over others, do they all have same barriers), and in the evaluation data collection (i.e. ability to collect data on all vaccines)

VaxSocial Evaluation Framework NB: where programmes are delivering more than one intervention, consider multiple evaluation frameworks Outputs Outtakes Short-term outcomes Long-term outcomes Inputs Additionality of social media activity VaxSocial activity What has been delivered, reach and execution Recognition, engagement, experience, salience and comprehension Costs and activities including intermediaries Initial Longer term behavioural change behavioural response, attitudes, beliefs, intentions What works for successful integration with SBC programmes? Impact What works in driving vaccine uptake? Vaccine trends Wider inputs Share of voice Stakeholder feedback Compounding factors VaxSocial attribution Take up pre- intervention and in control areas Other activity driving vaccine uptake What other conversations & promotions are working with or against you Sustainability and effectiveness of integration with SBC programmes What structural barriers may limit vaccine uptake Cost effectiveness

VaxSocial Activity: Inputs Costs and activities including intermediaries Why is this important? Accurate data on inputs is crucial to calculating cost effectiveness, and to help build an understanding of what works. Key considerations: are you accounting for all spend (not just media / direct costs)? This includes costs of planning and research. Evidence Source Notes Total spend Project Budget Consider how to account for costs of different phases of the project, and for different types of interventions. Including one off development costs, and ongoing media/programme spend. Clear mapping of different interventions to test & control groups / geographies where relevant. Media/partnership plans should be comprehensive and reference budget spent on different platform & types of communication. A record of the assets, materials, content generated. Activities, media plan and partnerships Project and media plans Content creation Final project materials Formative research Literature Reviews, Pre- testing, audience segmentations & profiling This would include any synthesis or review of existing attitudes, barriers, trends, audience insights. Also includes any fresh insight commissioned to inform the design and development of each programme. Build a clear theory of change, which also maps dependencies and assumptions. Theory of change / logic model MEL plan

VaxSocial Activity: Outputs What has been delivered, reach and execution Why is this important? Helps monitor scale and nature of activity delivered, and interaction with VaxSocial activity Key considerations: how successfully has your activity reached your target audience? What demographic data can you collect at each stage to help understand what works, and for who? Evidence Online reach Source Digital metrics (estimated reach), messages delivered Non-digital metrics Notes Inclusion of estimated reach as reported by digital metrics, but (where relevant) also record of the number of individuals contacted through the programme (e.g. where Whatsapp used to send direct messages). An ambition of the evaluation is to also understand how many of those not on social media may have been reached: this is relevant where you are planning other activity outside of social media, eg posters, leaflets etc A record of the partnerships that have been secured, and their contribution to the programme. This includes any content they have generated and delivered as intermediaries . Number of sessions, participants, materials conducted where this is part of your VaxSocial programme. Some grantees have included monitoring of misinformation and/or wider narratives of vaccine conversations online. Offline reach Partnerships secured Project reporting Training and capacity building activities Social media monitoring Project reporting Chosen aggregator platform

VaxSocial Activity: Outtakes Recognition, engagement, experience, salience and comprehension Why is this important? Helps to understand audience experience of VaxSocial activity, and whether the intervention has been executed as intended in the Theory of Change. Key considerations: are the desired outtakes the same for all of the activity you deliver how does this differ depending whether the focus of the content you generate is demand generation or demand fulfilment? Evidence Recognition Source Survey, qualitative / longitudinal research Digital metrics, Service Meta Data Digital metrics Pre-testing, survey, qualitative testing Notes This would typically include a post campaign survey of unprompted and prompted recognition, to monitor how many, and which groups, recall seeing the activity on social media. This could also be explored through qualitative interviews where used. Click Through Rates, Views, Reactions (e.g. likes), Replies and Comments, Completion rates, website hits or hotline calls (where relevant) Number of estimated impressions How well do audiences respond to content generated, is it relevant to them, salient, easy to understand? Do audiences take out key messages as intended / planned in theory of change? This could be achieved through a mix of quant or qual with the target audience. Where delivering a service for the public (e.g. chatbot, conversation with a Pharmacist) or training for health care professionals, what was their experience? Was it useful, told them something knew, answer their questions, something they would recommend to others? Active engagement Passive engagement Creative execution and comprehension Experience Surveys, qualitative research, user testing, attendee feedback

VaxSocial Activity: Short-term Outcomes Initial behavioural response, attitudes, beliefs, intentions Why is this important? VaxSocial is not able to control or address all of the barriers to taking a vaccine, it s therefore important to capture the impact on a wider range of behaviours that indicate interest/intent to take a vaccine. Key considerations: what are the assumptions and dependencies within the theory of change on the steps and considerations individuals take before getting a vaccine? Evidence Source Notes Contacts, messages and conversations received Service meta data Number of contacts received, for example Chatbots contacts, conversations with Pharmacists, or referrals made through the system. This may not be relevant to all types of interventions; however, this data should aim to demonstrate the impact of social media ie, where contacts have come directly from social media activity, and/or the difference in contacts between areas that have and have not received the social media activity. Response Survey, programmatic data What did people do immediately after exposure (including looked for more information, shared with friends, spoke to a family member, encouraged someone else etc ) Ability to demonstrate that VaxSocial activity has driven significant volume of change within the chosen population as intended in the theory of change (e.g. awareness, understanding, attitudes, emotions, confidence, social norms, intentions etc ) Aligned to the theory of change, this could include intention to get a vaccine, but also increasing conversations or seeking further information Change in thinking and feeling Survey (comparing test vs control areas, or between recognisers and non). Pre testing difference in difference. As above. Change in short term behaviours

VaxSocial Activity: Long-term outcomes Why is this important? This strand monitors vaccine uptake the ultimate goal of VaxSocial. Without this the data, an evaluation would not be able to establish the extent to which social media can shape offline vaccine behaviours Key considerations: are you able to collect vaccine data at a local level that can accurately show differences between different types of intervention, or between any test and control areas you have designed? Is there a risk of contamination or spillover between areas? Evidence Source Notes Claimed vaccine uptake Survey, longitudinal qualitative research A survey will capture self-reported vaccine behaviours. Within the design of the survey, consider how you can compare between local areas that have received different treatments/interventions, and control areas that have received no treatment. Also consider within the survey how you can demonstrate attribution to the social media activity (e.g. by asking recall/recognition). Longitudinal research could also be used to follow individual vaccine journeys over time. What is the time delay in reporting outcomes (how does that impact your ability to show change from VaxSocial)? At what geography is data reported? How does this align with your interventions (and any test and control areas)? What, if any, other demographic data is collected to help understand who is increasing vaccine behaviours? Vaccine records National and local government records / records of local partnerships The best possible outcome is to demonstrate where vaccines have been taken as a direct result of interaction with VaxSocial (consider the extent to which this may be possible).

VaxSocial Attribution Why is this important? VaxSocial is not operating in a vacuum. It is crucial to understand the wider context (levers and barriers) in which your target audience interact with vaccines; this will help attribute impact to VaxSocial activity. Key considerations: what are your sources for understanding the wider context in which VaxSocial operates. Ipsos can help conduct stakeholder interviews to ensure feedback can remain anonymous and open. Evidence Wider Inputs Source Stakeholders, Health Care Professionals Survey, social media monitoring, longitudinal qual Notes It is useful to identify, map and monitor any other major programmes that are also working towards the same goals/outcomes in your country/local area. This includes online and offline activity. Share of voice Rather than only capturing outputs from owned/earned content, social media monitoring could help identify wider conversations about vaccines. Surveys can also ask about awareness/exposure to other major activity (either through unprompted and prompted recognition, explore sources of information/influences). Longitudinal qual can also help tell a rich picture of the role of VaxSocial. Where VaxSocial is integrated with other SBC programmes, feedback will help understand the additionality of social media activity, and sustainability of future activity. What outtakes do you want stakeholders or HCPs to take from engagement with VaxSocial, or after capacity building? What evidence exists on wider barriers to vaccination, and how this changes over next 18 months? Asking about barriers to vaccination as part of survey and qualitative research will help map extent to which these can be overcome by VaxSocial activity, and whether they improve during VaxSocial. Understanding the direction of travel in vaccine uptake before VaxSocial (was this in decline, flat or already increasing), and monitoring outcomes in other areas that do not part of VaxSocial these areas should have similar characteristics to those with VaxSocial activity. Stakeholder feedback Local / national stakeholders Compounding factors Stakeholders, HCPs, lit review, survey, qual Government or partnership vaccine records Vaccine trends

VaxSocial Impact Where applied and designed well, RCTs and quasi-experimental designs will provide high quality evidence of impact of VaxSocial activity. However, these should also seek to (i) clearly attribute causation (rather than correlation) of VaxSocial activity on vaccine uptake, and (ii) evidence the additionality of social media activity where this is closely integrated with SBC programmes (i.e compared to if no social media activity took place). Triangulation of multiple evidence points will help build a contribution analysis across each programme, and VaxSocial as a whole. Supported by a Theory of Change, evidence from across the evaluation framework will help increase understanding of why the observed outcomes have occurred, and the role of both VaxSocial and wider external factors. As part of the wider evaluation, Ipsos has some budget to conduct advanced statistical analysis to help evidence and quantify attribution the feasibility of this will be explored further with grantees. We encourage a realist approach, acknowledging that the same intervention does not work everywhere and for everyone. Where possible, we welcome any attempts to include demographic analysis. Over the next few months, we will work with grantees to agree a framework for early optimisation, monitoring performance of what works across social media activity: including media plans, choice of platforms, messaging and execution. We will also agree a common framework for analysing cost effectiveness. Looking beyond standard metrics such as cost per click , this will include estimating costs mapped to the Theory of Change (e.g. unit cost per additional vaccine received, unit cost per change in attitude). Our original intervention typology will help compare performance of different intervention designs. Feedback from stakeholders will be crucial to helping understand how best to integrate social media activity to SBC programmes. Additionality of social media activity What works for successful integration with SBC programmes? What works in driving vaccine uptake? Cost effectiveness Slides 20-22 present three hypothetical examples of how outcomes from across the evaluation framework can build together to draw conclusions about an individual VaxSocial intervention.

VaxSocial Evaluation Framework: example 1 Outputs Outtakes Short-term outcomes Long-term outcomes Inputs VaxSocial activity High reach, engagement and recognition Mixed outcomes only resonates with subset of target audience Good Increase not unique to test areas, and mixed insights on outtakes & short- term outcomes. Limited reduction in hesitancy Significant increase in vaccine uptake in areas that receive the intervention investment, campaign on Facebook Despite correlation, something other than the VaxSocial activity is likely driving increase in vaccine uptake. Impact Vaccine trends Wider inputs Share of voice Stakeholder feedback Compounding factors VaxSocial attribution Some additional social media campaigns, and govt initiatives Data shows pre-existing positive uptake trend, increase also in control areas Large share of voice, and mis- information countered Positive experience from intermediaries No apparent major barriers to take up among those motivated

VaxSocial Evaluation Framework: example 2 Outputs Outtakes Short-term outcomes Long-term outcomes Inputs VaxSocial activity High engagement and increase in intention to get a vaccine, but no significant uplift in uptake. High reach, engagement and recognition Those who engage with activity take out key messages as intended Good Increase in intention to get a vaccine No significant increase in uptake investment, campaign on Facebook Despite non- favourable outcomes, likely that factors outside of VaxSocial are limiting impact of VaxSocial. Impact Vaccine trends Wider inputs Share of voice Stakeholder feedback Compounding factors VaxSocial attribution Some additional social media campaigns, and govt initiatives No change overall or difference between test and control Large share of voice, and mis- information countered Positive experience from intermediaries Vaccine shortage and difficult to access It otherwise proves effective.

VaxSocial Evaluation Framework: example 3 Despite no increase in vaccine uptake among the original target audience, the integration of social media activity does appear to be effective among a smaller subgroup. Outputs Outtakes Short-term outcomes Long-term outcomes Inputs VaxSocial activity Lower than anticipated reach, but strong engagement among a minority Those who engage with activity take out key messages as intended Good Increase in intention to get a vaccine among those who engaged No significant increase in uptake across wider target audience investment, campaign on Facebook Impact VaxSocial activity is therefore less cost- effective overall; however, there is value in reviewing whether revised media plan or execution would improve outcomes for a wider audience. Vaccine trends Wider inputs Share of voice Stakeholder feedback Compounding factors VaxSocial attribution Some additional social media campaigns, and govt initiatives No change overall or difference between test and control Low share of voice among wider narrative on vaccines Positive experience from intermediaries No apparent major barriers to take up among those motivated

Cost- effectiveness

What isnt in scope? This is not a vaccination campaign as would be defined by academic literature, national or local governments. We re not seeking to include costs of administering the vaccine (e.g. purchase of the vaccine) We re also not seeking to calculate the longer-term health benefits of having a vaccine (e.g. savings on future health treatments when people fall ill). Typically, these would be reported in the form of Incremental Cost-Effectiveness Ratios (ICERs): cost per quality-adjusted life year (QALY) gained or disability-adjusted life year (DALY) averted. Instead, we are interested in what contribution social media can make to wider programmes to improve vaccine rates.

VaxSocial Evaluation Framework (v2) Outputs Outtakes Short-term outcomes Long-term outcomes Inputs Additionality of social media activity VaxSocial activity What has been delivered, reach and execution Recognition, engagement, experience, salience and comprehension Costs and activities including intermediaries Initial Longer term behavioural change behavioural response, attitudes, beliefs, intentions What works for successful integration with SBC programmes? Impact What works in driving vaccine uptake? Monitor the costs and spend for different elements of VaxSocial delivery. There may be value in manually calculating the return on investment for your non paid-ad delivery. Measuring engagement and potential impressions Measuring whether the role of social media provided value for money in preparation for getting a vaccine By comparing vaccination rates across two points in time, or across two areas, we can calculate a CPC: Cost per Click CPM: Cost per Mille CpPR: Cost per Person Reached This is crucial for any cost-effectiveness calculation. Eg, cost of delivering capacity training Cost CPA: cost per acquisition / conversion Cost per Additional Vaccination effectiveness Cost per Person Influenced

VaxSocial Cost Effectiveness Framework Cost In-kind costs Effectiveness Engagement Capital costs Operating costs Direct advertising & promotion costs Spend on delivery of paid-for advert campaigns, or payments made to intermediaries Influence Impact Long-Term Benefits Costs incurred by other organisations to execute delivery of VaxSocial Development costs One off costs that have a longer life value beyond VaxSocial Creative development costs, or costs to develop the AI chatbot Formative research Regular spend that is needed to keep activity / campaign going Number of people reached and engaging with content or activity Increase in awareness, knowledge, contemplation, preparation Change in action, attributed to VaxSocial Longer term health benefits on having the vaccine HCP staff time Government staff time Evaluation team Staff time to execute owned, paid for and intermediary executions Analytical tools Ad spend per campaign and per platform CPC Cost per Click CPM Cost per 1,000 impressions CPA Cost per conversion (e.g. per recruitment, or per conversation with Chatbot) CpPR Cost per Person Reached CpPI Cost per Person Influenced CpAO Cost per Attributed Outcome CpAV Cost per additional vaccine

VaxSocial Cost Effectiveness Framework: Standard Cost In-kind costs Effectiveness Engagement Capital costs Operating costs Direct advertising & promotion costs Spend on delivery of paid-for advert campaigns, or payments made to intermediaries Influence Impact Long-Term Benefits Costs incurred by other organisations to execute delivery of VaxSocial Development costs One off costs that have a longer life value beyond VaxSocial Creative development costs, or costs to develop the AI chatbot Formative research Regular spend that is needed to keep activity / campaign going Number of people reached and engaging with content or activity Increase in awareness, knowledge, contemplation, preparation Change in action, attributed to VaxSocial Longer term health benefits on having the vaccine HCP staff time Government staff time Evaluation team Staff time to execute owned, paid for and intermediary executions Analytical tools Ad spend per campaign and per platform CPC Cost per Click CPM Cost per 1,000 impressions CPA Cost per conversion (e.g. per recruitment, or per conversation with Chatbot) CpPR Cost per Person Reached CpPI Cost per Person Influenced CpAO Cost per Attributed Outcome CpAV Cost per additional vaccine Expectation that everyone can deliver at this level Standard metrics collected as part of paid-for ad spend on social media Manual collection/analysis for owned content Consider comparison to costs for alternatives where known (e.g. recruitment) This will help us identify the most cost-effective platforms, messengers, format (as judged by engagement)

VaxSocial Cost Effectiveness Framework: Enhanced Cost In-kind costs Effectiveness Engagement Capital costs Operating costs Direct advertising & promotion costs Spend on delivery of paid-for advert campaigns, or payments made to intermediaries Influence Impact Long-Term Benefits Costs incurred by other organisations to execute delivery of VaxSocial Development costs One off costs that have a longer life value beyond VaxSocial Creative development costs, or costs to develop the AI chatbot Formative research Regular spend that is needed to keep activity / campaign going Number of people reached and engaging with content or activity Increase in awareness, knowledge, contemplation, preparation Change in action, attributed to VaxSocial Longer term health benefits on having the vaccine HCP staff time Government staff time Evaluation team Staff time to execute owned, paid for and intermediary executions Analytical tools Ad spend per campaign and per platform CPC Cost per Click CPM Cost per 1,000 impressions CPA Cost per conversion (e.g. per recruitment, or per conversation with Chatbot) CpPR Cost per Person Reached CpPI Cost per Person Influenced CpAO Cost per Attributed Outcome CpAV Cost per additional vaccine Should be an aspiration for everyone, though measures of effectiveness may vary Integrate broader costs incurred to execute VaxSocial CpPR only available to those collecting recognition data Need to establish key indicator, or group of indicators to calculate CpPI This will help establish whether (and where) social media is an effective tool for increasing vaccine uptake Evaluation team has some budget to support with advanced analysis in 2-3 case studies

VaxSocial Cost Effectiveness Framework: Expert Cost In-kind costs Effectiveness Engagement Capital costs Operating costs Direct advertising & promotion costs Spend on delivery of paid-for advert campaigns, or payments made to intermediaries Influence Impact Long-Term Benefits Costs incurred by other organisations to execute delivery of VaxSocial Development costs One off costs that have a longer life value beyond VaxSocial Creative development costs, or costs to develop the AI chatbot Formative research Regular spend that is needed to keep activity / campaign going Number of people reached and engaging with content or activity Increase in awareness, knowledge, contemplation, preparation Change in action, attributed to VaxSocial Longer term health benefits on having the vaccine HCP staff time Government staff time Evaluation team Staff time to execute owned, paid for and intermediary executions Analytical tools Ad spend per campaign and per platform CPC Cost per Click CPM Cost per 1,000 impressions CPA Cost per conversion (e.g. per recruitment, or per conversation with Chatbot) CpPR Cost per Person Reached CpPI Cost per Person Influenced CpAO Cost per Attributed Outcome CpAV Cost per additional vaccine Not expected to be in scope, but could be commissioned and would be of interest. Still important to be aware of for sustainability of similar initiatives: e.g. the burden incurred for other partners

Next steps

Immediate next steps Our aim is to ask grantees to submit revised and finalised MEL plans towards end of July, early August 2024. This will allow Ipsos and M&C Saatchi to finalise the global MEL plan and global Theory of Change, and submit a final Inception Report in September. To support this we have a range of activity planned across June and July: Initial MEL/Theory of Change workshops with each grantee (13th-25th June) Three masterclass sessions, open to all, to share best practice on (i) survey design; (ii) social media monitoring; (iii) cost effectiveness. Launch of Global Cross Grantee sessions (first session 6th August) Launch of Online Knowledge and Learning Hub (August) Launch of monthly Pause & Reflect sessions with each grantee (late August) To support a regular feedback loop, we ll then seek to continue with Pause & Reflect sessions with each grantee each month; plan regular Global Cross Grantee Sessions; and support informal networking between grantees. The first quarterly Pulse reports will be due before the end of 2024; these will share performance against MEL plans. We propose an Emerging Findings external event early in 2025.