Understanding the Importance of Survey Representativity

Survey Representativity IPC Capacity Development Pilot Workshop April 2024 1

1.Motivation for doing surveys 2.Total Survey Error Framework 3.Common sampling approaches 4.Common Concerns and Potential Remedies 5.Things to Know about a Survey Session Outline 2

Define what a survey does and why Understand conceptually how a survey approaches answering a question Describe why representativity of a survey is important Describe common approaches ensuring representativity of a survey Describe common concerns with regard to survey representativity Understand that representativity affects how you might use information from a survey Identify and understand standard things to know when using a survey Session Objectives 3

Purpose Quality matters IPC protocols requires analysts to evaluate evidence reliability We want to get analysis right need to limit bias in evidence Objectives of this training Help you understand why surveys are important Help you think critically about survey quality Identify issues that may drive significant concern about survey quality Give analysts a vocabulary for discussing survey quality Not an objective: Teach analysts how to conduct a survey Not an objective: Become an expert in survey research 4

Purpose 5

Purpose Why conduct a survey? Learn more about a topic of interest Ask a structured set of questions Identify key questions for further inquiry Why conduct a representative survey? Apply the results to a specific, and often much larger, population of interest. Thirty out of 300 households in Dollo Zone surveyed have severe household hunger. 10% of households in Dollo Zone, which has a population of 400,000 people, have severe household hunger. Definition A systematic method for gathering information from a sample of [people or households] in order to construct quantitative descriptors of the attributes of the larger population of which they are members. Source: Survey Methodology, 2ndEdition, Groves et al., 2009 6

Purpose Many reasons to conduct a survey Assess the prevalence of children 6-59 months suffering from global acute malnutrition. Assess the crude death rate in a population Assess the percentage of households with poor food consumption or facing severe hunger. Definition A systematic method for gathering information from a sample of [people or households] in order to construct quantitative descriptors of the attributes of the larger population of which they are members. Source: Survey Methodology, 2ndEdition, Groves et al., 2009 7

Purpose Quality matters IPC protocols requires analysts to evaluate evidence reliability We want to get analysis right need to limit bias in evidence Analysts need to understand what they are using in order to understand how to use it Let s imagine survey results indicate that 60% of a sample reports having severe hunger. How does our interpretation of that data change if: We learn the entire sample is composed entirely of newly arrived IDPs in a camp? We learn the sample is only from the host community, and excludes IDPs? We learn the sample is from IDPs only and IDPs represent only 10% of the population in an area? We learn the survey module was calculated incorrectly? We learn the survey implementer chose to select only worst-affected households? We learn the survey implement chose to select only better-off households? 8

Survey objectives 9

Survey Lifecycle - Process Representation Measurement from a smaller number of people to describe the larger population of which they are members. Gather information... Construct Target Population Measurement Sampling Frame Response Sample Edited response Respondents Post-survey adjust. Survey statistic 10

Importance of representativity 11

Importance of representativity Quality matters We want the analysis to be correct and poor quality survey data can undermine analysis The IPC Manual requires a critical examination of the reliability of evidence Assignment of reliability scores (Function 2, Protocol 2.4) Protocol 2.4 identifies factors to consider for the assignment of reliability scores Surveys must meet specific criteria in order to for analysts to assess Soundness of Method (includes consideration of sample sizes and number of clusters) IPC guidance on evaluating Soundness of Method, including estimated sample sizes and clusters, calculated for IPC reliability purposes only. Do not constitute a best practice for the design of any method, including surveys. IPC urges analysts to soundness of all methods, even though the IPC does not have globally comparable parameters for factors that can affect soundness of surveys, such as measurement error, selection bias, field practices, and analytical skills. 12

Total Survey Error Framework What could possibly go wrong? Representation Measurement Construct Target Population Coverage error Validity Measurement Sampling Frame Sampling error Measurement error Response Sample Processing Error Nonresponse error Edited response Respondents Adjustment error Post-survey adjust. Survey statistic 13

Total Survey Error Framework We can make any decision we want. But is it a good one? Representation Ultimately, we make a decision about whether to apply our survey statistic to a larger population. Target Population Coverage error Sampling Frame The elements on the right determine whether that decision is a good one. Sampling error Sample Nonresponse error Respondents Adjustment error Post-survey adjust. Survey statistic 14

Common approaches 15

Common Approaches Surveys can be designed in a lot of different way Surveys can be designed to achieve a lot of different goals, for different populations As a result, there are a lot of different survey designs, and different ways to sample populations This training focuses on a sampling approach known as probability sampling In probability sampling, a survey selects a sample at random. It does this in order to take human decision-making out of who should or should not be selected, in order to avoid intentionally or unintentionally influencing the results. Many surveys conducted in the humanitarian space involve one of two approaches to probability sampling: Simple random sampling Two-stage cluster sampling These approaches and best practices are outlined in the SMART guidance 16

Common approaches Sampling: How we select a few people, because we can t talk to everyone. Assuming you ve correctly selected a target population and assuming your sampling frame population well: We can draw a sample. How do we do that? There are two main ways: Simple random sampling Two-stage cluster sampling We almost always use two-stage cluster sampling. Generally a good idea to follow the SMART guidance. Representation Representation Target Population Target Population represents that target Sampling Frame Sampling Frame Sample Sample Respondents Respondents Post-survey adjust. Post-survey adjust. Survey statistic Survey statistic 17

Common approaches Simple random sampling 10% 10% 10% 10% 10% 10% 10% 10% 10% 10% 18

Common approaches Simple random sampling 10% 10% 10% 10% 10% 10% 10% 10% 10% 10% 19

Common approahces Two-stage cluster sampling 1. 2. Randomly pick Blue or Orange Cluster Randomly select a household from that group Blue Cluster 50% Orange Cluster 50% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 10% 10% 10% 10% 10% 10% 10% 10% 10% 10% 20

Common approaches The problem of unequal chances of selection 1. 2. Randomly pick Blue or Orange Cluster Randomly select a household from that group Blue Cluster 50% Orange Cluster 50% 14% 14% 14% 14% 14% 14% 14% 33% 33% 33% 17% 17% 17% 7% 7% 7% 7% 7% 7% 7% 21

Sampling Often rely on probability proportional to size 1. 2. Randomly pick Blue or Orange Cluster (weighted probability) Randomly select a household from that group Blue Cluster 30% Orange Cluster 70% 14% 14% 14% 14% 14% 14% 14% 33% 33% 33% 10% 10% 10% 10% 10% 10% 10% 10% 10% 10% 22

Common Concerns 23

Common concerns and remedies Introduction Even under ideal conditions, conducting a high-quality survey is challenging In contexts where IPC operates, conditions for surveys are far from ideal Insecurity, unreliable population information, time constraints, poor transportation conditions, large geographic areas, outside interference Such factors make it challenging to implement a high-quality survey Best opportunity for identifying threats to representativity is before a survey is conducted Opportunity to plan for and resolve issues before they occur When issues occur, best to inform analysts Understand the quality of the data they are using Understand potential strengths and limitations of the data 24

Common concerns and remedies Examples Part 1 Little/no description of survey design Inconsistency/mis-stating target population Mismatch between target population and sampling frame Mismatch between sampling unit and the expected respondents Poor population data results in poor ability to employ PPS Poor population data resulting in poor sampling of households No/poor replacement strategy, or poor implementation of replacement strategy Re-allocation of sample within stratum/domain. Remedy options Fix the problem before it occurs Fix the problem, if possible, after it occurs Use the data differently Don t use the data 25

Common concerns and remedies Examples Part 2 Not achieving target sample size Biased/subjective stratification Convenience sampling Non-random sampling Non using PPS for two-stage cluster sampling, or not using post-stratification weights to fix a lack of PPS Lack of accounting for access constraints in survey design Interference in survey implementation Remedy options Fix the problem before it occurs Fix the problem, if possible, after it occurs Use the data differently Don t use the data 26

Common concerns Example: Mismatch between target population and sampling frame Hypothetical Food Security Agency (HFSA) needs to assess food security among all populations in Example State in Country X. HFSA decides to conduct a survey to collect information from households. Approximately 20 percent of households are IDPs living in camps and who are not receiving assistance. In order to ensure the severity of the situation is captured , HFSA decides to conduct interviews only among households living in IDP camps and finds that 80% have a Food Consumption Score than is poor or borderline , and a 40% of households have a Household Hunger Scale of 5 or 6. As a result, HFSA concludes that 70% of the population Example State need humanitarian food assistance. What is the problem with this approach? 27

Common concerns Example: Mismatch between target population and sampling frame Hypothetical Food Security Agency (HFSA) needs to assess acute food insecurity among all populations in Example State in Country X. HFSA decides to conduct a survey to collect information from households. Example State is usually food secure, but approximately 20 percent of households are IDPs living in camps from a neighboring state and who are not receiving assistance. In order to ensure the severity of the situation is captured , HFSA decides to conduct interviews only among households living in IDP camps and finds that 80% of surveyed households have a Food Consumption Score than is poor or borderline , and a 60% of surveyed households have a Household Hunger Scale of 2, 3, 4, 5, or 6. As a result, HFSA concludes that 70% of the population Example State need humanitarian food assistance. What is the problem with the sampling frame and what problem does it cause? HFSA says they want the results to be representative of all populations in Example State, but their sampling frame (IDPs) only consists of households that are likely to have the worst food insecurity. If taken at face value, these results could over-estimate the levels of need in Example State overall. 28

Common concerns Remedies: Mismatch between target population and sampling frame Pre-survey options Post-survey options Re-design survey; adopt different sampling frame In this case, results probably cannot be re- weighted to produce representative results for entire Example State population Conduct two surveys: one for IDPs and one for non-IDPs Divide analysis into two (IDP and non- IDPs) and use survey results only for IDPs Use results to infer lower levels of AFI among non-IDPs 29

Common concerns Uncertainty and bias Survey can tell you how many people in a sample share a common characteristic But a lot of factors influence how much you can infer about a larger population based on that sample In a sample, 50 out of 200 households may report severe hunger If a strong justification is not provided for why those 200 households are likely to represent the broader population, it may be unwise to infer that 25% of the larger population suffer from severe hunger All probability samples inherently carry a degree of uncertainty in their estimates, due to randomness of sample selection Reporting confidence intervals is a best practice When survey design or execution is flawed, uncertainty can grow or create bias Impacts of flaws can be small or large 30

Things to Know 31

Things to Know Part 1 What is the target population of the survey? What kind of survey is being conducted? What is the target sample size? What parameters were used to determine that sample size? Are those parameters reasonable? What is the sampling frame? How well does that sampling frame relate to the target population? Is the sampling frame appropriate for the kind of survey being conducted? What is the approach for sampling from the sampling frame? Was the sample drawn fairly? 32

Things to Know Part 2 Once sample was identified, were there any issues encountered in achieving interviews with the sample? Security, interference, inaccessibility, lack of interviewee interest, interviewees not present? Was there any intentional over-sampling of populations, and if so, were appropriate sample weights applied? Was any other attempt made to correct results through post-survey weighting? 33

Things to Know We can make any decision we want. But is it a good one? Representation Ultimately, humans make a decision about whether to apply our survey statistic to a larger population. Target Population Coverage error Sampling Frame The elements on the right determine whether that decision is a good one. Sampling error Sample Nonresponse error Respondents Adjustment error Post-survey adjust. Survey statistic 34

For More Information Consult expert survey resources: SMART guidance (link) Survey Methodology (2ndedition) by Robert M. Graves, et al. Send questions: Tim Hoffine, EWT Quality Assurance and Methods Advisor 35