Implications of Data Collection Methods on Well-Being Measurement

Measurement of Well-Being IMPLICATIONS OF COLLECTION METHOD

WHAT WILL WE BE DISCUSSING? The objective of this presentation is to build on those presentations of the previous speakers and discuss the implications of the mode of data collection on the measurement of well-being including the international $1.90 per day poverty line (SDG 1.1.1) and the national cost-of-basic-needs poverty line (SDG 1.1.2). 1

WHAT ARE THE GLOBAL RECOMMENDATIONS? From the 2018 FAO guidelines on Food data collection in household consumption and expenditure surveys : For [HIES], a seven-day recall period for food consumption measurement should be adopted. In low- and middle-income countries, recall surveys are generally preferable to diary surveys, which should only be deployed with careful and continuous supervision and should not exceed 14 days. While a well-implemented diary is generally held as the gold standard for food expenditure data collection, there is ample evidence that in low- income settings with a prevalence of illiterate respondents, diaries are often implemented as a series of short recall interviews, with issues of respondent and enumerator fatigue affecting data quality, and with unsustainable implementation costs. 2

IS THIS AN ISSUE IN THE PACIFIC? While there has been more dedicated research in other parts of the world, there is evidence that diary implementation and data quality are an issue in the Pacific. Average number of transactions between week 1 and week 2 3

IS THIS AN ISSUE IN THE PACIFIC? Major problem for diary-keeping surveys is respondent fatigue Example: 2009/10 PNG HIES Fieldwork staggered start over days of the week and months of year Across all households, 37,000 transactions (acquisitions) are listed on day 1 (10 per household) By day 14 fallen to 23,000 or 6 per household with no apparent bundling into fewer, larger, transactions 3.4% per day fall in number of transactions and 4.4% per day fall in the value of transactions Households look poorer the longer we observe them. Headcount poverty rate of 41% if use week 1 diary records, but 47% if use week 2 records The fall in compliance will vary with the opportunity cost of time, and so introduces non-random errors in poverty profile. 4

IS THIS AN ISSUE IN THE PACIFIC? Diary Fatigue, PNG HIES 2009/10 40 10 Number (LHS) Value (LHS) Average Value (RHS) 35 9 Average value per transaction (Kina) Number ('000s), Sum of log value (K'000s) 30 8 25 7 20 6 15 5 10 4 5 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Diary-Keeping Day 5

IS THIS AN ISSUE IN THE PACIFIC? Even countries which generally have high quality data show evidence of respondent fatigue. The example below is from 2009 Tonga HIES. 10 9 Number of transactions 8 7 6 5 4 3 2 1 0 Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat Sun Week1 Week2 Total Home production cash purchase Gifts (received & given away) 6

WHAT ARE THE DIFFERENCES BETWEEN DIARY AND RECALL FOR CONSUMPTION? In reality, even diary surveys collect only part of the consumption and expenditure information through diaries. Low frequency transactions, such as the purchase of assets and durables, payment for utilities, home improvements, etc. are collected through recall. Recall-based Diary-based Food purchases, food away from home, high frequency transactions Diary Recall Low frequency transactions, asset purchases, durables, services Recall Calculated values: Imputed rental value, use value of durables Calculation 7

WHAT ARE THE DIFFERENCES BETWEEN DIARY AND RECALL FOR CONSUMPTION? While recall collects consumption directly by asking how much of item x was consumed during the recall period, diary surveys obtain consumption indirectly by measuring the means of acquisition. Purchases + + - - - = Own-production Gifts received Sales Gifts given Stock increases Consumption. In order to separate expenditure from consumption, it is necessary to collect the opening and closing stocks for major food items. 8

HOW DIFFICULT IS IT TO MEASURE OPENING AND CLOSING STOCKS? Households are more compliant for measuring stocks on day 1 compared to day 14. End stocks seem to be lower than starting stocks even though no reason for net destocking. Two examples 1. 2009/10 PNG HIES tried to measure stocks of over 100 items. Apparent destocking added 6% to the value of food consumption. This extra food consumption caused the headcount poverty rate to drop by four percentage points. 2. 2012/13 HIES for Solomon Islands, calorie totals that included stock changes were 6% higher, on average, with apparent net destocking adding 170 calories per person per day. 9

WHAT ISSUES CONCERN US? For recall surveys, data quality can be negatively impacted by: Omission : Respondents forgetting that certain items were purchased during the recall period. Telescoping: Respondents telescoping purchases into the recall period that in actuality occurred before the start. For diary surveys, data quality can be negative impacted by: Omission : Respondents forgetting that certain items were purchased. Respondent fatigue: Respondents becoming disinterested in the diary process and failing to record transactions. Interviewer fatigue: Interviewers becoming lax about monitoring. Inability to comply: Respondents not being able to complete the diary due to a structural limitation (such as literacy). 10

HOW CAN THE RMI EXPERIMENT HELP US TO UNDERSTAND THESE ISSUES? Because the respondents are randomized across the 5 arms of the experiment, we expect no real differences between the groups. If we do see issues, these differences are likely caused by two main factors: 1. Differing refusal / replacement rates some households choose not to participate based on the time requirements of the method. 2. Differing data quality based on the method which can vary with the characteristics of the respondent. 3. There is also a third possible factor that may play a minor role, which is the failure to correctly randomize on the part of the field teams. 11

THE RMI EXPERIMENT REFLECTS FIELD CONDITIONS (WARTS AND ALL) Researcher-driven survey experiments may be less informative about actual surveys because they are too tightly monitored/controlled The RMI experiment was designed and fielded by the agencies who actually carry out HIES in the Pacific, and as such not all went to plan: 1. Was meant to test bounded recall (to deal with telescoping) but wording stayed as last seven days while gap between the two visits varied (24% 7 days, 53% 8 days, 9% 9 days, 14% other) 2. Interviewers may have had too much freedom to allocate sample to arms, and in allowing reports of zero consumption (separately from refusals) or extremes of apparent food intake (exceeding 25,000 calories per person per day) CAPI could have constrained interviewers more 3. Concordances of stocks to COICOP and food recall groups 12

WHAT WERE THE RESPONSE RATES? replacement consumption completion analyzable completed interviews effective no food targeted rate rate 72 17.3% 75 16 59 81.9% highly monitored PAPI diary 216 19.9% 181 33 148 68.5% low monitored PAPI diary 72 18.8% 64 1 63 87.5% highly monitored CAPI diary 216 18.6% 199 19 180 83.3% CAPI 1 visit recall 216 19.3% 197 13 184 85.2% CAPI 2 visit recall 792 716 82 634 80.1% Total 13

ARE THERE DIFFERENCES BETWEEN ARMS? Yes. Among others: Household sizes are systematically larger among diary households. There is a very low number of female headed households in the low monitored diary. There are few differences in education. Legend 1. 2. 3. 4. 5. High monitored PAPI diary Low monitored PAPI diary High monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 14

WHAT DO OUR RESULTS SHOW? Overview of the findings: 1. Even for the highly monitored diary approaches, we see a decline in the number of items reported in the diary over the course of the 14 days, even if we remove the effects from the day of the week and the location of purchase. 2. We find evidence of systematic differences in the levels of expenditure for items collected through the diary and recall. 3. There is evidence of mode effects between the PAPI and CAPI diaries. 4. The differences that we find are mainly caused by the different response rates. In a hypothetical world where we could overcome those differences, 4 out of 5 methods (excluding low monitored PAPI) show similar results. 5. Accurately measuring stocks is an issue. 15

1. DECLINE IN NUMBER OF ITEMS 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 high monitored PAPI mean 16

1. DECLINE IN NUMBER OF ITEMS 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 low monitored PAPI mean 17

1. DECLINE IN NUMBER OF ITEMS 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 high monitored CAPI mean 18

1. DECLINE IN NUMBER OF ITEMS 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 high monitored PAPI mean low monitored PAPI mean high monitored CAPI mean 19

1. DECLINE IN NUMBER OF ITEMS We see these declines even if we use econometrics (partial linear regression) to remove the effects from the day of the week and the location of purchase. 20

2. SYSTEMATIC DIFFERENCES If we look at the raw comparison, we see the highest total per capita consumption for the CAPI 1 visit recall. We know, however, from the comparisons of the household characteristics that the arms are not interviewing identical populations. Legend 1. 2. 3. 4. 5. High monitored PAPI diary Low monitored PAPI diary High monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 21

2. SYSTEMATIC DIFFERENCES Applying survey weights to compensate for non-response at the island level does not make a large difference in the results. This finding indicates that we would not be able to use weights to compensate for the differences generated by the methods. Legend 1. 2. 3. 4. 5. High monitored PAPI diary Low monitored PAPI diary High monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 22

2. SYSTEMATIC DIFFERENCES The high monitored CAPI diary appears to show the highest level of consumption of food consumed outside the household, but the confidence interval is quite wide due to the low number of observations. mean CI 700 600 500 400 300 200 100 321 259 490 337 266 0 high monitored PAPI low monitoring PAPI high monitored CAPI CAPI 1 visit recall CAPI 2 visit recall 23

3. MODE EFFECTS The only difference between the high monitored PAPI diary and the high monitored CAPI diary is the mode of survey administration, yet there are differences in the results. This finding indicates the presence of mode effects, and is important because it shows that switching from paper to CAPI breaks the comparability of the series. Legend 1. 2. 3. 4. 5. High monitored PAPI diary Low monitored PAPI diary High monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 24

4. CAUSED BY RESPONSE RATES By using re-weighting techniques, we can force certain characteristics to align. In this case we used household size, head literacy, gender of head, years of education for head, broad categories of employment for the head, and the component of the consumption aggregate calculated from survey data (i.e. which does not vary between arms. original re-weighted 25

4. CAUSED BY RESPONSE RATES Using these new weights, we find almost no difference between the total per capita consumption between arms 1, 3, 4, and 5. This finding demonstrates that if it were possible to solve the issue with response rates, the choice of method would matter less. Legend 1. 2. 3. 4. 5. High monitored PAPI diary Low monitored PAPI diary High monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 26

5. MEASURING STOCKS Apparent consumption from diary-keeping arms overstated due to the apparent destocking, which really reflects fatigue 1. Of the n=270 diary-keeping households with analysable results, 236 have starting food stocks reported but only 211 have ending food stocks reported Of those with both starting and ending stocks measured, 2:1 ratio of having starting stocks seeming to be larger than ending stocks 2. Average contribution of the apparent destocking is equivalent to about 4% of total expenditure Removing this component, which is likely due to fatigue and declining compliance, makes poor performance of the low monitored PAPI diary even more apparent (equivalent to almost 400 calories/person/day) C.f. apparent daily per capita calorie contribution of about 130 for the CAPI diary and 220 for highly monitored PAPI diary 27

5. MEASURING STOCKS Comparing the change in estimated value of stocks from opening to closing for the 11 most commonly stocked items, all showed declines, many substantial declines. There is no theoretical reason to expect consistent net destocking, rather we would expect the mean to be about zero, with some households destocking and some increasing. Noodles/pasta Ketchup and sauce Canned mackerel Oil Soya sauce Eggs Flour Chicken leg Sugar Rice Canned tuna -80% -70% -60% -50% -40% -30% -20% -10% 0% 28

WHAT ARE THE FINAL NUMBERS? Annual per capita total consumption 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 3,338 3,719 2,787 3,046 3,662 3,750 4,013 3,639 3,393 3,459 0 raking weights raking weights raking weights raking weights raking weights household weights household weights household weights household weights household weights high monitored PAPI low mointored PAPI high monitored CAPI CAPI 1 visit recall CAPI 2 visit recall 29

WHAT ARE THE IMPLICATIONS FOR $1.90 POVERTY MEASUREMENT? Comparing the 1.90 per day poverty estimations across the five arms show that the low monitored diary generates poverty statistics more than three times higher than the high monitored CAPI diary. mean CI 10% 8% 6% 4% 2% 0% high monitored PAPI diary low monitored PAPI diary high monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall -2% 30

WHAT ARE THE IMPLICATIONS FOR $1.90 POVERTY MEASUREMENT? The differences if the raking weights are applied (which balance out the underlying population) are smaller, but the low monitoring PAPI diary still generates substantially higher estimates. mean CI raking weights 10% 8% 6% 4% 2% 0% high monitored PAPI diary low monitored PAPI diary high monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall -2% 31

WHAT ARE THE IMPLICATIONS FOR NATIONAL POVERTY MEASUREMENT? The results are consistent if we use a relative poverty line set as half of the median real per capita consumption, which would be akin to a national cost-of-basic-needs poverty line. mean CI raking weights 40% 30% 20% 10% 0% high monitored PAPI diary low monitored PAPI diary high monitored CAPI diary CAPI 1 visit recall CAPI 2 visit recall 32

WHAT ARE THE IMPLICATIONS FOR INEQUALITY MEASUREMENT? The differences in inequality are less pronounced at the national level but there are substantial differences at the sub-national level. 40 35 30 25 20 15 10 5 35.0 35.1 35.7 35.0 35.8 31.6 25.4 37.1 30.1 32.1 0 high monitored PAPI high monitored PAPI high monitored CAPI high monitored CAPI CAPI 1 visit recall CAPI 1 visit recall CAPI 2 visit recall CAPI 2 visit recall low monitored recall low monitored recall national Majuro 33