Impact of Survey Design on Dietary Energy Distribution

Analysis of impact of survey design on distribution of dietary energy consumption Pacific Statistics Meeting Board Auckland, New Zealand 23-24 May 2019 Nathalie Troubat (FAO) Bertrand Buffiere (SPC) Michael Sharp (SPC)

Why looking at the impact on distribution of DEC? The indicator SDG 2.1.1 (Prevalence of Undernourishment (PoU)) is looking at the percentage of the population whose energy consumption is lower than the energy needed to be in good health and maintain a certain level of physical activity PoU is based on a parametric approach assuming that energy is distributed according to a log normal with a mean corresponding to average dietary energy consumption in a population and a coefficient of variation reflecting disparity of the energy consumption in the population any change in the survey design affecting the distribution of DEC will have any change in the survey design affecting the distribution of DEC will have an impact on the SDG2.1.1 and its comparability over time an impact on the SDG2.1.1 and its comparability over time

Questions we will try to address Does it make a difference? We looked at the impact on the dietary energy consumption of a change in the survey design Pre-visit Use of CAPI High supervision Inclusion of FAFH Fatigue Which module performs the best with respect to our benchmark? We compare the prevalence of undernourishment estimated for each of the five arms using ARM 2 as benchmark and we look at the impact on diet diversity

Does it make a difference? We looked at the impact of having a pre-visit to the recall, including a section on FAFH, including stocks, increasing number of visit and potential effect of fatigue when filling the diary RECALL DIARY ARM3 (excl. stocks & excl. FAFH) ARM4 (excl. stocks & excl. FAFH) ARM2 (incl. stocks & excl. FAFH) ARM2 (excl. stocks & incl. FAFH) ARM2 - week 1 (excl. stocks & excl FAFH) ARM2 - week 2 (excl. stocks & excl. FAFH) ARM2 (excl. stocks & excl. FAFH) ARM1 (excl. FAFH) ARM5 (excl. FAFH) ARM1 (incl. FAFH) PAPI CAPI Recall Diary 14 days 7 days highly supervised pre visit x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x Stocks FAFH x x x x x x x pre visit Fatigue CAPI supervisio FAFH FAFH stocks

Does it make a difference to have a pre-visit? With no pre-visit the percentage of non response is slightly higher and more disparity in distribution but after correction for outliers the non bounded recall shows a higher average DEC but the probability that an observation from ARM1 will be higher than that of ARM5 is only equal to 54%* 20,000 Average dietary energy consumption per capita per day 15,000 10,000 5,000 No strong statistical evidence that the average DEC from ARM 5 is always under estimated compared to ARM 1 0 ARM1 & no fafh ARM5 & no fafh Average dietary energy consumption per capita per day ARM1 excl FAFH ARM 5 excl. FAFH 3200 3000 Average DEC (kcal/capita/day) Standard deviation Median DEC Coefficient of variation Percentage of "outliers" 2775 1738 2406 63 14% 2529 1705 2264 67 14% 2800 2600 2400 ARM1 & fafh ARM5 & fafh scenario 95% confidence intervals * Based on Wilcoxon rank sum test

Does it make a difference to have a section on FAFH? Inclusion of the specific section on FAFH increases drastically the average and median DEC in both diary and recall but probability to have an observation higher when FAFH is included is 60% in case of ARM 2 compared to 56% in ARM1 FAFH tends to be better captured in case of a well supervised diary than recall Average dietary energy consumption per capita per day 3500 3000 2500 FAFH tends to be beter captured in ARM2 2000 ARM1 & no fafh ARM2 & no stocks & fafh ARM1 & fafh ARM2 & no stoks & no fafh scenario 95% confidence intervals

Does it make a difference to use CAPI instead of PAPI The use of PAPI increases the number of non response and variability in the distribution of in house acquisition is high (CV higher of 110% with PAPI compared to 90% with PAPI) ARM2 & no stoks & no fafh The use of PAPI seems to underestimate the average DEC with and without inclusion of stocks and FAFH ( (probability to have a higher DEC from CAPI compared to PAPI is 64% when both stocks and FAFH are included) ARM3 & no stocks & no fafh ARM2 & stocks & FAFH 5000 ARM3 & stocks & FAFH Average dietary energy consumption per capita per day 4000 -10,000 -5,000 Average dietary energy consumption per capita per day 0 5,000 10,000 15,000 3000 2000 1000 ARM3 & no stocks & no fafh ARM2 & no stoks & no fafh ARM2 & stocks & FAFH ARM3 & stocks & FAFH scenario 95% confidence intervals

Does it make a difference to increase number of visits? With a less supervised diary we have a slightly higher percentage of non response and higher variability ARM3 & no stocks & no fafh The decrease in the number of visits tends to decrease the amount of calories reported (probability to have a higher DEC with 7 visits compared to 3 visist is close to 60%) ARM4 & no stocks & nofafh ARM3 & stocks & FAFH 3000 ARM4 & stocks & FAFH Average dietary energy consumption per capita per day 2500 -5,000 Average dietary energy consumption per capita per day 0 5,000 10,000 15,000 ARM3 excl FAFH excl stocks ARM4 excl FAFH excl stocks 2000 ARM3 & stocks & FAFH ARM4 & stocks & FAFH 1500 ARM3 & no stocks & no fafh ARM4 & no stocks & nofafh ARM3 & stocks & FAFH ARM4 & stocks & FAFH 120% 16 119% 30 CV of original distribution number of missing number of values higher than 8000kcal/person/day sample size Percentage of "outliers" 117% 16 133% 43 3 7 5 10 75 25% 181 28% 75 28% 181 22% scenario 95% confidence intervals

Does it make a difference to include stocks? If stocks are correctly reported and analyzed then inclusion of stocks increases drastically the average DEC* (excluding FAFH) and slightly reduces the overall disparity in the distribution of DEC. Not collecting stocks would have led to an under estimation of consumption that occurred over the last 14 days as many consumers might have not purchased food but consumed out of their stocks Average dietary energy consumption per capita per day 3500 3000 2500 scenario mean(ppd_cal) sd(ppd_cal) 2000 ARM2 & no stoks & no fafh 2380.164 1417.067 ARM2 & stocks & no fafh 2899.158 1474.772 ARM2 & no stoks & no fafh ARM2 & stocks & no fafh scenario 95% confidence intervals *(median are not statistically different at 7% level and probability to have a higher DEC when stocks are included is 60%)

Effect of fatigue? A decrease in number of records reported in week 2 compared to week 1 can be observed. The effect is more pronounced for not well supervised diary than well supervised diary. But the effect of fatigue seems to affect more in house section of the diary than the section on FAFH

Second question: which module departs the most from the benchmark in terms of impact on dietary energy consumption and diversity of the diet? We first need to treat the five distributions for extreme values and define our benchmark - We dropped all the 0 and negative consumption from the analysis as all the five arms are supposed to collect consumption and not acquisition - We trimmed the upper tail of the distribution using 1.5 the interquartile range Average Dietary energy consumption (kcal pcd) 2968 3450 2657 2434 2697 Number of missing and negative values final number of extreme values number of observation s Coefficient of variation Standard deviation Minimum Maximum ARM 1 ARM 2 ARM 3 ARM 4 ARM 5 19 3 16 30 14 8 3 5 8 7 172 58 54 143 176 61% 48% 57% 83% 65% 1800 1670 1502 2019 1744 18 736 192 18 6 7656 7109 7738 8245 7792

Impact on the prevalence of undernourishment

. ARM 1 seems to be the one for which average DEC is closest to the benchmark 4000 Average dietary energy consumption (kcal/capita/day) 3500 3000 2500 2000 1 2 3 4 5 survey arm 1 to 5 95% confidence intervals

Impact on diet Analysis of the contribution of energy from main food groups to the total energy (incl. FAFH) ARM 1 vs ARM 2 Higher contribution of cereals, meats, oils and fats and fruits and vegetables to the total energy consumed in case of recall than diary 60% arm2 arm1 50% 40% 30% 20% But contribution of processed food consumed away from home better captured in diary as well as fish and oil crops (coconut) 10% 0% Cereals and producs processed food (from FAFH section) meat (incl canned and processed) fish and fish products oil crops oils and fats sugar and syrups fruits and vegetables processed food (from recall/diary section)

Impact on diet Analysis of the contribution of energy from main food groups to the total energy (excl. FAFH) - ARM 1 vs ARM 2 When FAFH is excluded from the analysis then the contribution of main food groups to total dietary energy reverses: cereals and meets seem to be better captured in house consumption section of the diary than that of the recall 70% 60% arm2 arm1 50% 40% 30% 20% 10% contribution of fish and oil crops and sugar better captured in recall than in diary 0% Cereals and producs meat (incl canned and processed) fish and fish products oil crops sugar and syrups fruits and vegetables oils and fats processed food (from recall/diary section)

Limit to the analysis Correction of outliers (we dropped 0 values and cut at 1.5IQR, other threshold might lead to slightly different results especially for ARM 4) Results are conditioned to quality of the field work Size of the household not homogenously distributed throughout the 5 arms Different sample size ARM2 (diary 14 days 7 visits CAPI FAFH) 5.4 3.1 64.0 ARM3 (diary 14 days 7 visits PAPI FAFH) 5.31 2.67 ARM4 (diary 14 days 3 visits PAPI FAFH) 5.28 2.45 ARM5 Current version (diary 14 days PAPI no FAFH no stocks) 5.31 2.67 75 ARM1 (Recall 7 days bounded CAPI FAFH) 4.71 2.61 197 (recall 7 days not bounded FAFH) 4.38 2.39 199 average HH size standard deviation 75 181 Sampled HH

Conclusion FAFH has a significant impact on average DEC and it is recommended to add an individual section on FAFH in the HIES Recall bounded does not seem to improve quality of the data collected (but results are subject to how ARM 5 was administered) Use of CAPI reduces non response and variability in the distribution With more visits the average amount of energy reported is higher than when diary is less supervised Effect of fatigue tends to affect more in house reporting than FAFH

Thank you !