RMI HIES Experiment: Optimizing Data Collection Methods
The RMI HIES Experiment focused on testing new technologies and methodologies to enhance the collection of socioeconomic data in Pacific nations. By utilizing computer-assisted personal interviews (CAPI) and innovative survey modules, the experiment aimed to improve data accuracy and efficiency, especially in areas with limited connectivity. The results have paved the way for a revised HIES data collection approach in line with the 2030 Sustainable Development Agenda.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
RMI HIES Experiment Pacific Statistics Standing Committee meeting 30 October 2019 Michael Sharp
Reference PSMB October 2018 Paper 5 PSMB May 2019 papers and presentations PSMB May 2019 report and recommendation
Background HIES is conducted in all PICTs with multiple objectives, including: rebasing GDP; rebasing the CPI; conducting poverty and nutrition analysis; producing socioeconomic data to form policy across multiple sectors; and to populate indicators, including contributing data to report against almost one-fifth of the indicators under the Pacific Sustainable Development Goals (SDG). The regionally standardized HIES data collection method and associated instruments were developed under the guidance of a technical working group in 2011, however with technological advancement, changing consumption patterns and, in response to the increasing demand for high quality disaggregated data, the regionally standardized HIES methodology is no longer fit-for-purpose and needs revision. In response to this, a HIES experiment was conducted in the Republic of the Marshall Islands (RMI) to test optimized methods to collect socioeconomic data. The results of the HIES experiment have allowed the Pacific Statistics Methods Board to make recommendation for a new regionally standardized HIES data collection methodology in the Pacific region under the 2030 Sustainable Development Agenda.
The RMI HIES Experiment There were three main components to the experiment: Test the use of new technology in the form of using tablets to collect data through computer assisted personal interview (CAPI) and the potential use of new data backup protocols in small island states where electrification and internet connectivity is limited; Test the inclusion of complementary survey modules to increase the production of complementary data through HIES so as to avoid the need for stand-alone surveys; and Test new methodologies to collect consumption data through HIES.
New technology The RMI HIES experiment tested the use of computer assisted personal interview (CAPI) to collect HIES data. Given the complexity of the HIES questionnaire and the significant period of time that field teams spend in areas where internet connectivity and electrification is limited, it was necessary to test whether CAPI is an appropriate means for the capture of HIES data. The experiment also tested the use of a satellite phone for data backup in remote island locations where there is no internet.
Complementary survey modules In order to meet demand for more dynamic data and to eliminate the need to conduct stand- alone surveys, the below new survey modules were tested for inclusion in the core HIES questionnaire. Food away from home: recall and diary-based modules were tested to encourage more comprehensive reporting of food acquired and consumed away from the dwelling. Partakers: recall and diary-based modules were tested to capture partakers in meals in order to improve consumption estimates. Stocks: collection of opening and closing food stocks before and after the two-week food acquisition diary is implemented this will allow for estimation of apparent consumption, rather than just acquisition. Labour module: newly designed module in accordance with the resolution concerning the International Classification of Status in Employment (ICSE-18). Food Insecurity Experience Scale (FIES): to measure experiences of moderate-to-severe food insecurity and to report against SDG indicator 2.1.2. Shocks: household exposure to shocks through adverse environmental, economic and social change. Disability: addition of the Washington Group short set of questions to identify vulnerable populations. Improved modules: enhanced health and fisheries modules and means to capture information to better estimate household consumption of fixed assets.
Consumption data (diary v recall) Five experimental arms implemented to determine the efficacy of collecting consumption data through diary and recall means. ARM 1 - Recall CAPI: respondents recall consumption and value of food and non-food items in the last seven days. Answers are entered directly into the tables by enumerator. ARM 5 - Bounded recall CAPI: same as Recall CAPI, however the enumerator visits the household seven days before conducting the recall interview to ask the household to remember what was acquired/consumed over next seven day period. ARM 2 - Diary CAPI high monitored: household keeps a 14-day diary of all food and non-food items that the household acquired each day, plus opening and closing stocks. Enumerator visits the household every second day to enter the diary data into the tablet. ARM 3 - Diary PAPI high monitored: same as Diary CAPI high monitored, except the diary data is not entered into the tablet during the visit to the household; this is the current method. ARM 4 - Diary PAPI low monitored: this is the same as Diary PAPI high monitored, however the enumerator only visits the household to drop the diary off on day one, then pick up the first week diary on day seven and to drop the second week diary, then pick up the second week diary on day 14.
Research Questions The results of the RMI HIES experiment are expected to guide the development of an updated regionally standardised HIES, generally answering the following questions: Is CAPI an appropriate data capture means for HIES? Do the new survey modules produce the data required by the user? Does diary or recall produce more comprehensive consumption data and which implementation protocol will result in the most efficient production of consumption data? These questions are to be answered via analysis of the produced data and via a survey of the implementing NSO and field team.
Location RMI was selected as the location to conduct the HIES project as RMI is scheduled to conduct an official national HIES in 2019, so the HIES project will serve to build capacity of Economic Policy, Planning and Statistics Office (EPPSO) in conducting complex statistical collections in advance of the 2019 HIES. From a regional perspective, the geographical context of RMI will allow the testing of methods and technologies in remote island locations, which is key to determining field protocols in low- and-no connectivity environments, as well as a densely populated urban areas, such as in Ebeye.
Response Rates No food consumptio n Effective completion rate Replaceme nt rate Analyzable interviews Targeted Completed highly monitored PAPI diary (Arm 3) low monitored PAPI diary (Arm 4) highly monitored CAPI diary (Arm 2) CAPI unbounded recall (Arm 1) CAPI bounded recall (Arm 5) Total 72 17.3% 75 16 59 81.9% 216 19.9% 181 33 148 68.5% 72 18.8% 64 1 63 87.5% 216 18.6% 199 19 180 83.3% 216 19.3% 197 13 184 85.2% 792 716 82 634 80.1%
Feedback from EPPSO and the field team Use of CAPI and the satellite phone General assimilation of the use of tablets to conduct HIES. CAPI v PAPI: Use of CAPI is easier than PAPI when collecting individual-level, multiple choice and other(note) data types. CAPI (Survey Solutions) v CAPI (Other): Use of Survey Solutions software was equally as easy to experience using other CAPI software. Electrification: not a significant challenge in the RMI HIES Experiment. Power banks recommended. Internet connectivity: somewhat problematic, particularly in rural atolls. The use of the satellite phone to connect to the internet was generally unsuccessful. Inclusion of complementary survey modules No significant problems reported (FIES). More training on the implementation of specialised modules is required. Diary or recall Preferred methodologies for the collection of consumption data were, in order of preference, Arm 1 (7-day food consumption recall), followed by Arms 2 and 3 (highly monitored two week diary, on CAPI and PAPI respectfully). Some concerns about the respondent s ability to accurately recall consumption were raised about the use of 7-day recall. Concerns about respondent fatigue through multiple visits and low response to the diary were raised about Arms 2, 3 and 4.
Complementary survey modules Newly developed economic characteristics sequence: produces data required by ILO and in accordance to international standard. Food insecurity experience scale: needs adaptation to the Pacific context, however produces data required to report against SDG 2.1.2. WG-6: adds an important dimension to allow for consumption, poverty and income to be disaggregated by disability status. Partakers: impact mean dietary energy consumption (ref fig: mean DEC/capita/day).
Complementary survey modules Food away from home: adds to consumption and DEC (ref fig: % of DEC from FAFH). Stocks: if diary is adopted, the collection of opening and closing food stocks is necessary, but this needs very good management.
Diary or recall A transition between diary and recall approaches need not have adverse implications for the use of HIES data for: CPI, national accounts, poverty, food security and diet composition. Low-monitored diaries yield inferior data quality; the cost of recall is markedly lower than the cost of diary. Despite the high cost of the diary-keeping surveys, they yield data that are, overall, of worse quality. The effective completion rate with less monitored diaries is only two-thirds and apparent consumption is significantly lower (and poverty higher) compared to all other modules. The highly monitored diaries give similar results to using recall, but at much greater cost. Total cost per enumerati on area $4,626 $10,248 $8,331 $19,564 $7,624 $18,847 $7,918 $19,184 $6,031 $14,876 Total cost per household Indexed $289 $577 $1,666 $3,261 $1,220 $3,016 $551 $1,163 $380 $850 Arm Survey Type Location Urban Rural Urban Rural Urban Rural Urban Rural Urban Rural 1.00 1.00 5.76 5.65 4.22 5.22 1.91 2.01 1.31 1.47 1 1-visit 7-day recall, CAPI 2 7-visit 14-day diary, CAPI 3 7-visit 14-day diary, PAPI 4 3-visit 14-day diary, PAPI 5 2-visit 7-day recall, CAPI
Use of CAPI in HIES PSMB agreed that: CAPI is an appropriate means to capture data through HIES. Switching from paper to tablet-based collections constitutes a methodological break. The switch from collecting data using paper to using tablet-based forms therefore requires careful consideration. Should other methodological changes be made (e.g. changing from diary to recall to collect consumption data), they should be made once only and at the same time. PSMB recommended: Major methodological changes, such as switching from paper to tablet-based collection occur infrequently and that once the decision to make a major methodological change is made, the implementing PICT should maintain the new methodology for subsequent surveys so as not to break trends. Frequent methodological changes are discouraged.
Complementary survey modules PSMB recommended: Stocks: a Stocks Module is implemented when using a food acquisition diary in HIES; need for improved development, training and implementation. Partakers: a Partakers Module is implemented as a core HIES module. Food away from home: an individual-level FAFH Module is developed and implemented as a core HIES module. Economic activity: all PICTs note and manage series breaks and ensure that national-level reporting requirements and data needs are met; ILO and SPC develop a minimal version of this Labour Force Module as soon as possible, to be reviewed by PSMB, for inclusion in the 2020 round of population censuses in the Pacific. WG-6: module be included as a core HIES and census module. FIES: incorporated as a core HIES module; consideration of cultural adaptation of the module and development of Pacific-oriented resource materials, and improved training of field staff in implementing the module. Use value of durables: include questions in HIES to collect estimations of the use value of durables, given the application of this information to poverty measurement and national accounts. Anthropometric: identify alternative sources of anthropometric data; the inclusion of a module that collects anthropometric information is the decision of the implementing PICT.
Diary or recall PSMB: Agreed it is critical that NSOs are properly advised when deciding to move from a diary to a recall method, including how to accommodate the break in time series data; In light of the cost savings, and the consistency and quality of data generated through use of recall in the RMI HIES experiment, recommended PICTs consider the option of shifting from diary to recall-based methodologies for the collection of consumption data; Recommended that PICTs not make the change until they are ready, and then embrace the change (i.e., not switch back to diary for the next survey); Noted that there is still much to learn from the results of the RMI HIES experiment; Recommended a HIES manual be developed following the RMI experiment; Noted the need to identify a venue for communication of the results of the RMI HIES experiment to PICTs.
