Managing Survey Quality with Computer Audio Recording Technology

RTI International Computer Audio Recording: A Practical Technology for Managing Survey Quality M. Rita Thissen, Hyunjoo Park, Mai Nguyen International Total Survey Error Workshop Qu bec City, QC, June 2011 www.rti.org RTI International is a trade name of Research Triangle Institute.

RTI International Overview Survey process steps Technology and methodology of CARI* Incorporating CARI into the survey process Addressing structural and functional errors CARI adoption by the US Census Bureau Overall implementation plan American Community Survey Content Test use of CARI for behavior coding User feedback and future possibilities *CARI: Computer Audio-Recorded Interviewing

RTI International Survey Process Steps 1. Define research objectives 2. *Choose methods of data collection 3. *Detailed planning, including quality assurance plan 4. *Construct and pretest questionnaire 5. *Collect data 6. *Analyze and report * All except the first step may be affected by use of audio recording methods

RTI International CARI Technology Collect audio recordings during actual interview Capture screen image and response data for comparison with recorded audio Monitor recordings centrally after the interviewing session ends Quantify content through coding Review coding results and choose what action to take, if any

RTI International CARI System Concepts Support for total error management Multimode Multi-purpose More efficient and less burdensome than real-time monitoring

RTI International Survey Step 2: Choose Data Collection Methods ISSUE Concern about mode effects between telephone (CATI) and in-person (CAPI) interviews APPROACH Audio recordings for comparison across electronic modes Determine what data points might be sensitive to mode effects Plan to record those questions and responses, plus buffering by recording adjacent items Difficulty of data capture in semi-structured interviews or free-response items Capture open-ended responses through audio recording for later coding Standardized coding, such as occupation and industry, or drug and disease codes Allow standardized coding by centrally-located experts instead of by interviewers

RTI International Survey Step 3: Quality Assurance (QA) Plan ISSUE QA plan is for detection, control and remediation of error APPROACH Build CARI capabilities into QA processes for data quality and performance management Can use audio recordings for primary data entry or as a form of double data entry to confirm/correct data from CATI or CAPI Data entry challenges: Other, specify questions, long lists of response options, Code all that apply , or free-form responses Effort and expense of live- monitoring, field observation and verification May be able to reduce costs through use of recordings for performance review

RTI International Survey Step 4: Construct and Pretest the Questionnaire ISSUE Question-level problems: Wording, ambiguity, lack of clarity APPROACH Use recordings and behavior coding during pretest and production to identify issues Translation effects: Dialects, regionalisms or country- specific vocabulary may not be recognized until production Review of production responses by bilingual staff can confirm correctness or identify problems Key data points need extra attention Select questions to record, including buffer zones as needed.

RTI International Survey Step 5: Collect Data ISSUE Managing interviewer performance & adherence to protocol APPROACH Frequent review of a sample of cases helps keep an eye on interviewing Motivating interviewers when the going gets rough, especially late in a cycle Positive feedback can reinforce good behavior and provide encouragement Responsive design or adaptive total design; when subpopulations don t respond Review of recordings from targeted subpopulations may give insight into reluctance

RTI International Survey Step 6: Analyze and Report ISSUE Quantifying operational data for methodology reports APPROACH CARI coding contributes to paradata metrics Explaining difficulties or unusual situations encountered Audio recordings may provide insight into or detailed examples of the challenges Estimating survey error Coded monitoring results offer a way to standardize measurement of subjective error, such as consistency of questionnaire delivery

RTI International Addressing Structural Errors Translation errors Regional effects, such as vocabulary differences (Do you drink soda, pop or soft drinks? Would you drink from a water fountain?) Logic errors: complex questionnaire logic may hide errors, in question sequences rarely followed Values computed based on earlier responses and then used as fills; errors may be hard to detect during testing

RTI International Addressing Functional Errors Human error: Data entry mistakes, presentation or mode effects, respondent reactions Advance warning of mode effects, indications prior to analysis Additional information about item-level non- response, including both don t know answers and refusals

RTI International CARI Adoption by the US Census Bureau CARI Interactive Data Access System Multiple-year project, 2009 - 2012 Stage 1: Develop behavior coding and quality assurance (QA) interfaces (completed) Stage 2: Field test for behavior coding, recommendations for revision (completed) Stage 3: Enhancements and revisions for BC and QA and addition of a Coaching component (in progress) Stage 4: Field test for QA and Coaching (future)

RTI International Behavior Coding Field Test American Community Survey (ACS) Content Test Data collection, 2010 Mail: August September CATI: October CAPI: November Behavior coding immediately after data collection Dec 2010 March 2011 Data analysis in progress at the Census Bureau (Contact: Joanne Pascale)

RTI International ACS Content Test CATI CAPI Sample size Completed + partial cases Recording consent rate Interviews recorded Interviews selected for coding Training cases Cases coded out as inaudible Interviews coded Recordings coded 23,673 4,523 88.5% 4,005 807 32 48 727 27,163 15,202 6,384 64.8% 4,137 708 0 7 701 22,676

RTI International Coding Page Compact, feature-rich interface Playback for audio Option to view image Navigation bar at left (blue) for non-sequential access to recordings Next / Prev buttons for sequential access Dynamic code lists are structured into categories, groups and codes Grid display summarizes codes assigned Room for notes

RTI International General Feedback Features that the CARI System users found most valuable included the following: Unobtrusive nature of CARI Ability to select cases (or questions) for review from among those recorded Flexibility in defining what behaviors to code Image display offering the exact display of wording (including fills) as well as the actual data entry value Real-time monitoring of coding Data available for extraction at any time

RTI International Future Possibilities for Error Management Greater availability of recordings as a routine part of data collection Expanded use of recordings in performance management and training or retraining of interviewers Expanded use of CARI paradata in survey management Development of low-burden performance metrics for interviewers and coders Creation of an error profile for a survey And whatever else you can imagine!

RTI International Acknowledgements This project would not have been possible without the support of many people at the Census Bureau and RTI. We especially thank Sherry Thorpe, CARI Project Manager, Joanne Pascale, and the ACS Office of the Census Bureau. For more information, contact: Rita Thissen rthissen@rti.org Hyunjoo Park mpark@rti.org Mai Nguyen mnguyen@rti.org