Assessing Improvement in Instrument Design: Research Study Overview
This research study delves into the assessment of improving instrument design, particularly focused on a re-designed travel record to evaluate its effectiveness. The presentation outlines the background, rationale, research stages, findings, and conclusions of the study, highlighting evaluation methods, redesign principles, and the comparison between the existing and new travel records. Through a combination of quantitative and qualitative analysis, the study aims to determine whether the redesigned travel record provides a more effective approach than the original design.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
How do we assess whether we are improving instrument design? Alice McGee
How do we assess whether we are improving instrument design? Using multiple methods to evaluate whether a re-designed travel record was 'better' than the existing one
Presentation outline Background to NTS and Travel Record Rationale for re-design Research stages A - Evaluate the existing record B - Record re-design C - Evaluate the new record Findings Conclusions
Background to study National Travel Survey (NTS) Measures travel patterns and behaviour Face to face interview - household level Seven-day Travel Record - individual level Adult and child versions Nature of information - very detailed
Rationale Concerns surrounding data quality High proportions of item missing Anecdotal evidence suggesting measurement error International evidence supporting alternative designs Axhausen: Column based approach
Research stages A - Evaluate the existing record B - Re-design the record C - Evaluate the new record to assess whether the new was better than the original
A - Evaluation methods Quantitative 1. Secondary analysis of survey data Qualitative 2. Workshops with interviewers 3. Cognitive interviews 4. Respondent suggestions 5. Data editor comments
B - Re-designing the record Two underpinning elements Existing literature: theoretical basis for how people approach self-completion documents Empirical evidence from Stage A Existing literature Jenkins et al: Readers and Skimmers typology Principles of good questionnaire design
B - Aims of the re-design Make the task easier and more straightforward for all types of respondent through: providing navigational guides to help respondents find their way around the record improve the way information was organised and displayed NB: unable to address the more conceptual problems
B - Main changes to record Colour and layout more user-friendly Instructions brought together Visual guides (arrows) and clearer signposting Pictorial images Tick boxes Column heading posed as questions
C - Evaluation methods Quantitative 1. Comparison of sample of data from existing Record and pilot data (100 respondents) 2. Respondent debriefing questionnaires Qualitative 3. Cognitive interviews 4. Pilot interviewer comments 5. Data editor comments
Hypotheses All evidence sources reviewed and triangulated Hypotheses tested at Stage C: Cognitive respondents would experience narrower range of problems Problems relating to information organisation and navigation would diminish among cognitive respondents Systematic review of respondent error would show the new Record produced lower error rates
Findings and conclusions All three hypotheses proved correct Triangulation of quantitative and qualitative evaluation methods showed new Record to be better than the original Study demonstrates how a combination of evaluation methods can be brought together to substantively conclude the best approach
Future work Evaluation of Event History Calendar approach (EHC) England Longitudinal Study of Ageing (ELSA) Retrospective interview following Wave 3 Interactive calendar administered through CAPI Package of work involving multiple evaluation methods (quantitative and qualitative)
EHC - Evaluation methods Quantitative 1. Behaviour coding 2. Secondary analysis of survey data 3. Respondent debriefing questionnaires (closed responses) Qualitative 4. Interviewer diary 5. Respondent debriefing questionnaires (open responses) 6. Quasi transcripts 7. Behaviour coding (supporting notes)
References Axhausen, K (1995) Travel Diaries: An annotated catalogue, 2nd edition Jenkins, C.R., Ciochetto, S. and Davies, W. (1992) 'Results of cognitive research on the public school 1991-92 field test questionnaire for the schools and staffing survey'. Unpublished Jenkins, C.R. and Dillman, D.A. (1995) 'Towards a theory of self- administered questionnaire design' in Lyberg et al (1997) 'Survey Measurement and Process Quality' Wiley McGee, A, Gray, M and Collins, D (2006) NTS Travel Record Review Stage 1 DfT, web only McGee, A, Gray, M, Andrews, F, Legard, R, Wood, N and Collins, D (2006) NTS Travel Record Review Stage2 DfT, web only
