Survey Design by Matthew Pflipsen
In this survey design presentation by Matthew Pflipsen, MD, MHPE, the process of developing effective survey instruments is explored. Various steps, including literature review, interviews, and item development, are discussed. The goal is to create clear, understandable survey questions that accurately capture respondents' perspectives.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Survey Design Matthew Pflipsen, MD, MHPE
Why are survey instruments so popular?
The overarching goals are to develop a set of items that every respondent will Interpret the same way Respond to accurately Willing and motivated to answer
How can we do better? A Seven Step Process for Survey Design
Step 1: Literary Review Clearly define the construct Do measures of the construct already exist? Begins to establish validity
Step 2: Conduct interview and/or focus groups Does the conceptualization of the construct match how prospective respondents think about it?
Step 3: Synthesize the literature review and interview/focus groups Does the conceptualization of the construct makes theoretical sense to scholars in the field AND uses language that the population of interest understands?
Step 4: Develop items To ensure items are clear, understandable, and written in accordance with current best practices in survey design
47.4% Unevenly spaced response options 44.7% Used agreement response options 33.5% Did not label all response options 32.7% Nonsubstantive response options formatted together with substantive response options 16.6% Items were multibarreled 37 published articles in Academic Medicine, Medical Education and Advances in Health Sciences Education using survey methodology representing up to 571 Likert rated questions1
The high cost of health care is the most important issue in America today. strongly disagree disagree neutral agree strongly agree
Avoid Agreement Responses and statements The high cost of health care is the most important issue in America today. strongly agree agree neutral disagree strongly disagree
Agreement Responses Acquiescence Satisficing Does not emphasize the construct
Instead Use Construct Responses How important is the issue of high health care costs in America today? extremely important quite important moderately important slightly important not at all important
Use questions instead of statements How important is the issue of high health care costs in America today? extremely important quite important moderately important slightly important not at all important
Avoid reverse-scored items How important is the issue of high health care costs in America today? not at all important slightly important moderately important quite important extremely important extremely important quite important moderately important slightly important not at all important
Avoid using a leading or biased question How important is the issue of high health care costs in America today? extremely important quite important moderately important slightly important not at all important
How do you rate your attendings lack of knowledge and confidence about the discussion topic?
Avoid creating a negatively worded item Negatives can be easily overlooked Cognitively challenging Increases measurement error
Avoid Multibarreled items How do you rate your attending s knowledge and confidence about the discussion topic?
Avoid Multibarreled items How do you rate your attending s knowledge and confidence about the discussion topic? How do you rate your attending s knowledge? How do you rate your attending s confidence? How do you rate your attending s experience?
Choose formats that answer the questions of interest Which of the following topics are you interested in studying? Check all that apply
Choose formats that answer the questions of interest Which of the following topics are you interested in studying? Please check Yes or No for each one
Choose formats that answer the questions of interest Rank order with 1 being most interested in 5 being least interested
Step 5: Conduct Expert Validation Representativeness Clarity Relevance Distribution
Step 6: Conduct cognitive interviews Comprehension Retrieval Judgement Selection
Step 7: Conduct pilot testing Step 5: Conduct Expert Validation Step 6: Conduct Cognitive Interviews Step 7: Conduct Pilot Testing
References 1. Artino AR Jr, Phillips AW, Utrankar A, Ta AQ, Durning SJ. "The Questions Shape the Answers": Assessing the Quality of Published Survey Instruments in Health Professions Education Research. Acad Med. 2018 Mar;93(3):456-463. 2. Artino AR Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014 Jun;36(6):463-74. 3. Sullivan GM, Artino AR Jr. How to Create a Bad Survey Instrument. J Grad Med Educ. 2017 Aug;9(4):411- 415. 4. Gehlbach H, Artino AR Jr. The Survey Checklist (Manifesto). Acad Med. 2018 Mar;93(3):360-366. 5. Rickards G, Magee C, Artino AR Jr. You Can't Fix by Analysis What You've Spoiled by Design: Developing Survey Instruments and Collecting Validity Evidence. J Grad Med Educ. 2012 Dec;4(4):407-10. doi: 10.4300/JGME-D-12-00239.1. PMID: 24294413; PMCID: PMC3546565. 6. Artino AR Jr, Youmans QR, Tuck MG. Getting the Most Out of Surveys: Optimizing Respondent Motivation. J Grad Med Educ. 2022 Dec;14(6):629-633. doi: 10.4300/JGME-D-22-00722.1. PMID: 36591428; PMCID: PMC9765912. 7. Artino AR Jr, Gehlbach H, Durning SJ. AM last page: Avoiding five common pitfalls of survey design. Acad Med. 2011 Oct;86(10):1327. doi: 10.1097/ACM.0b013e31822f77cc. PMID: 21955721. 8. Artino AR Jr, Gehlbach H. AM last page. Avoiding four visual-design pitfalls in survey development. Acad Med. 2012 Oct;87(10):1452. doi: 10.1097/ACM.0b013e31826a06b2. PMID: 23010979.
Using answer choices that fail to include the full spectrum of potential answers
