Effective Survey Design: Best Practices and Policies at Hope College

Survey Design: Best Practices and Policies March 5, 2025 Don Friesner and Daryl Van Tongeren

Contact Information Donald Friesner Director of Institutional Research Friesner@hope.edu Daryl Van Tongeren Director of the Frost Center for Social Science Research vantongeren@hope.edu

Hope Colleges Survey Policies

Survey Fatigue Due to the demand for information and the increased use of online surveys, survey responses are declining. National Survey of Student Engagement (NSSE) Response Rates (national) 2021 29% 2022 28% 2023 26% 2024 23%

Survey Fatigue and Hope Survey Calendar First-Year Students (Fall Semester) New Student Survey Student Congress Survey Admissions Enrolled Student Survey (ESS) ETS Intercultural Competency & Diversity ETS Critical Thinking Assessment Some smaller populations get additional surveys

Surveys that Frost Center does not coordinate on: Course evaluations Administrative record collection Workshop, event or customer service evaluations or feedback Elections such as faculty elections, homecoming court, etc. Meeting scheduling polls such as Wizard, Doodle, etc. Office, committee, student organization or faculty department internal surveys Student developed surveys using a convenience sample and used as part of a class Academic assessment and departmental review activities

Hope College Survey and Research Policies Goal Improve the volume, flow, frequency, quality and validity of this important process. The goal is to improve the coordination of inquiries, which will benefit the entire Hope College community while fostering research and protecting participants.

Survey Policies College-wide surveys of constituency groups (students, staff, faculty, alumni) should be planned well in advance. Research involving human subjects requires HSRB approval and requests also need to be reviewed by the Assessment Committee in conjunction with our current large-scale survey schedule. Qualtrics is the campus-side survey platform.

Survey Policies Please consider other alternatives as appropriate (i.e., Focus Groups, Sample populations) Please think about questions carefully (Needed data to answer questions vs. curiosity) Please remove data from Qualtrics once project is complete.

Best Practices in Survey Research

Best Practices in Survey Research 1. Research Design Research Design: What question(s) are you trying to answer? Consider your research question to guide your decision on whether a survey is appropriate When is a survey the best? Broad sample, not overly intensive Describe a population Collect standardized data Representative sample Experimental manipulations are unethical (e.g., smoking research)

Best Practices in Survey Research 2. Sampling and Recruitment: Sampling and Recruitment: Who do you wish to sample? Who will receive your survey? How will you advertise your survey? How will you direct people to your survey?

Best Practices in Survey Research 3. Methodology: Methodology: What are useful pragmatics? How long do you want your survey to be (keep it short; < 15 minutes) How long do you want to keep your survey open? When / how do you want to send reminders? Only collect data you know you will need and plan to do something with Offer incentives Offer to share the findings

Best Practices in Survey Research 4. Writing questions: Writing questions: Developing efficient and effective items Single-barreled vs. double-barreled items Clear, direct, short Fewer is better Quantify it on a scale (rather than dichotomous or qualitative) Avoid: Avoid: leading, ambiguous, negative, biased items, or slang

Best Practices in Survey Research 5. Survey Flow: Survey Flow: Using Qualtrics to offer contingency questions Based on previous responses, future items can be displayed (display logic) What is your sex? Male Female Have you ever been pregnant? No Yes

Best Practices in Survey Research 6. Survey Design: Survey Design: Using Qualtrics to increase efficiency Use matrix questions to present items in a table form

Best Practices in Survey Research 7. Survey Organization: Survey Organization: Broad and unbiased Early questions should be broad and should not influence later questions More sensitive questions should be later on in the survey Attention wanes and fatigue increases as the survey goes on Ask for demographics last

Best Practices in Survey Research 8. Key components: Key components: Consent, instructions, debriefing Remember to start with the informed consent and instructions Don t forget to include the debriefing at the end How you code your items in Qualtrics will translate to how the data are downloaded (e.g., ordering of items) Preview and test, test, test before sending out!

Best Practices in Survey Research 9. Data Integrity: Data Integrity: Ensuring high quality data Avoiding bots / AI responses Attention and quality-control checks Commitment statements Don t ask for data you already have (or can get access to)

Best Practices in Survey Research 10. Data Protection: Data Protection: Ensuring confidentiality and security Separate emails or any identifiers from data (de-identify data) Keep data secure behind password-protection

Best Practices in Survey Research What specific questions do you have?