Non-Response Bias in NYC Health Panel Surveys

Sometimes (or Late) is Not the Same as Never: Health Survey Panelist Response Emily Wichmann University of North Carolina, Chapel Hill Joint work with Tashema Bholanath and Stephen Immerwahr (New York City Department of Health and Mental Hygiene) Federal Computer Assisted Survey Information Collection Workshops April 17, 2024

PRESENTATION OUTLINE 1 Background 2 Methods 3 Results 4 Summary and Conclusions 5 Next Steps and Recommendations

BACKGROUND NYC Health Panel surveys have provided critical citywide information on important health issues. Enrolled panelists, recruited from multiple probability-based frames (RDD and ABS), receive up to 10 survey invitations per year and some did not respond to any invitations. Survey weights can partially correct for known demographic differences in response propensity, hopefully reducing potential bias due to differential nonresponse. Valuable to know the potential direction of nonresponse bias. Panelists response lag time can inform whether fielding period duration could introduce bias. Late responders could also be treated as proxies for nonresponders. 3

BACKGROUND 29 studies and 11 reports were used to evaluate best practices. Non-respondents have a default consideration as homogenous, which studies of response groups have disproven. Early and late categorization relies on when survey participants were invited, reminders administered, or when one responded. Studies have considered NRFU and refusers as late rather than nonrespondents.

RESEARCH OBJECTIVE To assess the potential bias in NYC Health Panel survey estimates (and in health estimates, in particular) stemming from non-response.

RESEARCH QUESTIONS What are the demographic and health characteristics of enrolled survey panelists that are associated with subsequent response or nonresponse to survey invitations? 01 Independent of demographics, what are the health characteristics of respondents in the first half of a field period ( early ) compared to those in the second half ( late )? 02

METHODS Analysis 24 Surveys Variables Demographics (e.g., age, race/ethnicity, education, household size, survey language, health insurance status, etc.) Chi-square Tests Multivariable Logistic Regression Agency Omnibus Surveys (Health Opinion Polls) COVID Social Determinants of Health Energy Insecurity Food Access Sexual Health Others Health-Related Quality of Life core: general health status, past 30 days poor physical health, past 30 days poor mental health, past 30 days impacted activity

EXCLUSION AND INCLUSION CRITERIA Exclusion Criteria: Respondents determined by invite and response variables Missing, Improbable, and Outliers Early: responded before halfway point of fielding period Refusals/never responded considered non- respondent Withdrawals Late: responded after halfway point of fielding period Response Propensity: often (more than 50% responses), sometimes (50% or fewer responses), and never (non-respondent) to survey invites.

PANELISTS RESPONSE PROPENSITY (N=12,423) NON-RESPONDENTS RESPONDENTS Number of Panelists 6702 (53.9%) 3124 (25.1%) 2597 (20.9%) Never Sometimes Often Response Propensity 27

RESPONSE PROPENSITY 1 RESULTS: Non-responding vs Responding Panelists Significantly* associated with non-response (multivariable models) Gender, age, race/ethnicity, household poverty, education, survey language, household language, and contact mode. Demographics Health None of the 4 health measures. * *p < .05

RESPONSE PROPENSITY 2 RESULTS: Never, Sometimes, Often Panelists Chi-square tests indicate significant* difference between never, sometimes, often respondents across multiple individual demographics (except for household size) and all 4 health measures. As expected, in multivariable ordinal logistic regression models there were consistently significant demographic associations with response propensity: gender, race/ethnicity, household poverty, education, language, and contact mode. However, of the 4 health measures, general health and impacted activity days were significantly associated with response propensity independent of demographics, i.e., those reporting better health and fewer impacted activity days were more likely to respond to subsequent invitations Past 30 days of poor physical health and days of poor mental health were not associated with response independent of demographics. * *p < .05

RESPONSE LAG TIME RESULTS: Early vs. Late Responses to 2- and 4-week Surveys Omnibus Surveys (n=11,280 Responses) Focused Panel Surveys (n=27,015 Responses) Late, 9093, 34% Late, 3875, 34% Early, 7405, 66% Early, 17922, 66% Omnibus Surveys (Early: 1 week; Late: >1 week ) Late significantly* associated with multiple demographics and independently worse general health, more days of poor mental health, and more impacted activity days. Focused Panel Survey (Early: 2 week; Late: >2 week ) Late significantly* associated with multiple demographics, and worse general health, more days of poor physical health, more days of poor mental health, and more impacted activity days. * *p < .05

STUDY LIMITATIONS Response timing, environmental Primarily English- language panelists 01 03 events Maximum field period was four weeks Multivariable models not strong 02 04

SUMMARY AND CONCLUSIONS At empanelment, never respondents reported worse general health and had more physically unhealthy days Independent of demographics, good general health at empanelment was positively associated with subsequent response Weighting on demographics might not be sufficient to address a healthy response propensity bias in panel surveys Even in 4-week surveys, late respondents had significantly worse health vs. early respondents Longer field periods may be needed for unbiased health estimates

POTENTIAL NEXT STEPS & RECOMMENDATIONS POTENTIAL NEXT STEPS RECOMMENDATION #1 Recommendation 1 Panel surveys that ask about health should consider asking general health status at empanelment/registration. Add health measures (or just general health status) to existing nonresponse adjustments. #2 Overall Observation Potential healthier response bias overall and over field periods. Use inverse probability weighting with propensity models including health measures. #3 Compare resulting estimates.

THANK YOU! Questions? Emily Wichmann University of North Carolina, Chapel Hill etwich@unc.edu