Plutowski, Luke

Zechmeister, Elizabeth

English

Scholarship@Western

Western University Scholarship@Western Workshop: Survey Design Under Constraints Conference Proceedings 2022 Do Question Topic and Placement Shape Breakoff Rates Luke Plutowski Vanderbilt University Elizabeth Zechmeister Vanderbilt University Follow this and additional works at: https://ir.lib.uwo.ca/c-dem_conf_survey  Part of the Political Science Commons Citation of this paper: Plutowski, Luke and Zechmeister, Elizabeth, "Do Question Topic and Placement Shape Breakoff Rates" (2022). Workshop: Survey Design Under Constraints. 2. https://ir.lib.uwo.ca/c-dem_conf_survey/2 Do Question Topic and Placement Shape Survey Breakoff Rates?Luke Plutowski & Liz Zechmeister (LAPOP Lab at Vanderbilt University)Presentation for C-Dem Workshop @ MapleMeth ConferenceFriday, May 27, 2022Summary● Hypothesis. Beginning a survey with questions about highly salient issue reduces breakoffs  ● Data. Original experiment in Haiti (April-June 2020) randomizes placement of module about COVID-19● Results. Those who received COVID questions at top of survey were marginally less likely to breakoff (especially among those more concerned about COVID)2Background & Theory3Motivation● At the beginning of lockdown, respondents seem more interested in surveys about COVID-19● Common advice for surveys: begin with most interesting questions4Recommendations from Pew Research (top) and Qualtrics (bottom) regarding interesting questionsLiterature● Breakoffs can inhibit data quality and data collection efficiency (Keeter et al. 2016; Peytchev 2009; Roßmann, Blumenstiel, and Steinbrecher 2015) ● Questionnaire features can influence cooperation rates (Galesic 2006; McGonagle 2013; Roßmann, Blumenstiel, and Steinbrecher 2015)● Response rates are influenced by stated survey topic (Groves, Presser, and Dipko2004; Groves, Singer, and Corning 2000; Holland and Christian 2009; Keusch 2013; Krosnick and Presser 2010; Martin 1994; Van Kenhove, Wijnen and De Wulf 2002)5Do question topic and order influence breakoff rates?6Theory● “Topic-induced motivation” – more important or relevant question content ought to increase engagement and, therefore, decrease breakoff● Three non-rival explanations:○ Temporary bump in respondent’s “bank” of interest○ Convinces respondent that research is worthwhile or contributing to an important cause○ Establishes rapport by acknowledging a sensitive or timely issue7Research Design8Research Design (I)● Goal. Leverage the onset of the pandemic to assess whether the questions about the pandemic creates topic-induced motivation to engage in a survey● Questionnaire Design. Questions about pandemic within a broader survey about other topics● Experiment. Random assignment into 2 conditions: receive COVID module at beginning or end of survey● COVID module. 10 questions ○ About 35 questions in between treatments9Research Design (II)● Outcomes.○ Break off rate: breakoffs as a percentage of breakoffs + completes ○ Time-to-breakoff: number of questions asked before breakoff○ Item non-response: percentage of answers with “don’t know”/ “no response” ● Hypotheses. Respondents who receive COVID questions first:○ H1: Are less likely to breakoff○ H2: Answer more questions before breaking off○ H3: Give fewer non-responses○ H1a: If theory is true, effects should be stronger among those who are more concerned about COVID 10Data (I)● Original data collection effort in Haiti○ April 23-June 10, 2020○ Computer-Assisted Telephone Interviewing (CATI)○ Nationally representative sample of voting age cellphone users using list of activated cell numbers● Sample size○ 3,327 connected calls ○ 1,037 excluded due to ineligibility, rejection, pre-treatment breakoff, or quality control○ 2,290 remain and are assigned to treatment■ 61 early terminations due to inadvertent reasons (e.g. dropped call)● Excluded from breakoff analysis, included in non-response■ 2,018 completes■ 211 breakoffs (9.5% breakoff rate)11Data (II)● 10 question COVID module○ COVID-First treatment: asked immediately after consent question○ COVID-Late treatment: asked at end of “core”, before ending demographic bloc● COVID concern: “How serious of How serious of a problem do you think the coronavirus outbreak is for Haiti?” ○ (1) Very serious (2) Somewhat serious (3) Not so serious (4) Not serious at all  (5) Have not thought much about this● About 35 questions between two treatments12Analysis Strategy● Tests. Difference of proportions z-test (H1) and difference of means t-test (H2, H3) ○ Standard errors clustered by interviewer○ One-tailed p-values● “Limited Sample”. 439 observations indicated COVID outbreak was “not so serious”, “not at all serious”, or “have not thought much about it”○ Excluded in some analysis – treatment should have no effect○ 1,407 said “very” or “somewhat” serious; 274 non-responses or N/A13Results14H1 and H1a – Breakoff Rate1534On a 2000-person survey, by placing COVID questions first, 34 fewer respondents need to be interviewed in order to reach the target sample size (or 62, if everyone thought the topic was a serious issue) 16H2 – Questions Answered Before Breakoff17H3 – Item Non-Response Rate18Extension 1: Ecuador Experiment19Analysis Strategy● Replicated experiment in similar study in Ecuador (Dec 2020-Jan 2021)● Sample characteristics○ Random digit dial of cell-phone numbers ○ Target sample size of 800 voting-age citizens of Ecuador● Questionnaire characteristics○ Experiment embedded within broader study of democratic attitudes○ Same structure as Haiti questionnaire, though COVID module and other questions vary slightly ● Respondents much more highly engaged in Ecuador20Results21Extension 2: Four Country Correlation Analysis22Analysis Strategy● Theory. Observable implication of topic-induced motivation theory: among those who begin survey with questions about COVID, those who think it is serious issue should be less likely to break off than those who do not ● Data. Similar studies conducted in Mexico (July-Aug 2020) and Peru (July 2020), which featured COVID questions at the beginning (not randomized)● Test. Logistic regression of breakoff (0/1) on 5-point concern about COVID measure23Results24Conclusion25Recap & Implications● Findings. Beginning questionnaire in Haiti & Ecuador with items about COVID-19, a salient issue, modestly decreased breakoff rate, especially among group who is more concerned about COVID○ Marginal significance, and no treatment effect on time-to-breakoff or NR rate● Literature. Question topic influences completion rates as survey topic influences response rates, though perhaps not to same extent● Practice. All else equal, better to begin with questions on more salient issue, though may not be worthwhile if it adds to length of questionnaire or interrupts survey flow26Future Research● Mechanisms. Is treatment effect driven by interest in the question, belief that the survey is worthwhile, interviewer rapport, or something else? ● Other topics. Does the same pattern hold for other types of issues?  ● Interviewer effects. Breakoffs varied greatly by interviewer.  Why?27Luke.Plutowski@Vanderbilt.eduLiz.Zechmeister@Vanderbilt.edu28

Do Question Topic and Placement Shape Breakoff Rates

https://ir.lib.uwo.ca/cgi/viewcontent.cgi?article=1004&context=c-dem_conf_survey

Do Question Topic and Placement Shape Breakoff Rates

Abstract

Similar works

Full text

Available Versions

Scholarship@Western