Mobile Web Surveys: a First Look at Measurement, Nonresponse, and Coverage Errors.

This dissertation focuses on the use of smartphones for Web surveys. The current state of knowledge about whether respondents are willing and able to accurately record their answers when using such devices is evolving, but far from complete. The primary purpose of my research is therefore to investigate the implications of this new mode for various sources of error using a Total Survey Error (TSE) perspective. Each chapter reports on a different aspect of a mode experiment that I designed to compare the effect of completion device (smartphone vs. computer) on survey errors. The experiment was carried out using the LISS panel (Longitudinal Internet Studies for the Social Sciences), a probability-based Web panel administered by CentERdata at Tilburg University in the Netherlands. The first analysis (Chapter 2) compares response quality in the two modes. When using smartphones, respondents in this study really were more mobile and more engaged with the other people and other tasks compared to when using computers. Despite this, response quality – conscientious responding and disclosure of sensitive information – was equivalent between the two modes of data collection. The second analysis (Chapter 3) investigates the causes of nonresponse in the mobile Web version of the experiment. I found that several social, psychological, attitudinal, and behavioral measures are associated with nonresponse. These include factors known to influence participation decisions in other survey modes such as personality traits, civic engagement, and attitudes about surveys as well as factors that may be specific to this mode, including smartphone use, social media use, and smartphone e-mail use. The third analysis (Chapter 4) estimates multiple sources of error simultaneously in the mobile Web version of the experiment. Errors are estimated as a mode effect against the conventional Web survey, which serves as the benchmark. I find few overall mode effects and no evidence whatsoever of measurement effects, but a significant impact of non-coverage bias for over one-third of the estimates. Collectively, these findings suggest that non-observation errors (i.e., coverage and nonresponse), not measurement errors, are the largest obstacle to the adoption of mobile Web surveys for population-based inference.PhDSurvey MethodologyUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/116722/1/antoun_1.pd

A comparison of question order effects on item-by-item and grid formats: visual layout matters

Question order effect refers to the phenomenon that previous questions may affect the cognitive response process and respondents' answers. Previous questions generate a context or frame in which questions are interpreted. At the same time, in online surveys, the visual design may also shift responses. Past empirical research has yielded considerable evidence supporting the impact of question order on measurement, but few studies have investigated how question order effects vary with the visual design. Our main research question was whether question order effects are different on item-by-item formats compared to grid formats. The study uses data from an online survey experiment conducted on a non-probability-based online panel in Hungary, in 2019. We used the welfare-related questions of the 8'th wave of ESS. We manipulated the questionnaire by changing the position of a question that calls forth negative stereotypes about such social benefits and services. We further varied the visual design by presenting the questions in separate pages (item-by-item) or one grid. The results show that placing the priming questions right before the target item significantly changed respondents' attitudes in a negative way, but the effect was significant only when questions were presented on separate pages. A possible reason behind this finding may be that respondents engage in a deeper cognition when questions are presented separately. On the other hand, the grid format was robust against question order, in addition, we found little evidence of stronger satisficing on grids. The findings highlight that mixing item-by-item and grids formats in online surveys may introduce measurement inequivalence, especially when question order effects are expected

Agree or disagree: Does it matter which comes first? An examination of scale direction effects in a multi-device online survey

Previous research shows that the direction of rating scales can influence participants’ response behavior. Studies also suggest that the device used to complete online surveys might affect the susceptibility to these effects due to the different question layouts (e.g., horizontal grids vs. vertical individual questions). This article contributes to previous research by examining scale direction effects in an online multi-device survey conducted with panelists in Spain. In this experiment, respondents were randomly assigned to two groups where the scale direction was manipulated (incremental vs. decre mental). Respondents completed the questionnaire using the device of their choosing (57.8% used PCs; 36.5% used smartphones; and 5.7% used tablets). The results show that scale direction influenced response distributions but did not significantly affect data quality. In addition, our findings indicate that scale direction effects were comparable across devices. Findings are dis cussed and implications are highlighted. Survey research has proved that many design features of scales affect how respondents process and use them to construct their responses (DeCastellarnau 2018; Menold and Bogner 2016; Yan and Keusch 2015). For instance, it is well established that the presentation order of categorical response options influences survey responses. This type of response bias, known as response-order effects, distinguishes between primacy and recency effects (Chang and Krosnick 2009; Galesic et al.2008; Stern et al. 2007). Primacy effects refer to higher endorsements of response categories presented early in the list, while recency effects refer to higher endorsements of response categories presented later in the list (Schwarz and Hippler 2004). Although previous research has widely documented the impact on sur vey responses produced by varying the order of categorical response options, fewer studies have examined response-order effects with ordinal scales, despite their extensive use in survey research. The first studies that analyzed these effects date back to the 1960s, when Belson (1966) found that survey responses tended to shift toward the starting point of the rating scales, regardless of scale length and respondent characteristics. More recent studies, however, have turned up mixed evidence. Some have reported a similar tendency for responses to be biased toward the starting point of scales (Garbarski et al. 2018; Ho¨hne and Krebs 2017; Israel 2006; Toepoel et al. 2009; Yan and Keusch 2018), while others have found no effect of the direction of rating scales on survey responses (Krebs and Hoffmeyer-Zlotnik 2010; Rammstedt and Krebs 2007; Weng and Cheng 2000). In addition, some studies have reported scale direction effects on a limited number of questions and no effect on other questions within the same experiment (Christian et al. 2009; Elway 2013; Ho¨hne and Krebs 2017

Analyzing Survey Characteristics, Participation, and Evaluation Across 186 Surveys in an Online Opt-In Panel in Spain

"Survey designers often ask about the best length for their questionnaires and the best format for their questions. Much research has already addressed these issues. However, the answers to these questions may vary with the population of interest, the mode of data collection used, and other factors. The goal of this paper is twofold: 1. To give an overview of the present situation in opt-in online panels, in terms of survey characteristics, participation, and evaluation, by reviewing 186 surveys managed by the panel company Netquest in Spain in 2016. This will be useful to determine areas where further research needs to focus. 2. To study how key characteristics of questionnaires impact survey evaluation and levels of survey break-off. This will allow us to highlight the characteristics that best reduce break-off and improve respondents’ survey evaluation. Based on these results, we will propose practical recommendations for future survey design within the framework of opt-in online panels." (author's abstract

How Much is a Box? The Hidden Cost of Adding an Open-ended Probe to an Online Survey

Probing questions, essentially open-ended comment boxes that are attached to a traditional closed-ended question, are increasingly used in online surveys. They give respondents an opportunity to share information that goes beyond what can be captured through standardized response categories. However, even when probes are non-mandatory, they can add to perceived response burden and incur a cost in the form of lower respondent cooperation. This paper seeks to measure this cost and reports on a survey experiment that was integrated into a short questionnaire on a German salary comparison site (N = 22,306). Respondents were randomly assigned to one of three conditions: a control without a probing question; a probe that was embedded directly into the closed-ended question; and a probe displayed on a subsequent page. For every meaningful comment gathered, the embedded design resulted in 0.1 break-offs and roughly 3.7 item missings for the closed-ended question. The paging design led to 0.2 additional break-offs for every open-ended answer it collected. Against expectations, smartphone users were more likely to provide meaningful (albeit shorter) open-ended answers than those using a PC or laptop. However, smartphone use also amplified the adverse effects of the probe on break-offs and item non-response to the closed-ended question. Despite documenting their hidden cost, this paper argues that the value of the additional information gathered by probes can make them worthwhile. In conclusion, it endorses the selective use of probes as a tool to better understand survey respondents

Smileys, Stars, Hearts, Buttons, Tiles or Grids: Influence of Response Format on Substantive Response, Questionnaire Experience and Response Time

Studies of the processes underlying question answering in surveys suggest that the choice of (layout for) response categories can have a significant effect on respondent answers. In recent years, the use of pictures, such as emojis or stars, is often used in online communication. It is unclear if pictorial answer categories can replace traditional verbal formats as measurement instruments in surveys. In this article we investigate different versions of a Likert-scale to see if they generate similar results and user experiences. Data comes from the non-probability based Flitspanel in the Netherlands. The hearts and stars designs received lower average scores compared to the other formats. Smileys produced average answer scores in line with traditional radio buttons. Respondents evaluated the smiley design most positively. Grid designs were evaluated more negatively. People wanting to compare survey outcomes should be awar

Towards a Reconsideration of the Use of Agree-Disagree Questions in Measuring Subjective Evaluations

Agree-disagree (AD) or Likert questions (e.g., “I am extremely satisfied: strongly agree … strongly disagree”) are among the most frequently used response formats to measure attitudes and opinions in the social and medical sciences. This review and research synthesis focuses on the measurement properties and potential limitations of AD questions. The research leads us to advocate for an alternative questioning strategy in which items are written to directly ask about their underlying response dimensions using response categories tailored to match the response dimension, which we refer to as item-specific (IS) (e.g., “How satisfied are you: not at all … extremely”). In this review we: 1) synthesize past research comparing data quality for AD and IS questions; 2) present conceptual models of and review research supporting respondents’ cognitive processing of AD and IS questions; and 3) provide an overview of question characteristics that frequently differ between AD and IS questions and may affect respondents’ cognitive processing and data quality. Although experimental studies directly comparing AD and IS questions yield some mixed results, more studies find IS questions are associated with desirable data quality outcomes (e.g., validity and reliability) and AD questions are associated with undesirable outcomes (e.g., acquiescence, response effects, etc.). Based on available research, models of cognitive processing, and a review of question characteristics, we recommended IS questions over AD questions for most purposes. For researchers considering the use of previously administered AD questions and instruments, issues surrounding the challenges of translating questions from AD to IS response formats are discussed

Comparing data quality from personal computers and mobile devices in an online survey among professionals

It is increasingly common for respondents to complete web surveys using mobile devices (smartphones and tablets) rather than personal computers/laptops (PCs). Evidence of the impact of the use of mobile devices on response and data quality shows mixed results and is only available for general population surveys. We looked at response quality for a work-related survey in the UK among general practitioners (GPs). GPs were sent email invitations to complete a web survey and half (55%) completed it on a mobile device. While GPs using a mobile device were less likely to complete the full questionnaire than those using a PC, we found no differences in data quality between mobile and PC users, except for PC users being more likely to respond to open-ended questions

Do respondents using smartphones produce lower quality data? Evidence from the UK Understanding Society mixed-device survey

We live in a digital age with high level of use of technologies. Surveys have started adopting technologies including smartphones for data collection. There is a move towards online data collection in the UK, including an ambition to collect 75% of household responses online in the UK 2021 Census. Major social household surveys in the UK have either transitioned to online data collection or are in the process of preparation for the transitioning. The Covid-19 pandemic forced rapid transitions to online data collection for many social surveys globally, with this mode of data collection being the only possibility at the moment. There are still concerns regarding allowing respondents to use smartphones to respond to surveys and not much is known about data quality produced by respondents using smartphones for survey completion in the UK context. This paper uses the first available in the UK, large scale mixed-device survey, Understanding Society Wave 8 where 40% of the sample were assigned to online mode of data collection. It allows comparison of data quality between different devices within the online mode of data collection with a special focus on smartphones. This analysis is very timely and fills the gap in knowledge. Descriptive analysis and then various regressions are used depending on the outcome variables to study data quality indicators associated with different devices in the online part of the survey. The following data quality indicators are assessed: break-off rates, item nonresponse, response style indicators, completion times, differential reporting indicators including self-reporting of risky behaviours, and consent to data linkage. Comparisons to limited results available in the UK are drawn. The results suggest that even in the context of non-optimised for smartphone questionnaire, we should not be concerned about respondents using smartphones for future social surveys, even for longer surveys such as the Understanding Society, as break off rates are very low and data quality between devices is not very different