Lugtig, Peter

Toepoel, Vera

Name not available

Online surveys are mixed-device surveys: issues associated with the use of different (mobile) devices in web surveys

Toepoel, V.

Lugtig, P.J.

Leerstoel Heijden

Methodology and statistics for the behavioural and social sciences

NARCIS 

Online surveys are mixed-device surveys. Issues associated with the use of different (mobile) devices in web surveys

Utrecht University Repository

SSOAR - Social Science Open Access Repository 

DOI: 10.12758/mda.2015.009methods, data, analyses | Vol. 9(2), 2015, pp. 155-162Online Surveys are Mixed-Device Surveys. Issues Associated with the Use of Different (Mobile) Devices in Web SurveysVera Toepoel & Peter LugtigUtrecht University1 Issues in Mixed-Device SurveysSurvey research is changing in a more rapid pace than ever before, and the con-tinuous and exponential growth in technological developments is not likely to slow down. Online surveys are now being completed on a range of different devices: PC, laptops, tablets, mobile phones or hybrids between these devices. Each device var-ies in screen sizes, modes of operationalization and technological possibilities. We define online surveys that are in practice being completed on different devices as mixed-device surveys. This special issue discusses issues in the design and imple-mentation of mixed-device surveys, with the aim to bring survey research to the next level: in our view all web surveys should from now be thought of as mixed-device surveys.Theory and best practices for mixed-device surveys are still in its infancy. The current state of knowledge about the dynamics of taking surveys on mobile devices is not as advanced as necessary in times of rapid change. While current technology opens great possibilities to collect data via text, apps, and visuals, there is little sci-entific research published about the actual uses and best practices of these applica-tions to increase data quality. Researchers and survey methodologists in particular need to find ways to keep up with fast changing technologies. © The Author(s) 2015. This is an Open Access article distributed under the terms of the Creative Commons Attribution 3.0 License. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.methods, data, analyses | Vol. 9(2), 2015, pp. 155-162 156 Direct correspondence to  Vera Toepoel, Utrecht University E-mail: v.toepoel@uu.nl1.1 Mobile Penetration Rates and Mobile Survey CompletionThe penetration rate of mobile phones with Internet connection has increased dra-matically in the last couple of years. Europe tops the global market on smartphone penetration. In the Netherlands, for example, there has been an increase from around 36% of the population owning a mobile phone with Internet access in 2010 to 72% in 2013 (SN, 2013). In the United States, figures increased from 35% in 2011 to 56% in 2013 (PEW, 2013). Although the majority of the population owns a smart-phone, only a small part of the population is actually using it for survey completion. This is probably related to the fact that online surveys are often not yet adapted to be completed on small devices. However, if the questionnaire is dynamically pro-grammed and suitable for completion on small devices, more people are inclined to use a mobile device for survey completion. We found for example that 57% of panel members with a mobile phone used it when being prompted in a dynamically programmed survey (Toepoel & Lugtig, 2014).1.2 Mixed Device Surveys – a Research Agenda Representation The main drawback of online surveys has always been the lack of a sampling frame of email addresses for the general population. Mobile devices, and especially mobile phones, may in the future be used to overcome this problem, because they offer so many channels of communication.For example, mobile surveys can draw on the advantages of probability-based sampling via Random Digit Dialling (RDD). Second, mobile surveys can easily switch between self-administered and interviewer-administered questions and approach respondents using multiple methods (apps, sms, e-mails and calls). This can be especially useful in the context of a panel study. When respondents are inter-viewed multiple times, respondents can be approached in multiple ways. On top of this, the mode of survey administration can also be switched within measurements. We know little about what works in practice, and formal studies that document the combined effects on coverage and nonresponse error of different sampling methods for mobile devices are to our knowledge non-existent.157 Toepoel, Lugtig: Online Surveys are Mixed-Device SurveysMeasurementEarlier studies have shown that some survey questions are better asked in a par-ticular survey mode. Data quality is generally higher in self-administration modes (Saris & Gallhofer, 2007; Campanelli et al., 2013), especially when the topic of interest is in some way sensitive (Kreuter et al.). On the other hand, an interviewer may lead to better data when questions are complicated; for example when working out a respondent’s life history.Mobile surveys can draw on technological innovations that come with big data. Sensor data such as GPS, accelerometers, or biomarkers are available on almost all mobile phones and tablets. They offer new and better ways to collect data on specific questions, and can be used to investigate how context affects data quality (see Link et al., 2014). In addition, sensor data can alleviate the burden of survey completion for respondents in time-consuming time budget, health and travel studies. Although mobile devices offer new possibilities, they are not without their pit-falls. The screen size is smaller than on traditional computers, there is a variability in how questions are displayed (depending on the type of device, personal prefer-ences and browsers) and entering data works differently. In addition, people use mobile devices differently from traditional computers. People are used to using mobile phones for short messaging, not for taking long surveys. This means that questionnaires should probably be shortened, or split into multiple short questions. In the future, surveys on mobile phones may consist of only a few questions at a time, asked in several bursts.The fact that available studies often show mixed findings on for example response timings, break-off rates, and survey evaluation in mixed-device studies , can be (partly) an outcome of the rapid changes in technology over time in addition to increased societal learning and growing comfort with devices and their many features (AAPOR Taskforce Report on Mobile Technologies, 2014). The fact that respondents complete online surveys on traditional desktop PCs as well as new mobile devices makes designing surveys a challenge. Issues associated with mixed-mode surveys – for example whether the questionnaire should be optimized for each mode versus a generalized design- can be extended to a mixed-device context. 1.3 Moving from Online Surveys to Mixed-Device SurveysIn order to adapt our surveys to new technologies, we need to redesign our surveys. For example, we have to rethink the use of some question formats. It took time before survey methodologists understood how to redesign paper-and-pencil sur-veys to online surveys. Now, we have to redesign online surveys to become mixed-device surveys.methods, data, analyses | Vol. 9(2), 2015, pp. 155-162 158 Long matrix questions are not suitable for small devices. For example, slider bars, and especially Visual Analogue Scales that work on a point-and-click-princi-ple, save space on the screen (Toepoel, 2016). In addition to question formats, ques-tionnaire length is important to take into account when designing a multi-device survey. Research shows mixed results when it comes to measurement differences between devices (e.g., Bosnjak et al., 2013; de Bruijne & Wijnant, 2013; Buskirk, 2015; Busse & Fuchs, 2012; Lynn & Kaminska, 2013; Peytchev & Hill, 2010; Lugtig & Toepoel, 2015; Vehovar, Berzelak & Lozar-Manfreda, 2010). If questions are dynamically programmed and designed for mixed-devices, measurement dif-ferences seem to be minimal.Mobile phones are rapidly replacing key tasks formerly done on PC and lap-tops. It seems a matter of time that mobile phones or mobile devices in general are preferred for survey completion over regular desktop PCs. For example, Toepoel (2016) shows that respondents evaluate the completion of surveys on mobile phones better when they have more experience in mobile phone survey completion. 2 Papers in this Special IssueThe papers in this special issue on mixed-device surveys all study the issues men-tioned in the previous section, and provide a start for understanding how to design mixed-device surveys. They offer a unique view on questionnaire design in an era where researchers will not know in advance what device a respondent is going to use to complete a survey, let alone how the questionnaire looks on the respondent’s device. We can, however, try to predict respondent behavior, in addition to design-ing our online questionnaires with care. The first paper in this special issue by Axinn, Gatny, and Wager is titled “max-imizing data quality using mode switching in mixed-device survey design”. Since the advantages of the web mode for studies with frequent re-interviews can be offset by the serious disadvantage of low response rates and the potential for nonresponse bias, the authors examine the potential for a mixed-device approach with active mode switching to reduce attrition bias. The Relationship Dynamics and Social Life (RDSL) study design allows panel members to switch modes by integrating telephone interviewing into a longitudinal web survey with the objective of col-lecting weekly reports. The authors found that allowing panel members to switch modes kept more participants in the study compared to a web only approach. In addition, they found that the characteristics of persons who ever switched modes were different from those who did not. Mode options and mode switching can therefore be important for the success of longitudinal web surveys to maximize participation and minimize attrition.159 Toepoel, Lugtig: Online Surveys are Mixed-Device SurveysIn the second paper, Arn, Klug, and Kolodziejski look at the challenge of optimizing survey layout in online research to enable multi-device use. This paper presents results of the implementation of a new adapted design at the panel of DemoSCOPE that allows the participants to take part in a survey on multiple (espe-cially mobile) devices. To evaluate this adapted design, the authors compare inter-view data and question timings of panellists who participated before and after the design transition. The key outcomes in this study are the completion rate, item non-response, open questions, straightlining, timing of single question and the length of the total interview are presented. In addition, the authors have presented examples of both old and new designs to the panel community and invited them to assess these examples concerning orientation, colour, design and usability. The authors evaluate the differences in these assessments before and after the design transition for smartphone and desktop users. They end with suggestions for best practices for online studies on different devices.Andreadis shows in the third contribution to this special issue that computer users and smartphone users give responses of almost the same quality. Combining a design of one question in each page and innovative page navigation methods, we can get high quality data by both computer and smartphone users. The two groups of users are also compared with regard to their precisely measured item response times. The analysis shows that using a smartphone instead of a computer increases the geometric mean of item response times by about 20%. The data analyzed in this paper were collected by a smartphone-friendly web survey. As a result, there are no significant interactions between smartphone use and either the length of the question or the age of the respondent. Thus, the longer response times among smartphone users should be attributed to other causes, such as the likelihood of smartphone users being distracted by their environment. Buskirk, Saunders, and Michaud note that survey researchers are still trying to understand which online design principles directly translate into presentation on mobile devices and which principles have to be modified to incorporate sepa-rate methods for these devices. One such area involves the use of input styles such as sliding scales that lend themselves to more touch centric input devices such as smartphones or tablets. Operationalizing these types of scales begs the question of an optimal starting position and whether these touch centric input styles are equally preferred by respondents using less touch capable devices. While an outside start-ing position seems optimal for slider questions completed via a desktop computer, this solution may not be optimal for completion via mobile devices. The experiment presented in the paper by Buskirk, Saunders and Michaud moves the mixed device survey literature forward by directly comparing outcomes from respondents who completed a collection of survey scales using their smartphone, tablet or computer. Within each device, respondents were randomly assigned to complete one of 20 possible versions of scale items determined by a combination of three experimental methods, data, analyses | Vol. 9(2), 2015, pp. 155-162 160 factors including input style, length and number formatting. Results from this study suggest more weaknesses than strengths for using slider scales to collect survey data using mobile devices and also suggest that preference for these touch centric input styles varies across devices and may not be as high as the preference for the more traditional radio button style. Struminskaya, Weyandt, and Bosnjak use the data from six online waves of the GESIS Panel, a probability-based mixed-mode panel representative of the Ger-man population to study whether the responses provided using tablets or smart-phones differ on indicators of measurement and nonresponse errors than responses provided via personal computers or laptops. They extend the scope of past research by exploring whether data quality is a function of device-type or respondent-type characteristics using multilevel intercept-only models. Overall, they find that responding with mobile devices is associated with a higher likelihood of measure-ment discrepancies compared to PC/Laptop survey completion. For smartphone survey completion, the indicators of measurement and nonresponse error tend to be higher than for tablet completion. However, the effects are relatively small and some indicators (such as straightlining) are not related to a device but are attribut-able to a respondent.In all, this special issue on mixed-device surveys in methods, data, analyses offers food for thought on how to design surveys in the modern era. The future will tell us whether the design principles discussed in this issue will hold when new devices arise. Until then, we are happy that we live in exciting times for survey methodology.ReferencesAAPOR Taskforce Report on Mobile Technologies. (2014). Mobile Technologies for Con-ducting, Augmenting and Potentially Replacing surveys: Report of the AAPOR Task Force on Emerging Technologies in Public Opinion Research. Retrieved from: http://www.aapor.org/Mobile_Technologies_Task_Force_Report.htmBuskirk, T. D. (2015). The Rise of Mobile Devices: From Smartphones to Smart Surveys. The Survey Statistician, 72, 25-35. Busse, B., & Fuchs M. (2012). Recruiting Respondents for a Mobile Phone Panel. Metho-dology, 1-10.Bosnjak, M. et al. (2013). Online Survey Participation via Mobile Devices. Retrieved 29th of July 2013 from: http://www.psyconsult.de/bosnjak/pages/publications/conference-contributions.phpCampanelli, P., Nicolaas, G., Jackle, A., Lynn, P., Hope S., Blake M., & Gray M. (2013). A Classification of Question Characteristics Relevant to Measurement Error and Con-sequently Important for Mixed Mode Questionnaire Design. Paper presented at the Presented 11 October 2011 at the Royal Statistical Society, London, UK. Retrieved 161 Toepoel, Lugtig: Online Surveys are Mixed-Device Surveysfrom: http://www.natcenweb.co.uk/genpopweb/documents/other-resources/RSS-Oct-2011-Handout-Recommendations.pdfDe Bruijne, M., & Wijnand, A. (2013). Comparing Survey Results Obtained via Mobile Devices and Computers. An Experiment with a Mobile Web Survey on a Heterogene-ous Group of Mobile Devices Versus a Computer-Assisted Web Survey. Social Science Computer Review, 31, 482-504.Groves, R. M., & Lyberg L. (2010). Total Survey Error. Past, Present, and Future. Public Opinion Quarterly, 74, 849-879.Lynn, P., & Kaminska, O. (2013). The Impact of Mobile Phones on Survey Measurement Error. Public Opinion Quarterly, 77, 586-605.Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity. Public Opinion Quar-terly 72, 847–65Link, M., Duan, S., Bristol K., & Lai, J. (2014). The Generational Technology Divide and Implications for Smartphone Data Collection. Paper presented at the Conference of the American Association of Public Opinion Research, May 15-18, Anaheim.Lugtig, P., & Toepoel, V. (2015). Mixed Devices in a Probability Based Panel Sur-vey. Effects on Survey Measurement Error. Social Science Computer Review. DOI: 10.1177/0894439315574248Peytchev, A., & Hill, G. (2010). Experiments in Mobile Web Survey Design: Similarities to Other Modes and Unique Considerations. Social Science Computer Review. 28, 319-335.PEW Research Center Report. Smartphone Ownership. (2013). Update June 5 2013 by Aaron Smith. Retrieved from: http://www.pewinternet.org/files/old-media//Files/Re-ports/2013/PIP_Smartphone_adoption_2013_PDF.pdfSchouten, B., Cobben, F., van der Laan, J., & Arends, J. (2014). The impact of contact ef-fort and interviewer performance on modespecific nonresponse and measurement bias. CBS Discussion Paper, 2014/05.Statistics Netherlands. (2013). Retrieved from February 3 2014 from: http://statline.cbs.nl/statweb/Toepoel, V. (2016). Buttons of balken, klikken of slepen: wat werkt er nu het beste op mo-biele telefoons, tablets of PCs? In A. E. Bronner, P. Dekker, E. de Leeuw, L. J. Paas, K. de Ruyter, A. Smidts, & J. E. Wierenga (Eds.), Ontwikkelingen in het Marktonderzoek. Jaarboek 2016 Markt Onderzoek Associatie. Haarlem: Spaar en Hout.Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation, and analysis of questionnaires for survey research. Hoboken, NJ: Wiley. Toepoel, V., & Lugtig, P. (2014). What happens if you offer a mobile option to your web panel? Social Science Computer Review, 544-560.Vehovar, V. et al. (2010). Mobile Phones in an Environment of Competing Survey Modes: Applying Metric for Evaluation of Costs and Errors. Social Science Computer Review, 28, 303-318.

Online surveys are mixed-device surveys: issues associated with the use of different (mobile) devices in web surveys

Abstract

Similar works

Full text

Available Versions

Name not available

NARCIS

Utrecht University Repository

Name not available

SSOAR - Social Science Open Access Repository