The use of mobile devices to complete online surveys is trending upward and is likely to continue unabated for some time. With this growth, there is a growing interest in how to make mobile devices a viable way to administer self-report online surveys among researchers. Utilizing a large-scale online panel survey, the current study compares the quality and pattern of responses collected from mobile device respondents to those collected from PC/laptop respondents. It is found that although mobile respondents tended to provide shorter answers to open-ended questions than PC/laptop respondents, response quality was not significantly different between the two groups. Additionally, compared to PC/laptop respondents, mobile device respondents were younger, more likely to be female and with a higher income level. Practical implications related to our research findings are also discussed

Li, Xiang (Robert)

Tan, Karen P.S.

Zou, Suiwen

English

ScholarWorks@UMass Amherst

University of Massachusetts AmherstScholarWorks@UMass AmherstTravel and Tourism Research Association:Advancing Tourism Research Globally 2018 ttra International ConferenceMobile versus PC: Does Device Type AffectOnline Survey Response Quality for TourismResearch?Suiwen ZouTemple UniversityKaren P.S. TanTemple UniversityXiang (Robert) LiTemple UniversityFollow this and additional works at: https://scholarworks.umass.edu/ttraThis Event is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Travel and TourismResearch Association: Advancing Tourism Research Globally by an authorized administrator of ScholarWorks@UMass Amherst. For moreinformation, please contact scholarworks@library.umass.edu.Zou, Suiwen; Tan, Karen P.S.; and Li, Xiang (Robert), "Mobile versus PC: Does Device Type Affect Online Survey Response Qualityfor Tourism Research?" (2018). Travel and Tourism Research Association: Advancing Tourism Research Globally. 16.https://scholarworks.umass.edu/ttra/2018/Academic_Papers_Oral/16 Mobile versus PC:  Does Device Type Affect Online Survey Response Quality for Tourism Research?  Introduction The use of mobile devices, such as smartphones and tablets, to fill out online surveys has grown steadily (Toepoel & Lugtig, 2013). It is reported that 35% of respondents completed their most recent web-based survey on a mobile device (Pew Research Center, 2015). In the tourism context, using mobile devices to collect survey data enables researchers to access precise spatial and temporal data (Raun, Ahas & Tiru, 2016; Ayscue, Boley & Mertzlufft, 2016), which is critical to study travel, a phenomenon that is geographical and mobile in nature. Therefore, it is important to understand the features of using mobile devices collecting data from a methodological perspective.  While mobile technology opens up new opportunities for data collection, some researchers have expressed their concerns about the quality of data collected from mobile devices (Peytchev & Hill, 2010). Compared to PCs and laptops, mobile devices have a smaller screen and a virtual keyboard, making it more challenging for respondents to read and provide inputs (Peytchev & Hill, 2010). Furthermore, respondents using mobile devices are more likely to be in an away-from-home/work social environment (Mavletova, 2013) and are hence more likely to be distracted (Antoun, Couper & Conrad, 2017).  Given the above, the objective of this study is to compare the quality of online survey data collected from mobile device respondents to those collected from respondents using PCs/laptops. To effectively evaluate response quality, we use four indicators: (1) the level of straightlining of grid questions; (2) internal consistency of scales; (3) the length of answers to open-ended questions; and (4) the likelihood of selecting the “Other” option (that required typing a response).  Literature Review  One criticism of online surveys is that researchers have limited control over the way respondents complete the survey (Toepoel & Lugtig, 2013); respondents are able to choose the time, place and device. As more people using mobile devices to fill out web-based surveys these days, it is imperative for researchers to understand how device type would affect response quality before designing a questionnaire (Peytchev & Hill, 2010). However, only a handful of studies have addressed this topic, and to the authors’ best knowledge, none of them are in the field of tourism. There are a number of potential problems that survey respondents on mobile devices might face when compared to respondents on PCs/laptops. With a small display, fingertip-typing keyboard and different methods of navigation, mobile devices may inhibit respondents’ ability to comprehend and complete the survey (Toepoel & Lugtig, 2013; Peytchev & Hill, 2010). Besides feature differences, the portable nature of mobile devices may also have an undesirable effect on response quality. For instance, Mavletova (2013) reported that 20% of mobile respondents took the survey away-from-home, compared to 3% of desktop respondents. In addition, Antoun, Couper and Conrad (2017) found that more smartphone respondents than PC respondents were multitasking while completing the survey. It is reasonable to assume that with multitasking and being away-from-home, respondents are more likely to be distracted, less focused on the survey, resulting in low quality and accuracy in their responses.  Methodology This study utilized an online panel survey conducted via a professional third-party data collection company during September 2017 among American leisure tourists. The population of this study comprised American travelers (1) who are 18 years or older; and (2) who have taken a leisure vacation in the past 12 months. The sample was demographically balanced based on geographical region (e.g. West, Midwest, South, and Northeast) and age (e.g. 18-29, 30-44, 45-59 and 60+ years old). The questionnaire included multiple choice questions (including check-all-that-apply questions), Likert-type scales in a grid format, and open-ended questions.  Results The average age of the respondents was 46.4, 51.1% of them were female, and 82.7% were White. Among the 2,643 respondents, 48.7% used a mobile device to complete the survey and 51.3% used a computer. Mobile respondents tended to be younger, more likely to be female, and more likely to have a high household income level. Straightlining refers to a measurement error in which survey respondents select the same response for every item in a grid question (Antoun et al., 2017). Chi-square test results showed that PC/laptop respondents were more likely to straightline than mobile device.  The time that the respondents took to complete the survey was recorded and measured in seconds. The t-test result revealed that it took longer for mobile device respondents to complete the survey than for PC/laptop respondents. In terms of the time of a day that respondents responded to the survey invitation, results revealed that more PC/laptop respondents took the survey at 3-6am, 2 pm, and 8 pm, while more mobile respondents took the survey during 7-10am. T-tests were conducted on four open-ended questions to detect differences in response length. Results showed that for the open-ended questions in sequence, a significant difference was found in one (the 3rd) of the three questions. Moreover, a significant difference was found in the response length of Q4 (i.e., the open-ended question positioned at the very end of the survey). The Feldt test (Feldt, 1969) was performed to compare the alpha coefficients between the two groups. Statistical differences in alpha coefficients were found in only two of the six scales. In both scales, the alpha coefficients of PC/laptop respondents were higher than those of mobile respondents.  Two binomial logistic regressions were performed to ascertain the effects of device type used by respondents on the likelihood of choosing the “Other” option, controlling demographic variables of age, gender and income. The results of both models revealed that device type was not a statistically significant predictor (p >.05) in holding age, gender and income constant.  Discussions and Implications As more respondents choose to take a web-based survey via mobile devices, it is critical for researchers to bear in mind the features of mobile devices when designing research plans and web-based surveys. Based on our findings, we draw the following conclusions regarding online survey design. Firstly, since mobile respondents tend to provide shorter responses to open-ended questions, surveys with a relatively large share of open-ended questions should limit the use of mobile devices during data collection. Secondly, if a substantial number of respondents are  anticipated to use mobile devices, open-ended questions should be placed at the front of the survey to prevent the negative effect of fatigue or impatience. Thirdly, based on the results of the current study, online survey data collected from mobile devices is not of lower quality, which appears to be promising for tourism researchers who want to take advantage of mobile devices in their research design. Last but not least, through our survey, mobile device respondents were found to be younger, more likely to be females, have a higher income level, and more likely to respond a survey in the morning. Researchers should take these demographic and behavioral patterns into consideration during the sampling stage.  There are limitations to the current study which can be addressed by future research opportunities. Respondents in this study self-selected their device instead of being randomly assigned to one. To reduce self-selection effects, future research is recommended to conduct experiments with random assignment to verify the results of this study. Additionally, this study only described what the differences are in response quality between the mobile and PC/laptop. Future research can look at why and how survey experience completed on mobile devices differ from that on PCs/laptops by utilizing advanced methods such as eye-tracking technology.   Acknowledgements We acknowledge financial support for this study from the China National Tourist Office, Los Angeles and Nankai University, China. References  Antoun, C., Couper, M. P., & Conrad, F. G. (2017). Effects of Mobile versus PC Web on Survey Response Quality. Public Opinion Quarterly, 81, 280–306.  Ayscue, E. P., Boley, B. B., & Mertzlufft, C. E. (2016). Mobile technology &amp; resident attitude research. Tourism Management, 52, 559–562.  Feldt, L. S. (1969). A test of the hypothesis that Cronbach's alpha reliability coefficient is the same for two tests administered to the same sample. Psychometrika, 45, 99-105. Mavletova, A. (2013). Data Quality in PC and Mobile Web Surveys. Social Science Computer Review, 31(6), 725–743.  Peytchev, A., & Hill, C. A. (2010). Experiments in Mobile Web Survey Design Similarities to Other Modes and Unique Considerations. Social Science Computer Review, 28(3), 319–335.  Pew Research Center. (2015). Tips for Creating Web Surveys for Completion on a Mobile Device. Retrieved from http://www.pewresearch.org/2015/06/11/tips-for-creating-web-surveys-for-completion-on-a-mobile-device/. Raun, J., Ahas, R., & Tiru, M. (2016). Measuring tourism destinations using mobile tracking data. Tourism Management, 57, 202–212.  Toepoel, V., & Lugtig, P. (2013). What Happens if You Offer a Mobile Option to Your Web Panel? Evidence From a Probability-Based Panel of Internet Users. Social Science Computer Review, 32(4), 544–560.   

Mobile versus PC:  Does Device Type Affect Online Survey Response Quality for Tourism Research?

https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=2215&context=ttra

Mobile versus PC: Does Device Type Affect Online Survey Response Quality for Tourism Research?

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst