Maria Francesca Romano

Maurizio Himmelmann

English

Research Papers in Economics

Methodological aspects of comparisons between Web &
paper surveys
Maria Francesca Romano and Maurizio Himmelmann
Sant’Anna School of Advanced Studies
piazza Martiri, 33 – 56127 Pisa, Italy
romano@sssup.it – himmel@sssup.it
In recent years there are many studies that compare diﬀerent modes in collecting data,
especially web vs paper mode. Comparisons are made between surveys or in the same survey;
in some study respondents are randomly assigned to modes and in others people can choose
the preferred mode. Results are often discordant, depending on type of population and/or by
study’s year.
We are presenting a comparison between paper and Web response mode, using data from
a longitudinal continuous survey with a choice-free response mode. The survey is extended to
a special population, deﬁned as graduates at Pisa University since 1995; survey’s goals are to
acquire university assessment at the moment of graduation and to help graduates in ﬁnding
job, collecting data for almost three years, both at successive waves and at graduate’s request.
Survey started as a paper survey in June 1997; since March 1998 a Web questionnaire was
implemented but we decided to not force people in using a mode that could be perceived not
adequate or not interesting: each respondent were, and is still, free to make a choice of response
mode.
At a ﬁrst glance, from our data it is evident that Web responses have a clear increasing
rate during time: the following table contains respondents classiﬁed for response mode, sur-
vey enrollment year and gender. It regards a subset of respondents, and precisely those who
compiled a questionnaire after Web questionnaire availability.
Changing in mode response from 1997 to 2002
1997 F M Total 2000 F M Total
mixed mode 36.1 63.9 9.72 mixed mode 50.5 49.5 18.69
only paper 52.5 47.5 90.28 only paper 57.7 42.3 63.23
only Web —- —- —- only Web 46.6 53.4 18.08
Total 50.9 49.1 1224 Total 53.7 46.2 2279
1998 2001
mixed mode 29.8 70.2 11.27 mixed mode 39.5 60.5 9.91
only paper 52.1 47.9 81.72 only paper 56.9 43.1 56.28
only Web 38.8 61.2 7.00 only Web 42.2 57.8 33.80
Total 48.5 51.5 1658 Total 50.3 49.7 2594
1999 2002
mixed mode 44.0 56.0 14.14 mixed mode 45.2 54.8 6.05
only paper 56.7 43.3 71.84 only paper 55.4 44.6 56.13
only Web 36.4 63.6 14.02 only Web 40.8 59.2 40.80
Total 51.7 48.2 1683 Total 49.1 50.9 2323
Survey characteristics and number of respondents permit us to assess comparisons between
paper and Web mode in diﬀerent manners. In this paper we compare:
1. response rate over time in order to investigate why graduates choose Web vs paper mode,
taking into account known variables (gender, year, discipline, place of residence, honorsand/or ﬁnal vote);
2. mode patterns among respondents to an equal number of questionnaires (in a longitudinal
way) in order to understand which variables (if any) could be able to predict choice;
3. questionnaires by year of compilation (or other interval of time) in a cross-sectional way:
each questionnaire is treated for itself, not considering if it is the ﬁrst or the nth one
compiled.
References
[1] DM Shannon CC Bradshaw. A comparison of response rate, response time, and costs
of mail and electronic surveys. Journal of Experimental Education, 70(2):179–192, WIN
2002.
[2] BF Frey. Investigating moral intensity with the world-wide web: a look at participant
reactions and a comparison of methods. Behavior Research Methods Instruments and
Computers, 32(3):423–431, 2000.
[3] RD Klassen J Jacobs. Experimental comparison of web, electronic and mail survey tech-
nologies in operations management. Journal of Operations Management, 19(6):713–728,
NOV 2001.
[4] AM Paolo GA Bonaminio C Gibson T Partridge K Kallail. Response rate comparisons
of e-mail- and mail-distributed student evaluations. Teaching and Learning in Medicine,
12(2):81–84, SPR 2000.
[5] C Cobanoglu B Warde PJ Moreo. A comparison of mail, fax and web-based survey meth-
ods. International Journal of Market Research, 43(4):441–452, 2001.
[6] A Grimsmo E Hagman E Falko L Matthiessen T Njalsson. Patients, diagnoses and pro-
cesses in general practice in the nordic countries - an attempt to make delta from com-
puterised medical records available for comparable statistics. Scandinavian Journal of
Primary Health Care, 19(2):76–82, JUN 2001.
[7] RM Carini JC Hayek GD Kuh JM Kennedy JA Ouimet. College student responses to web
and paper surveys: does mode matter? Research in Higher Education, 44(2):1–19, 2003.
[8] HW Perkins, GH Yuan. A comparison of web-based and paper-and-pencil library satisfac-
tion survey results. College and Research Libraries, 62(4):369–377, JUL 2001.
[9] JF Etter TV Perneger. A comparison of cigarette smokers recruited through the internet
or by mail. International Journal of Epidemiology, 30(3):521–525, JUN 2001.
[10] L Kleinman NK Leidy J Crawley A Bonomi P Schoenfeld. A comparative trial of paper-
and-pencil versus computer administration of the quality of life in reﬂux and dyspepsia
(qolrad) questionnaire. Medical Care, 39(2):181–189, 2001.
[11] B Medlin D Whitten. A comparison of internet and mail survey methodologies. Quirk´ s
Marketing Research Review, July 2001.

A comparative trial of paperand-pencil versus computer administration of the quality of life in reﬂux and dyspepsia (qolrad) questionnaire.

A comparison of cigarette smokers recruited through the internet or by mail.

A comparison of internet and mail survey methodologies.

A comparison of mail, fax and web-based survey methods.

A comparison of response rate, response time, and costs of mail and electronic surveys.

A comparison of web-based and paper-and-pencil library satisfaction survey results.

College student responses to web and paper surveys: does mode matter?

Experimental comparison of web, electronic and mail survey technologies in operations management.

Investigating moral intensity with the world-wide web: a look at participant reactions and a comparison of methods.

Patients, diagnoses and processes in general practice in the nordic countries - an attempt to make delta from computerised medical records available for comparable statistics.

Response rate comparisons of e-mail- and mail-distributed student evaluations.

Interpreting Web vs paper mode choice using supplementary data

Interpreting Web vs paper mode choice using supplementary data

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics