Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience

This article focuses on assessing item comparability in cross-national surveys by asking probing questions in Web surveys. The "civil disobedience" item from the "rights in a democracy" scale of the International Social Survey Program (ISSP) serves as a substantive case study. Identical Web surveys were fielded in Canada (English-speaking), Denmark, Germany, Hungary, Spain, and the U.S. A category-selection and a comprehension probe, respectively, were incorporated into the Web surveys after the closed-ended "civil disobedience" item. Responses to the category selection-probe reveal that notably in Germany, Hungary, and Spain the detachment of politicians from the people and their lack of responsiveness is deplored. Responses to the comprehension probe show that mainly in the U.S. and Canada violence and/or destruction are associated with civil disobedience. These results suggest reasons for the peculiar statistical results found for the "civil disobedience" item in the ISSP study. On the whole, Web probing proves to be a valuable tool for identifying interpretation differences and potential bias in cross-national survey research

Asking probing questions in web surveys: which factors have an impact on the quality of responses?

Cognitive interviewing is a well-established method for evaluating and improving a questionnaire prior to fielding. However, its present implementation brings with it some challenges, notably in terms of small sample sizes or the possibility of interviewer effects. In this study, the authors test web surveys through nonprobability online panels as a supplemental means to implement cognitive interviewing techniques. The overall goal is to tackle the above-mentioned challenges. The focus in this article is on methodological features that pave the way for an eventual successful implementation of category-selection probing in web surveys. The study reports on the results of 1,023 respondents from Germany. In order to identify implementation features that lead to a high number of meaningful answers, the authors explore the effects of (1) different panels, (2) different probing variants, and (3) different numbers of preceding probes on answer quality. The overall results suggest that category-selection probing can indeed be implemented in web surveys. Using data from two panels - a community panel where members can actively get involved, for example, by creating their own polls, and a "conventional" panel where answering surveys is the members' only activity - the authors find that high community involvement does not increase the likelihood to answer probes or produce longer statements. Testing three probing variants that differ in wording and provided context, the authors find that presenting the context of the probe (i.e., the probed item and the respondent's answer) produces a higher number of meaningful answers. Finally, the likelihood to answer a probe decreases with the number of preceding probes. However, the word count of those who eventually answer the probes slightly increases with an increasing number of probes. (author's abstract

Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys

This article examines the use of probing techniques in web surveys to identify validity problems of items. Conventional cognitive interviewing is usually based on small sample sizes and thus precludes quantifying the findings in a meaningful way or testing small or special subpopulations characterized by their response behavior. This article investigates probing in web surveys as a supplementary way to look at item validity. Data come from a web survey in which respondents were asked to give reasons for selecting a response category for a closed question. The web study was conducted in Germany, with respondents drawn from online panels (n = 1,023). The usefulness of the proposed approach is shown by revealing validity problems with a gender ideology item

Cognitive probes in web surveys: on the effect of different text box size and probing exposure on response quality

In this study, we explore to what extent the visual presentation of open-ended probes, in connection with different prior probing exposure, impacts on response quality. We experiment with two text box sizes for a specific immigrant probe (Which type of immigrants were you thinking of when you answered the question?). On the one hand, we use a standard size equal to the other text box sizes in the survey but oversized for the specific response task. On the other hand, we use a smaller text box which fits the response task. The other probes in the survey that use the standard text box are mainly category-selection probes that ask for a reasoning for the chosen answer value. Due to randomized rotation of questions, respondents receive different numbers of category-selection probes prior to the immigrant probe, resulting in different degrees of exposure to category-selection probing prior to the immi-grant probe. For the immigrant probe, we find that respondents who receive the standard text box and who have had a high exposure to category-selection probing are more likely to provide mismatching answers: The mismatch consists of not answering the specific immigrant probe but rather providing a reasoning answer as typically expected for a category-selection probe. Thus, previous experience with probing in the questionnaire can override the actual probe wording. This problem can be minimized by considering possible carryover effects of prior probes and using an appropriate survey design strategy

Prenotification in web-based access panel surveys: the influence of mobile text messaging versus e-mail on response rates and sample composition

To compare the effectiveness of different prenotification and invitation procedures in a webbased three-wave access panel survey over 3 consecutive months, we experimentally varied the contact mode in a fully crossed two-factorial design with (a) three different prenotification conditions (mobile short messaging service [SMS], e-mail, no prenotice) and (b) two “invitation and reminder" conditions (SMS, e-mail). A group with nearly complete mobile phone coverage was randomly assigned to one of these six experimental conditions. As expected, SMS prenotifications outperformed e-mail prenotifications in terms of response rates across all three waves. Furthermore, e-mail invitation response rates outperformed those for SMS invitations. The combination of SMS prenotification and e-mail invitation performed best. The different experimental treatments did not have an effect on the sample composition of respondents between groups. (author's abstract

Effekte des Erhebungsverfahrens? Ein Vergleich zwischen einer Web-basierten und einer schriftlichen Befragung zum ISSP-Modul Umwelt

'Eine inhaltlich identische Befragung (Umwelteinstellungsmodul des International Social Survey Programme, ISSP) wurde sowohl im Rahmen einer bevölkerungsrepräsentativen schriftlichen Befragung als auch in Form einer Web-basierten Version mit Hilfe eines für die Internetnutzerschaft repräsentativen Online-Access-Panels durchgeführt. Im direkten Gruppenvergleich über beide Erhebungsmodi zeigten sich zunächst deutliche Unterschiede über inhaltliche wie auch soziodemografische Variablen. Eine Anpassungsgewichtung der Stichprobe der Internetnutzer an die bevölkerungsrepräsentative Stichprobe auf der Basis soziodemografischer Variablen ergab für den Vergleich über inhaltliche Items ein inkonsistentes Bild. Beschränkt sich dagegen der Vergleich unter beiden Erhebungsmodi auf solche Personen, die über einen ähnlichen Bildungshintergrund verfügen, so sind weder über die Itemmittelwerte noch über ausgewählte Item-Interkorrelationen praktisch bedeutsame Unterschiede ermittelbar. D.h. die Online erhobenen Daten stimmen für eine ausgewählte High-Coverage Gruppe (hier: Personen mit hoher Bildung) weitgehend überein mit denen der schriftlichen Variante.' (Autorenreferat)'The module 'environmental attitudes and values' of the International Social Survey Programme (ISSP) was administered both as a paper-and-pencil questionnaire to a representative sample of the German population and as a Web-based survey to an online access panel representative for German Internet users. These two Samples differ significantly with regard to sociodemographic and Substantive variables. An attempt to weight the data of the Web-based sample an the basis of distribution characteristics of several socio-demographic variables resulted in rather inconsistent findings for the comparison of substantive items. However, if only respondents with similar education levels are compared, neither relevant differences in item means nor differences in selected inter-item correlations are observable. This means that for high coverage groups (e.g. subjects with a high education level), the data gathered via the Web are basically identical to those obtained in a traditional self-administered mode.' (author's abstract)

ALLBUS Baseline-Studie 1991: Methodenbericht zum DFG-Projekt

Im Rahmen des ALLBUS (Allgemeine Bevökerungsumfrage der Sozialwissenschaften) wurde 1991 eine Baseline-Studie in beiden Teilen Deutschlands durchgeführt, um Grunddaten für die künftige Erfassung des zu erwartenden gesellschaftlichen Wandels zu erhalten und Zeitreihen zu begründen. Dieser Bericht befaßt sich mit Planung, Vorbereitung und Durchführung der Studie, wobei die Grundkonzeption, das Fragenprogramm, der Pretest, das Stichprobendesign sowie die Interviewersituation und -kontrolle dargestellt und dokumentiert werden. (psz

Participation in web surveys: a typology

'Während man in der Umfrageforschung im Wesentlichen von drei (Nicht)Antwortmustern bei Befragungen ausgeht (Unit nonresponse, Item nonresponse und vollständige Befragungsteilnahme), können bei Web-basierten Befragungen (Web surveys) Daten über den Befragungsprozess miterhoben werden, die einen erweiterten Einblick in das tatsächliche Geschehen bei der Beantwortung von Fragen geben. Aufbauend auf diesen Bearbeitungsprozessdaten lassen sich mindestens sieben (Nicht-)Antwortmuster voneinander unterscheiden, die im Rahmen dieses Beitrages beschrieben sowie hinsichtlich ihrer theoretischen und praktischen Implikationen diskutiert werden.' (Autorenreferat)'While traditional survey literature has addressed three possible (non-)response patterns (unit nonresponse, item nonresponse, and complete response), web surveys can capture data about a respondent's answering process, enabling researchers to attain more fine-grained information about individual reactions to such surveys. Based on this information, at least seven (non-)response patterns can be distinguished. This paper describes these seven patterns in a typology of (non-)response. Finally, theoretical and practical implications are discussed.' (author's abstract)