An assessment of the difficulty of questions used in the ISSP-questionnaires, the clarity of their wording, and the comparability of the responses

'International vergleichende Umfrageforschung basiert auf der Annahme, daß erstens die Antworten der Befragten in ausreichendem Maße Auskunft geben über das Verhalten und die Meinungen, die im Fragebogen abgefragt werden, und zweitens daß die Antworten der Befragten über die verschiedenen Länder hinweg ausreichend vergleichbar sing trotz der Unterschiede in den verwendeten Sprachen. Im Rahmen dieser Arbeit wird untersucht, ob diese Annahmefür eine große Auswahl von Umfragen gerechtfertigt ist, die im Kontext des International Social Survey Program (ISSP) entstanden sind. Auf der Basis eines konzeptionellen Modells, basierend auf methodischer Forschung zu Responseeffekten, sollen die Fragebögen bewertet werden. Es kann gezeigt werden, daß die Qualität der Umfrage von der Schwierigkeit und Klarheit der Fragen ebenso abhängig ist wie von der Länge des Fragebogens sowie der Präsenz oder Abwesenheit von verzerrenden Faktoren. Die Fragen des ISSP werden anhand von zehn Kriterien bewertet, die den Schwierigkeitsgrad der Fragen widerspiegeln und anhand von drei Indikatoren für die Eindeutigkeit der Fragestellungen. Die Vergleichbarkeit der Antworten wurde unter verschiedenen Aspekten bewertet, u.a. anhand der 'Vergleichbarkeit der Worte in der Frageformulierung' und anhand der 'Ähnlichkeit der zu bewertenden Objekte'. Die Ergebnisse zeigen, daß einige Fragen in ISSP-Surveys ziemlich schwierig und andere unklar formuliert sind. Darüber hinaus ist die Vergleichbarkeit der Antworten reduziert, da z. T unterschiedliche Intervieweranweisungen in den teilnehmenden Ländern gegeben wurden, und weil einige Fragen zu den Einstellungen in den Ländern unterschiedliche Bedeutungen haben.' (Autorenreferat)'The rationale of cross-national survey research rests on the assumptions (1) that the answers given by respondents are sufciently informative about the behaviours and opinions asked for in the questionnaires, and (2) that the answers of respondents participating in surveys, conducted in different countries, are sufficiently comparable, despite differences in the languages used. It is investigated whether these assumptions are justied for a large selection of surveys conducted within the context of the International SocialSurvey Program (ISSP). A conceptual model based an methodological research after response effects guides this evaluation of questionnaires. The model asserts that responsequality depends on the difficulty and clarity of the questions, on the length of the questionnaire and on the presence or absence of biasing factors. Questions from ISSP-questionnaires are evaluated by using ten criteria relating to question difficulty and three indicators concerning the clarity of the questions. The comparability of responses to these questions is evaluated by looking at six different aspects, like 'equivalent wording of the question' and 'similarity of the objects to be evaluated': this evaluation shows that some questions asked in Ihe ISSP-surveys are rather difficult while others are not very clear. The comparability of the responses is also hampered by the fact that different modes of questionnaire administration are used in the participating countries and that the attitude objects to be evaluated by respondents may differ between questionnaires.' (author's abstract)

Interviewer and Survey Researcher

Interviewer et survey researcher, dépendance mutuelle : L’explication du comportement demande l’utilisation de concepts subjectifs comme des opinions et des attitudes. Pour mesurer ces concepts non-directement observables, la standardisation stricte du processus de répondre aux questions est nécessaire. Si des interviewers ont été employés pour assurer une procédure correcte dans la collecte des données, ces interviewers peuvent aussi empêcher la standardisation. L’interviewer, est-il une aide  indispensable au chercheur et au répondant, ou un obstacle difficile dans l’obtention de réponses non-biaisées et comparables ? Pour répondre à ces questions, des transcriptions d’entretiens d’enquête ont été analysées concernant surtout les méthodes utilisées par les interviewers dans la réparation des réponses non-adequates des répondants. Les résultats de ces analyses et d’autres de la cybernétique montrent clairement qu’il n’est pas efficace de standardiser fortement le comportement de réparation des interviewers. L’analyse montre aussi la relation mutuelle entre l’interviewer et le survey researcher : celui-ci, en construisant le questionnaire et en fournissant des instructions concernant le comportement correct de « réparation », peut aussi être une aide ou un obstacle pour le travail de l’interviewer.The explanation of behavior requires the use of subjective concepts like opinions and attitudes. For the measurement of these non-directly observable concepts, strict standardization of the process of question answering is necessary. If interviewers have to be used to ensure a proper process of data collection, these interviewers may also hamper the standardization. Is the interviewer an indispensable help to researcher and respondent, or a difficult obstacle for obtaining unbiased and comparable answers? To answer this question, transcripts of survey interviews have been analyzed; especially the methods interviewers use to repair inadequate answers of respondents. The outcome of this analysis, and insights gained from cybernetics, make clear that it is ineffective to strongly standardize repair behavior of the interviewers. The analysis also points at the mutual relationship between interviewer and survey researcher: the survey researcher, when constructing the questionnaire and giving instructions about appropriate ‘repair’ behavior may also be a help or a hindrance for the interviewer

Control processes in survey interviews: A cybernetic approach

Purpose - In survey interviews information is transferred to the researchers via a communication process between interviewers and respondents. This process is controlled directly by the interviewers, and indirectly by the researchers who constructed the questionnaire and instructed and supervised the interviewers. In spite of these control activities, errors occur. This paper investigates the sources of these errors. Design/methodology/approach - In order to investigate the sources of these errors, transcripts of 200 interviews were analyzed using a detailed coding scheme. Findings - In 30 percent of all question-answer sequences interviewer and respondent stick to the "script" designed by the researcher. In these "paradigmatic" sequences the open loop control by the researcher works well. In 25 percent of the sequences this control is not sufficient, but additional closed loop control, via "repair" activities of the interviewers, appears to be successful. In the remaining sequences both the open loop control of the researcher and the closed loop control by the interviewer failed. Originality/value - The recently developed systematic analysis of question-answer sequences in survey interviews, employed in this research, offers detailed insight into the errors occurring during the interview process, and illustrates the need for improved question design and improved training of interviewers

The effect of the question topic on interviewer behavior; an interaction analysis of control activities of interviewers

Question topic, Retrospective questions, Survey interviews, Interviewer related measurement error, Interviewer behavior, Behavior coding, Interaction analysis,