Use of Eye Tracking in Cognitive Pretests (Version 1.0)

Cognitive pretesting is generally considered to be indispensable for the successful development of new survey questions, and hence for the quality of the data obtained by the survey. Supplementing cognitive interviewing with the method of eye tracking offers the possibility to observe eye movements of respondents in real-time providing additional information about cognitive processes of respondents. Research suggests that combining both methods helps to identify additional problems with questions that would remain undetected if only one method was applied. This contribution provides an introduction to cognitive interviewing in combination with eye tracking. The following questions are addressed: What is the rationale behind combining cognitive interviewing and eye tracking? How should eye tracking be implemented into cognitive interviewing? How can eye-tracking data be used and analyzed in the context of cognitive pretesting

Incorporating eye tracking into cognitive interviewing to pretest survey questions

In this study, we investigated whether incorporating eye tracking into cognitive interviewing is effective when pretesting survey questions. In the control condition, a cognitive interview was conducted using a standardized interview protocol that included pre-defined probing questions for about one-quarter of the questions in a 52-item questionnaire. In the experimental condition, participants' eye movements were tracked while they completed an online version of the questionnaire. Simultaneously, their reading patterns were monitored for evidence of response problems. Afterward, a cognitive interview was conducted using an interview protocol identical to that in the control condition. We compared both approaches with regard to the number and types of problems they detected. We found support for our hypothesis that cognitive interviewing and eye tracking complement each other effectively. As expected, the hybrid method was more productive in identifying both questionnaire problems and problematic questions than applying cognitive interviewing alone

A Comparison of Two Cognitive Pretesting Techniques Supported by Eye Tracking

In questionnaire pretesting, supplementing cognitive interviewing with eye tracking is a promising new method that provides additional insights into respondents' cognitive processes while answering survey questions. When incorporating eye tracking into cognitive interviewing, two retrospective probing techniques seem to be particularly useful. In the first technique - retrospective probing - participants complete an online questionnaire, while cognitive interviewers monitor participants’ eye movements in an adjacent room and note down any peculiarities in their reading patterns. Afterward, the interviewers ask targeted probing questions about these peculiarities in a subsequent cognitive interview. In the second technique - gaze video cued retrospective probing - respondents are additionally shown a video of their eye movements during the cognitive interview. This video stimulus is supposed to serve as a visual cue that may better enable respondents to remember their thoughts while answering the questions. We examine whether one of the two techniques is more effective when it comes to identifying problematic survey questions. In a lab experiment, participants' eye movements (n = 42) were tracked while they completed six questions of an online questionnaire. Simultaneously, their reading patterns were monitored by an interviewer for evidence of response problems. After completion of the online survey, a cognitive interview was conducted. In the retrospective probing condition, probing questions were asked if peculiar reading patterns were observed during the eye-tracking session (e.g., rereadings of specific words or text passages). In the other condition, participants were shown a video of their recorded eye movements, in addition to receiving probing questions about the questions displayed. Results show that both techniques did not differ in terms of the total number of problems identified. However, gaze video cued retrospective probing identified fewer unique problems and fewer types of problems than pure retrospective probing

Eye tracking in questionnaire pretesting

It is generally acknowledged that new questions or survey instruments require some form of pre-evaluation before they are actually fielded, in order to check their validity and minimize measurement error. This is the task of questionnaire pretesting. Eye tracking is one of the most recent additions to the survey pretester’s toolbox. During eye tracking, the position of respondents’ eyes is observed, to detect where they are looking. While being recognized as a promising technique to indicate potential problems with survey items and obtain insights into the underlying cognitive processes, there is little resilient evidence on its productivity. The research presented in this thesis contributes to survey pretesting methodology by examining the productivity of eye tracking in problem detection and question design, both in combination with cognitive interviewing or as a stand-alone technique

An experimental test of the effectiveness of cognitive interviewing in pretesting questionnaires

Pretesting survey questions via cognitive interviewing is based on the assumptions that the problems identified by the method truly exist in a later survey and that question revisions based on cognitive interviewing findings produce higher-quality data than the original questions. In this study, we empirically tested these assumptions in a web survey experiment (n = 2,200). Respondents received one of two versions of a question on self-reported financial knowledge: either the original draft version, which was pretested in ten cognitive interviews, or a revised version, which was modified based on the results of the cognitive interviews. We examined whether the cognitive interviewing findings predicted problems encountered in the web survey and whether the revised question version was associated with higher content-related and criterion-related validity than the draft version. The results show that cognitive interviewing is effective in identifying real question problems, but not necessarily in fixing survey questions and improving data quality. Overall, our findings point to the importance of using iterative pretesting designs, that is, carrying out multiple rounds of cognitive interviews and also testing the revisions to ensure that they are indeed of higher quality than the draft questions

International Social Survey Programme (ISSP) 2015: Cognitive Pretest

The International Social Survey Programme (ISSP) is an international cooperation programme that conducts an annual joint survey on topics relevant to the social sciences. Since its foundation in 1984, the ISSP has grown to 48 member countries in 2013. The subject of this pretest are questions from the ISSP survey in Germany about work-life-balance, working hours, and the bond with the employer. The questions were tested via face-to-face cognitive interviews

Questionnaire design decisions when transitioning from an interviewer-administered to a self-administered online mode (Version 1.0)

Large-scale surveys are increasingly moving from (face-to-face or telephone-based) interviewer-administered to self-administered online modes. To ensure high measurement quality and maximum comparability and equivalence between the source questionnaire and its adaptation - across modes as well as across survey waves - various aspects of question design must be considered and several decisions need to be made concerning question presentation and wording. This survey guideline summarizes good practices on how to transition questionnaires from interviewer- to self-administered web surveys and gives recommendations and examples for major adaptation issues as well as general questionnaire design elements relevant to web surveys. In this context, we focus on the switch from an interviewer-based to an online mode (although mixed-mode designs are also conceivable).Groß angelegte Studien werden zunehmend von interviewerbasierten (persönlichen oder telefonischen) Befragungen auf selbstverwaltete Onlineerhebungen umgestellt. Um eine hohe Datenqualität und ein Höchstmaß an Vergleichbarkeit und Äquivalenz zwischen dem Ausgangsfragebogen und seiner Adaptation - sowohl über verschiedenen Modi als auch Erhebungswellen hinwe - zu gewährleisten, müssen verschiedene Aspekte des Fragebogendesigns berücksichtigt und eine Reihe von Entscheidungen in Bezug auf die Präsentation und Formulierung von Fragen getroffen werden. Dieser Erhebungsleitfaden fasst bewährte Methoden zur Umwandlung der Fragebögen von interviewgestützten zu selbstgestützten Onlineerhebungen zusammen und gibt Empfehlungen und Beispiele für wichtige Anpassungsfragen sowie zu allgemeinen Gestaltungselementen von Fragebögen, die für Onlineumfragen relevant sind. In diesem Zusammenhang konzentrieren wir uns auf den Wechsel von einem interviewergestützten zu einem Onlinemodus (obwohl auch gemischte Modi denkbar sind)

Cognitive Pretesting (Version 2.0)

In survey research, cognitive pretesting is generally considered an essential prerequisite for the successful development of a questionnaire and thus for the quality of the data collected within the framework of a survey. This contribution provides an introduction to the topic of questionnaire pretesting and focuses especially on the planning and implementation of cognitive pretests. The following questions are addressed, in particular: What is a pretest and why should you conduct questionnaire pretests? What pretesting procedures are available and which one should you choose? What techniques are applied in cognitive pretests? How are cognitive pretests conducted? How are cognitive pretests evaluated