Eye tracking in retrospective think-aloud usability testing: Is there added value?

Eye tracking is the process of recording users’ eye movements while they are looking at the location of an object. In usability testing, this technique is commonly used in combination with think-aloud protocols. This paper presents an experimental study involving 24 participants; with the aim of comparing two variants of retrospective think-aloud (RTA) methods, that is, video-cued RTA method and gaze-cued RTA method, to address the value of having an extra eye-cue in retrospective think-aloud usability testing. Results suggest that both RTA variants are effective in detecting major usability problems. Moreover, the combination of eye tracking techniques and think-aloud protocols can further help evaluators to detect more usability problems, especially minor navigational and comprehension problems. It also helps participants to remember their behavior details, such as what they were looking at on a web page, as mouse movement alone might not be representative of their actual thoughts. Nevertheless, we found that participants might become distracted while seeing their eye movement, which can affect their verbalization performance and, hence, they might experience longer silence periods

The Tracer Method: Don\u27t Blink or You Might Miss it. A Novel Methodology Combining Cognitive Task Analysis and Eye Tracking

This thesis describes the development and first demonstration of a new Human Factors method, The Tracer Method, which is a combination of Cognitive Task Analysis (CTA) and Eye Tracking. The study evaluated whether the two methods together produce new and different information than either method alone could provide. The method was tested using a video game, Overwatch, a dynamic, complex, and multiplayer game. The evaluation included: 1. Examining both in the same context (game), 2. Establishing unique contributions of each method alone, and 3. Evaluating overlapping information. Results identified some overlap between the two methods that provided some cross-validation of the data. Cognitive Task Analysis provided higher level strategies and course of actions that players implement during their games, while eye tracking provided visual patterns of search (order of eye movements). However, when combined, the two methods provide strategy information in context that neither method alone can provide. CTA elicits insight into how individuals make decisions and apply previous knowledge, experience, and environmental information. Eye tracking can support this through predictive models of individual’s eye tracking, to understand which elements are utilized in making predictions and situational assessments. We provide a tutorial and insight into best practices for implementation of The Tracer Method. This is the initial development of the new method, and on-going research is validating it in different environments. The Tracer Method is the first combined and documented systematic methodology that utilizes a changing and complicated environment and tests the interaction and output of Critical Decision Method and Eye Tracking

Navigating the Web: A Qualitative Eye Tracking-based Study of Translators’ Web Search Behaviour

This Element reports an investigation of translators’ use of web-based resources and search engines. The study adopted a qualitative eye tracking-based methodology utilising a combination of gaze replay and retrospective think aloud (RTA) to elicit data. The main contribution of this Element lies in presenting not only an alternative eye tracking methodology for investigating translators’ web search behaviour but also a systematic approach to gauging the reasoning behind translators’ highly complex and context-dependent interaction with search engines and the Web

Navigating the Web. A Qualitative Eye Tracking–Based Study of Translators' Web Search Behaviour

This Element reports an investigation of translators' use of web-based resources and search engines. The study adopted a qualitative eye tracking-based methodology utilising a combination of gaze replay and retrospective think aloud (RTA) to elicit data. The main contribution of this Element lies in presenting not only an alternative eye tracking methodology for investigating translators' web search behaviour but also a systematic approach to gauging the reasoning behind translators' highly complex and context-dependent interaction with search engines and the Web

Deaf and Hard of Hearing Readers and Science Comics: A Mixed Methods Investigation on Process

Deaf and hard of hearing (DHH) students bring diverse language and literacy backgrounds to the task of academic reading, which becomes increasingly complex and abstract in the upper grades. Teachers often differentiate their instruction by providing multimedia resources, of which students interact with verbal and pictorial information. A growing body of research supports multimedia learning; however, most of the studies have focused exclusively on learning outcomes, leaving teachers in the dark about the cognitive processes underlying these effects. This mixed methods study addresses this gap by using a nonfiction comic to investigate the reading processes of DHH 7th -12th grade students. Eye tracking and cued retrospective protocol were employed in a concurrent nested design to answer the question, how do DHH students read and learn from multimedia science texts? This study was guided by the cognitive theory of multimedia which states that reading comprehension is better supported when learning from words and pictures rather than words alone, especially when readers cognitively integrate the two representations to form a coherent mental model. Temporal and sequential eye tracking results revealed that readers’ transitions between related words and pictures were a statistically significant variable in explaining factual knowledge learning outcomes. These strategic shifts in attention were further explained by readers’ retrospective verbal reports of their thinking. Students’ descriptions of their vii reading processes were interpreted into the following themes: repairing, connecting representations, passive transitions, and connecting to self. The integration of quantitative and qualitative methods at the interpretation stage revealed that although the theme of repairing was equally distributed across all student reports, the theme of connecting representations was largely present in the reports from students who made high counts of integrative transitions. The major findings of this study align with the cognitive theory of multimedia, that students’ learning outcomes were significantly predicted by the deliberate strategies to cognitively integrate words and pictures to form and maintain a coherent mental model. The discussion includes ways in which teachers can capitalize on explicit modeling of these behaviors and employ students’ “think alouds” to better understand and support the development of effective multimedia reading processes

An Investigation of the Impact of Task-Types on the Reactivity of the Concurrent Think-Aloud in Usability Testing

The Concurrent think-aloud (CTA) is primarily used to understand users’ task based cognitive processes. However, is not without limitations. CTA procedures varies widely among practitioners. Also, it has been known to cause reactivity: an artificial change in task performance. This is problematic because it may alter the accuracy of task performance. Also, research on reactivity within usability testing have shown mixed findings. Thus, conclusions cannot be drawn to attest to whether reactivity occurs due to varying administration procedures and therefore we must now consider its relationship to other test-based factors. This research will be the first to systematically investigate the impact of task-type on reactivity of the CTA and the first to systematically investigates practitioners working habit in terms of their views on reactivity when using CTA in practice. Three studies were conducted, the first study investigates the Impact of task-types on the Reactivity of CTA and uses a mixed design. The results suggest that, thinking aloud during usability testing does not cause reactivity, and task type does not impact concurrent think-aloud. However, sensemaking tasks increase mental demand. The second study investigates the impact of task-type on two different think-aloud protocols and uses a mixed design. The result indicates that, the classic think-aloud method led to more successful task completion and no reactivity, while the explicit instruction produced fewer successful task completions and a higher mental workload. The explicit instruction produced less verbalisation, resulting in fewer relevant explanatory utterances, contradicting expectations. The third study uses an interview method to explores practitioners' experiences, views on reactivity and challenges when using the think-aloud method within usability testing. These studies demonstrates unequivocally that CTA should not be abandoned in usability studies as it provided valuable think-aloud data and helped identify usability issues. Additionally, practitioners should not replace the traditional think- aloud approach with explicit instruction, as explicit instruction had a greater influence on participants' behaviour. Ericsson and Simon's recommendations should be used for concurrent data collection, as it ensures data validity and generates the same type of data as explicit instruction while reducing reactivity