Improving the understanding of web user behaviors through machine learning analysis of eye-tracking data

Eye-tracking techniques are widely used to analyze user behavior. While eye-trackers collect valuable quantitative data, the results are often described in a qualitative manner due to the lack of a model that interprets the gaze trajectories generated by routine tasks, such as reading or comparing two products. The aim of this work is to propose a new quantitative way to analyze gaze trajectories (scanpaths) using machine learning. We conducted a within-subjects study (N = 30) testing six different tasks that simulated specific user behaviors in web sites (attentional, comparing two images, reading in different contexts, and free surfing). We evaluated the scanpath results with three different classifiers (long short-term memory recurrent neural network—LSTM, random forest, and multilayer perceptron neural network—MLP) to discriminate between tasks. The results revealed that it is possible to classify and distinguish between the 6 different web behaviors proposed in this study based on the user’s scanpath. The classifier that achieved the best results was the LSTM, with a 95.7% accuracy. To the best of our knowledge, this is the first study to provide insight about MLP and LSTM classifiers to discriminate between tasks. In the discussion, we propose practical implications of the study results

Representative Scanpath Identification for Group Viewing Pattern Analysis

Scanpaths are composed of fixations and saccades. Viewing trends reflected by scanpaths play an important role in scientific studies like saccadic model evaluation and real-life applications like artistic design. Several scanpath synthesis methods have been proposed to obtain a scanpath that is representative of the group viewing trend. But most of them either target a specific category of viewing materials like webpages or leave out some useful information like gaze duration. Our previous work defined the representative scanpath as the barycenter of a group of scanpaths, which actually shows the averaged shape of multiple scanpaths. In this paper, we extend our previous framework to take gaze duration into account, obtaining representative scanpaths that describe not only attention distribution and shift but also attention span. The extended framework consists of three steps: Eye-gaze data preprocessing, scanpath aggregation and gaze duration analysis. Experiments demonstrate that the framework can well serve the purpose of mining viewing patterns and “barycenter” based representative scanpaths can better characterize the pattern

Development of methodologies to analyze and visualize air traffic controllers’ visual scanning strategies

The Federal Aviation Administration (FAA) estimates a 60 million air traffic volume by 2040. However, the available workforce of expert air traffic controllers (ATCs) might not be sufficient to manage this anticipated high traffic volume. Thus, to maintain the same safety standard and service level for air travel, more ATCs will need to be trained quickly. Previous research shows eye tracking technology can be used to enhance the training of the ATC’s by reducing their false alarm rate, thereby helping to mitigate the impact of increasing demand. Methods need to be developed to better understand experts’ eye movement (EM) data so as to incorporate them effectively in ATCs’ training process. However, it’s challenging to analyze ATCs’ EM data for several factors: (i) aircraft representation on radar display (i.e. targets) are dynamic, as their shape and position changes with time; (ii) raw EM data is very complex to visualize, even for the meaningful small duration (e.g. task completion time of 1 min); (iii) in the absence of any predefined order of visual scanning, each ATC employ a variety of scanning strategies to manage traffic, making it challenging to extract relevant patterns that can be taught. To address these aforementioned issues, a threefold framework was developed: (i) a dynamic network-based approach that can map expert ATCs’ EM data to dynamic targets, enabling the representation of visual scanning strategy evolution with time; (ii) a novel density-based clustering method to reduce the inherent complexity of ATCs’ raw EM data to enhance its visualization; (iii) a new modified n-gram based similarity analysis method, to evaluate the consistency and similarity of visual scanning strategies among experts. Two different experiments were conducted at the FAA Civil Aerospace Medical Institute in Oklahoma City, where EM data of 15 veteran ATCs’ (> 20 years of experience) were collected using eye trackers (Facelab and Tobii eye trackers), while they were controlling a high-fidelity simulated air traffic. The first experiment involved en-route traffic scenario (with aircraft above 18,000 feet) and the second experiment consisted of airport tower traffic (aircraft within 30 miles radius from an airport). The dynamic network analysis showed three important results: (i) it can be used to effectively represent which are the important targets and how their significance evolves over time, (ii) in dynamic scenarios, having targets having variable time on display, traditional target importance measure (i.e. the number of eye fixations and duration) can be misleading, and (iii) importance measures derived from the network-based approach (e.g. closeness, betweenness) can be used to understand how ATCs’ visual attention moves between targets. The result from the density-based clustering method shows that by controlling its two parameter values(i.e. spatial and temporal approximation), the visualization of the raw EM data can be substantially simplified. This approximate representation can be used for better training purpose where expert ATC’s visual scanning strategy can be visualized with reduced complexity, thereby enhancing the understanding of novices while maintaining its significant pattern (key for visual pattern mining). Moreover, the model parameters enable the decision-maker to incorporate context-dependent factors by adjusting the spatial (in pixel) and temporal (in milliseconds) thresholds used for the visual scanning approximation. The modified n-gram approach allows for twofold similarity analysis of EM data: (i) detecting similar EM patterns due to exact sequential match in which the targets are focused and/or grouped together visually because of several eye fixation transitions among them, and (ii) unearth similar visual scanning behavior which is otherwise small perturbed version of each other that arise as a result of idiosyncrasies of ATCs. Thus, this method is more robust compared to other prevalent approaches which employ strict definitions for similarity that are difficult to empirically observe in real-life scenarios. To summarize, the three methods developed allow us to apply a comprehensible framework to understand the evolving nature of the visual scanning strategy in complex environments (e.g. air traffic control task) by: (i) by identifying target importance & their evolution; (ii) simplifying visualizing of complex EM strategy for easier comprehension; (iii) evaluating similarity among various visual scanning strategies in dynamic scenarios

Multimodal Neuroergonomic Approaches to Human Behavior and Cognitive Workload in Complex High-Risk Semantically Rich Environments: A Case Study of Local & En-Route Air Traffic Controllers

Fast-paced technology advancements have enabled us to create ecologically valid simulations of high risk, complex, and semantically rich environments in which human interaction and decision-making are the keys to increase system performance. These advances have improved our capabilities of exploring, quantifying, and measuring the underlying mechanisms that guide human behavior using sophisticated neuroergonomic devices; and in turn, improve human performance and reduce human errors. In this thesis, multimodal approaches consisted of a self-report analysis, eye-tracking analysis, and functional near-infrared spectroscopy analysis were used to investigate how veteran local & en-route air traffic controllers carry out their operational tasks. Furthermore, the correlations among the cognitive workload and physiological measures (i.e. eye movement characteristics and brain activities) were investigated. Combining the results of these experiments, we can observe that the multimodal approaches show promise on exploring the underlying mechanisms of workload and human interaction in a complex, high-risk, and semantically rich environment. This is because cognitive workload can be considered as a multidimensional construct and different devices or approaches might be more effective in sensing changes in either the task difficulty or complexity. The results can be used to find ways to better train the novices

Investigation of Pilots\u27 Visual Entropy and Eye Fixations for Simulated Flights Consisted of Multiple Take-Offs and Landings

Eye movement characteristics might provide insights on pilots\u27 mental fatigue during prolonged flight. The visual entropy, eye fixation numbers, and eye fixation durations of ten novice pilots and ten expert pilots were analyzed for a four-hour simulated flight task consisting of four consecutive flight legs. Each flight leg lasted approximately one hour and contained five flight phases: takeoff, climb, cruise, descend, and landing. The pilots maneuvered the simulated B-52 aircraft following instrument flight rules (IFR) in a moderate-fidelity Microsoft Flight Simulator environment. Our results indicate that experts’ eye movement characteristics were significantly different from those of novices. In detail, novices\u27 eye movements were more random, produced longer eye fixation durations, and had fewer eye fixation numbers on the areas of interest (AOIs) than the experts. In addition, the repetitive task (i.e., four consecutive flights) significantly impacted the eye movement characteristics for both experts and novices. Visual entropy and eye fixation duration increased, while eye fixation numbers decreased for both groups as the repetition index increased. Finally, the flight phases also affected eye movement characteristics. The results show that both experts\u27 and novices\u27 visual entropies were relatively higher during climb, cruise, and descend phases, whereas those were relatively lower during the takeoff and landing phases. The present results provide a foundation for us to better understand the similarities and dissimilarities of eye movement characteristics between the experts and novices for a prolonged flight. Lastly, potential scaffolding training methods and pilot anomaly alerting systems, derived from such eye movements, are introduced

Using Eye Movement Data Visualization to Enhance Training of Air Traffic Controllers: A Dynamic Network Approach

The Federal Aviation Administration (FAA) forecasted substantial increase in the US air traffic volume creating a high demand in Air Traffic Control Specialists (ATCSs). Training times and passing rates for ATCSs might be improved if expert ATCSs’ eye movement (EM) characteristics can be utilized to support effective training. However, effective EM visualization is difficult for a dynamic task (e.g. aircraft conflict detection and mitigation) that includes interrogating multi-element targets that are dynamically moving, appearing, disappearing, and overlapping within a display. To address the issues, a dynamic network-based approach is introduced that integrates adapted visualizations (i.e. time-frame networks and normalized dot/bar plots) with measures used in network science (i.e. indegree, closeness, and betweenness) to provide in-depth EM analysis. The proposed approach was applied in an aircraft conflict task using a high-fidelity simulator; employing the use of veteran ATCSs and pseudo pilots. Results show that, ATCSs’ visual attention to multi-element dynamic targets can be effectively interpreted and supported through multiple evidences obtained from the various visualization and associated measures. In addition, we discovered that fewer eye fixation numbers or shorter eye fixation durations on a target may not necessarily indicate the target is less important when analyzing the flow of visual attention within a network. The results show promise in cohesively analyzing and visualizing various eye movement characteristics to better support training. 

A review of machine learning in scanpath analysis for passive gaze-based interaction

The scanpath is an important concept in eye tracking. It refers to a person's eye movements over a period of time, commonly represented as a series of alternating fixations and saccades. Machine learning has been increasingly used for the automatic interpretation of scanpaths over the past few years, particularly in research on passive gaze-based interaction, i.e., interfaces that implicitly observe and interpret human eye movements, with the goal of improving the interaction. This literature review investigates research on machine learning applications in scanpath analysis for passive gaze-based interaction between 2012 and 2022, starting from 2,425 publications and focussing on 77 publications. We provide insights on research domains and common learning tasks in passive gaze-based interaction and present common machine learning practices from data collection and preparation to model selection and evaluation. We discuss commonly followed practices and identify gaps and challenges, especially concerning emerging machine learning topics, to guide future research in the field

Scanning Signatures: A Graph Theoretical Model to Represent Visual Scanning Processes and A Proof of Concept Study in Biology Education

In this article we discuss, as a proof of concept, how a network model can be used to analyse gaze tracking data coming from a preliminary experiment carried out in a biodiversity education research project. We discuss the network model, a simple directed graph, used to represent the gaze tracking data in a way that is meaningful for the study of students’ biodiversity observations. Our network model can be thought of as a scanning signature of how a subject visually scans a scene. We provide a couple of examples of how it can be used to investigate the personal identification processes of a biologist and non-biologist when they are carrying out a task concerning the observation of species-specific characteristics of two bird species in the context of biology education research. We suggest that a scanning signature can be effectively used to compare the competencies of different persons and groups of people when they are making observations on specific areas of interests

Worth a Glance: Using Eye Movements to Investigate the Cognitive Neuroscience of Memory

Results of several investigations indicate that eye movements can reveal memory for elements of previous experience. These effects of memory on eye movement behavior can emerge very rapidly, changing the efficiency and even the nature of visual processing without appealing to verbal reports and without requiring conscious recollection. This aspect of eye movement based memory investigations is particularly useful when eye movement methods are used with special populations (e.g., young children, elderly individuals, and patients with severe amnesia), and also permits use of comparable paradigms in animals and humans, helping to bridge different memory literatures and permitting cross-species generalizations. Unique characteristics of eye movement methods have produced findings that challenge long-held views about the nature of memory, its organization in the brain, and its failures in special populations. Recently, eye movement methods have been successfully combined with neuroimaging techniques such as fMRI, single-unit recording, and magnetoencephalography, permitting more sophisticated investigations of memory. Ultimately, combined use of eye-tracking with neuropsychological and neuroimaging methods promises to provide a more comprehensive account of brain–behavior relationships and adheres to the “converging evidence” approach to cognitive neuroscience