Using Eye Movement Data Visualization to Enhance Training of Air Traffic Controllers: A Dynamic Network Approach

The Federal Aviation Administration (FAA) forecasted substantial increase in the US air traffic volume creating a high demand in Air Traffic Control Specialists (ATCSs). Training times and passing rates for ATCSs might be improved if expert ATCSs’ eye movement (EM) characteristics can be utilized to support effective training. However, effective EM visualization is difficult for a dynamic task (e.g. aircraft conflict detection and mitigation) that includes interrogating multi-element targets that are dynamically moving, appearing, disappearing, and overlapping within a display. To address the issues, a dynamic network-based approach is introduced that integrates adapted visualizations (i.e. time-frame networks and normalized dot/bar plots) with measures used in network science (i.e. indegree, closeness, and betweenness) to provide in-depth EM analysis. The proposed approach was applied in an aircraft conflict task using a high-fidelity simulator; employing the use of veteran ATCSs and pseudo pilots. Results show that, ATCSs’ visual attention to multi-element dynamic targets can be effectively interpreted and supported through multiple evidences obtained from the various visualization and associated measures. In addition, we discovered that fewer eye fixation numbers or shorter eye fixation durations on a target may not necessarily indicate the target is less important when analyzing the flow of visual attention within a network. The results show promise in cohesively analyzing and visualizing various eye movement characteristics to better support training. 

Development of methodologies to analyze and visualize air traffic controllers’ visual scanning strategies

The Federal Aviation Administration (FAA) estimates a 60 million air traffic volume by 2040. However, the available workforce of expert air traffic controllers (ATCs) might not be sufficient to manage this anticipated high traffic volume. Thus, to maintain the same safety standard and service level for air travel, more ATCs will need to be trained quickly. Previous research shows eye tracking technology can be used to enhance the training of the ATC’s by reducing their false alarm rate, thereby helping to mitigate the impact of increasing demand. Methods need to be developed to better understand experts’ eye movement (EM) data so as to incorporate them effectively in ATCs’ training process. However, it’s challenging to analyze ATCs’ EM data for several factors: (i) aircraft representation on radar display (i.e. targets) are dynamic, as their shape and position changes with time; (ii) raw EM data is very complex to visualize, even for the meaningful small duration (e.g. task completion time of 1 min); (iii) in the absence of any predefined order of visual scanning, each ATC employ a variety of scanning strategies to manage traffic, making it challenging to extract relevant patterns that can be taught. To address these aforementioned issues, a threefold framework was developed: (i) a dynamic network-based approach that can map expert ATCs’ EM data to dynamic targets, enabling the representation of visual scanning strategy evolution with time; (ii) a novel density-based clustering method to reduce the inherent complexity of ATCs’ raw EM data to enhance its visualization; (iii) a new modified n-gram based similarity analysis method, to evaluate the consistency and similarity of visual scanning strategies among experts. Two different experiments were conducted at the FAA Civil Aerospace Medical Institute in Oklahoma City, where EM data of 15 veteran ATCs’ (> 20 years of experience) were collected using eye trackers (Facelab and Tobii eye trackers), while they were controlling a high-fidelity simulated air traffic. The first experiment involved en-route traffic scenario (with aircraft above 18,000 feet) and the second experiment consisted of airport tower traffic (aircraft within 30 miles radius from an airport). The dynamic network analysis showed three important results: (i) it can be used to effectively represent which are the important targets and how their significance evolves over time, (ii) in dynamic scenarios, having targets having variable time on display, traditional target importance measure (i.e. the number of eye fixations and duration) can be misleading, and (iii) importance measures derived from the network-based approach (e.g. closeness, betweenness) can be used to understand how ATCs’ visual attention moves between targets. The result from the density-based clustering method shows that by controlling its two parameter values(i.e. spatial and temporal approximation), the visualization of the raw EM data can be substantially simplified. This approximate representation can be used for better training purpose where expert ATC’s visual scanning strategy can be visualized with reduced complexity, thereby enhancing the understanding of novices while maintaining its significant pattern (key for visual pattern mining). Moreover, the model parameters enable the decision-maker to incorporate context-dependent factors by adjusting the spatial (in pixel) and temporal (in milliseconds) thresholds used for the visual scanning approximation. The modified n-gram approach allows for twofold similarity analysis of EM data: (i) detecting similar EM patterns due to exact sequential match in which the targets are focused and/or grouped together visually because of several eye fixation transitions among them, and (ii) unearth similar visual scanning behavior which is otherwise small perturbed version of each other that arise as a result of idiosyncrasies of ATCs. Thus, this method is more robust compared to other prevalent approaches which employ strict definitions for similarity that are difficult to empirically observe in real-life scenarios. To summarize, the three methods developed allow us to apply a comprehensible framework to understand the evolving nature of the visual scanning strategy in complex environments (e.g. air traffic control task) by: (i) by identifying target importance & their evolution; (ii) simplifying visualizing of complex EM strategy for easier comprehension; (iii) evaluating similarity among various visual scanning strategies in dynamic scenarios