Mitigation of Conflicts with Automation: Use of Cognitive Countermeasures

Objective: The aim of this study was to empirically assess the efficacy of cognitive countermeasures based on the technique of information removal to enhance human operator attentional disengagement abilities when facing attentional tunneling. Background: Lessons learned from human factors studies suggest that conflict with automation leads to the degradation of operators’ performance by promoting excessive focusing on a single task to the detriment of the supervision of other critical parameters. Method: An experimental setup composed of a real unmanned ground vehicle and a ground station was developed to test the efficiency of the cognitive countermeasures.The scenario (with and without countermeasure) involved an authority conflict between the participants and the robot induced by a battery failure.The effects of the conflict and, in particular, the impact of cognitive countermeasures on the participants’ cognition and arousal were assessed through heart rate measurement and eye tracking techniques. Results: In the control group (i.e., no countermeasure), 8 out of 12 participants experienced attentional tunneling when facing the conflict, leading them to neglect the visual alarms displayed that would have helped them to understand the evolution of the tactical situation. Participants in the countermeasure group showed lower heart rates and enhanced attentional abilities, and 10 out of 11 participants made appropriate decisions. Conclusions: The use of cognitive countermeasures appeared to be an efficient means to mitigate excessive focus issues in the unmanned ground vehicle environment. Applications: The principle of cognitive counter- measures can be applied to a large domain of applications involving human operators interacting with critical systems

Cognitive conflict in human–automation interactions: A psychophysiological study

The review of literature in sociology and distributed artificial intelligence reveals that the occurrence of conflict is a remarkable precursor to the disruption of multi-agent systems. The study of this concept could be applied to human factors concerns, as man-system conflict appears to provoke perseveration behavior and to degrade attentional abilities with a trend to excessive focus. Once entangled in such conflicts, the human operator will do anything to succeed in his current goal even if it jeopardizes the mission. In order to confirm these findings, an experimental setup, composed of a real unmanned ground vehicle, a ground station is developed. A scenario involving an authority conflict between the partici- pants and the robot is proposed. Analysis of the effects of the conflict on the participants’ cognition and arousal is assessed through heart-rate measurement (reflecting stress level) and eye-tracking techniques (index of attentional focus). Our results clearly show that the occurrence of the conflict leads to perseveration behavior and can induce higher heart rate as well as excessive attentional focus. These results are discussed in terms of task commitment issues and increased arousal. Moreover, our results suggest that individual differences may predict susceptibility to perseveration behavior

Effects of the audiovisual conflict on auditory early processes

Auditory alarm misperception is one of the critical events that lead aircraft pilots to an erroneous flying decision. The rarity of these alarmsassociatedwith their possible unreliabilitymay play a role in thismisperception. In order to investigate this hypothesis, we manipulated both audiovisual conflict and sound rarity in a simplified landing task. Behavioral data and event related potentials (ERPs) of thirteen healthy participants were analyzed. We found that the presentation of a rare auditory signal (i.e. an alarm), incongruent with visual information, led to a smaller amplitude of the auditory N100 (i.e. less negative) compared to the condition in which both signals were congruent.Moreover, the incongruity between the visual information and the rare sound did not significantly affect reaction times, suggesting that the rare sound was neglected. We propose that the lower N100 amplitude reflects an early visual-to-auditory gating that depends on the rarity of the sound. In complex aircraft environments, this early effect might be partly responsible for auditory alarm insensitivity. Our results provide a new basis for future aeronautic studies and the development of countermeasures

Anticipating human error before it happens: Towards a psychophysiological model for online prediction of mental workload

Mental workload is a key factor influencing the occurrence of human error; specifically in remotely-operated vehicle operations. Both low and high mental workload has been found to disrupt performance in a nonlinear fashion at a given task; however, research that has attempted to predict individual mental workload has met with little success. The objective of the present study is to investigate the potential of the dual-task paradigm and prefrontal cortex oxygenation as online measures of mental workload. Subjects performed a computerized object tracking task in which they had to follow a dynamic target with their aircraft. Task difficulty was manipulated in terms of processing load and difficulty of control: two critical sources of workload associated with remotely operating a vehicle. Mental workload was assessed by a secondary concurrent time production task and a functional near infrared spectrometer. Results show that the effects of task difficulty differ across measures of mental workload. This pattern of behavioural and neurophysiologic results suggests that the empirically-based selection of an appropriate secondary task for the measure of mental workload is critical as its sensitivity may vary considerably depending on task factors

Frequency analysis of a task-evoked pupillary response: Luminance-independent measure of mental effort

The pupil diameter is a widely studied cognitive load measure, which, despite its convenience for non-intrusive operator state monitoring in complex environments, is still not available for in situ measurements because of numerous methodological limitations. The most important of these limitations is the influence of the pupillary light reflex. Hence, there is the need of providing a pupil-based cognitive load measure independent of light conditions. In this paper, we present a promising technique of pupillary signal analysis resulting in luminance-independent measure of mental effort that could be used in real-time without a priori on luminous conditions. Twenty two participants performed a short-term memory task under different screen luminance conditions. Our results showed that the amplitude of pupil dilation due to load on memory is luminance-dependent with higher amplitude corresponding to low-luminance condition. Furthermore, our experimentation showed that the load on memory and luminance factors express themselves differently according to frequency. Therefore, as our statistical analysis revealed, the ratio between low (0–1.6 Hz) and high frequency (1.6–4 Hz) bands (LF/HF ratio) of power spectral densities of pupillary signal is sensitive to the cognitive load but not to luminance. This result is promising for the measurements of load on memory in ecological settings

Towards a neuroergonomics approach to understand inattentional deafness in aviation

Aviation safety analyses often reveal that pilots persist in erroneous decision-making despite auditory warning. Classically, the lack of response to alarms is attributed to pilots ignoring consciously such warnings due to decision biases or risk taking. However these explanations do not fully account for the misperception of critical auditory alarms as observed in flight simulators (Dehais, Tessier, Christophe, & Reuzeau, 2009). A relevant approach is to consider the concept of inattentional deafness, that is the absence of reaction to auditory stimuli (Macdonald & Lavie, 2011). Since high mental demand is a key element of flying, it is likely that the important information processing of instruments could interfere with the concurrent appraisal of unexpected auditory alarms and leads to inattentional deafness. To investigate this hypothesis, we adopted a neuroergonomics approach (Parasuraman, & Rizzo, 2007) that promotes the use of brain imaging techniques to understand the neural mechanisms of human error. Indeed, we recorded electrophysiological measurements (EEG) while participants were supervising an automated landing sequence by considering both visual and auditory signals. Our results revealed evidence of an early visual-to-auditory gating mechanism that occurs when visual parameters (“land”) were contradictory to the auditory alarm (“go around”). This mechanism attenuates early auditory processing (N100) and could partly explain the “inattentional deafness” in aeronautics (Scannella, Causse, Chauveau, Pastor, & Dehais, 2013). In a second experiment, pilots were required to perform a landing decision task based on the analysis of visual indicators while continuous EEG measurements were performed under different load conditions. During the task, a tone was presented, either standard, which participants were told to ignore, or deviant (“the alarm”) which participants were told to report. Preliminary behavioral results showed that up to 30% of deviant sounds were not detected during the high load condition. The analysis of the event related potential showed that a drastic diminution of the late auditory component (P300) amplitude was concomitant with the occurrence of inattentional deafness (Giraudet, St-Louis, & Causse, 2012). This shows that some sounds may remain unnoticed despite the fact that they are perfectly audible. These protocols were adapted and replicated in a motion flight simulator and showed that critical task load may totally impair auditory alarm perception (Dehais, Causse, Vachon, Regis, Menant, & Tremblay, 2013). These three recent experiments were the very firsts to demonstrate the existence of inattentional deafness to aircraft alarms and provide a methodology as well as objective measurements to assess the efficiency of future auditory alarms

High Rate of Inattentional Deafness in Simulated Air Traffic Control Tasks

The Air Traffic Control (ATC) environment is complex and safety-critical; operators work in dynamic situations and must make high-risk decisions under stress and temporal pressure. The high perceptual load involved in ATC means that controllers’ attention must be shared between several subtasks, with few or no remaining attentional capacity for processing information that is not related directly to the focal task. In this kind of situation, the likelihood of a controller failing to become aware of an auditory alarm, i.e. inattentional deafness, is high. We designed an ecological ATC thanks to the simulation environment called the “LABY” microworld. Twenty participants were required to guide one (low cognitive load) or two planes (high cognitive load) around a given route, while dealing with visual notifications relating to peripheral aircrafts. During the task, participants were played either standard tones which they were told to ignore, or deviant tones (“the alarm”, probability = 0.20) which they were told to report (20 alarms per scenario). We hypothesized that the detection rate of auditory alarms will decrease with cognitive workload. In order to explore this possibility, Behavioral results showed that 28.8% of alarms were not reported when guiding one plane, and up to 46.2% when guiding two planes (high load). The cognitive load increase led to a reduced visual notification detection rate, but the performance to guiding the central aircrafts was maintained, as well as the reaction times to report auditory alarms when perceived. This high rate of inattentional deafness is essential to further physiological studies on alarm omission in aeronautics, such as ERP or eye movement analysis. Potential applications are related to the integrative online detection and prevention of alarm omission, and the online measurement of workload in ecological situation

The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures

The Air Traffic Control (ATC) environment is complex and safety-critical. Whilst exchanging information with pilots, controllers must also be alert to visual notifications displayed on the radar screen (e.g., warning which indicates a loss of minimum separation between aircraft). Under the assumption that attentional resources are shared between vision and hearing, the visual interface design may also impact the ability to process these auditory stimuli. Using a simulated ATC task, we compared the behavioral and neural responses to two different visual notification designs—the operational alarm that involves blinking colored “ALRT” displayed around the label of the notified plane (“Color-Blink”), and the more salient alarm involving the same blinking text plus four moving yellow chevrons (“Box-Animation”). Participants performed a concurrent auditory task with the requirement to react to rare pitch tones. P300 from the occurrence of the tones was taken as an indicator of remaining attentional resources. Participants who were presented with the more salient visual design showed better accuracy than the group with the suboptimal operational design. On a physiological level, auditory P300 amplitude in the former group was greater than that observed in the latter group. One potential explanation is that the enhanced visual design freed up attentional resources which, in turn, improved the cerebral processing of the auditory stimuli. These results suggest that P300 amplitude can be used as a valid estimation of the efficiency of interface designs, and of cognitive load more generally

Momentary lapse of control: A cognitive continuum approach to understanding and mitigating perseveration in human error

Everyday complex and stressful real-life situations can overwhelm the human brain to an extent that the person is no longer able to accurately evaluate the situation and persists in irrational actions or strategies. Safety analyses reveal that such perseverative behavior is exhibited by operators in many critical domains, which can lead to potentially fatal incidents. There are neuroimaging evidences of changes in healthy brain functioning when engaged in non-adaptive behaviors that are akin to executive deficits such as perseveration shown in patients with brain lesion. In this respect, we suggest a cognitive continuum whereby stressors can render the healthy brain temporarily impaired. We show that the dorsolateral prefrontal cortex is a key structure for executive and attentional control whereby any transient (stressors, neurostimulation) or permanent (lesion) impairment compromises adaptive behavior. Using this neuropsychological insight, we discuss solutions involving training, neurostimulation, and the design of cognitive countermeasures for mitigating perseveration

N270 sensitivity to conflict strength and working memory: A combined ERP and sLORETA study

The event-related potential N270 component is known to be an electrophysiological marker of the supramodal conflict processing. However little is know about the factors that may modulate its amplitude. In particular, among all studies that have investigated the N270, little or no control of the conflict strength and of the load in working memory have been done leaving a lack in the understanding of this component. We designed a spatial audiovisual conflict task with simultaneous target and cross-modal distractor to evaluate the N270 sensitivity to the conflict strength (i.e., visual target with auditory distractor or auditory target with visual distractor) and the load in working memory (goal task maintenance with frequent change in the target modality). In a first session, participants had to focus on one modality for the target position to be considered (left-hand or right-hand) while the distractor could be at the same side (compatible) or at opposite side (incompatible). In a second session, we used the same set of stimuli as in the first session with an additional distinct auditory signal that clued the participants to frequently switch between the auditory and the visual targets. We found that (1) reaction times and N270 amplitudes for conflicting situations were larger within the auditory target condition compared to the visual one, (2) the increase in target maintenance effort led to equivalent increase of both reaction times and N270 amplitudes within all conditions and (3) the right dorsolateral prefrontal cortex current density was higher for both conflicting and active maintenance of the target situations. These results provide new evidence that the N270 component is an electrophysiological marker of the supramodal conflict processing that is sensitive to the conflict strength and that conflict processing and active maintenance of the task goal are two functions of a common executive attention system