A time series feature of variability to detect two types of boredom from motion capture of the head and shoulders

Boredom and disengagement metrics are crucial to the correctly timed implementation of adaptive interventions in interactive systems. psychological research suggests that boredom (which other HCI teams have been able to partially quantify with pressure-sensing chair mats) is actually a composite: lethargy and restlessness. Here we present an innovative approach to the measurement and recognition of these two kinds of boredom, based on motion capture and video analysis of changes in head and shoulder positions. Discrete, three-minute, computer-presented stimuli (games, quizzes, films and music) covering a spectrum from engaging to boring/disengaging were used to elicit changes in cognitive/emotional states in seated, healthy volunteers. Interaction with the stimuli occurred with a handheld trackball instead of a mouse, so movements were assumed to be non-instrumental. Our results include a feature (standard deviation of windowed ranges) that may be more specific to boredom than mean speed of head movement, and that could be implemented in computer vision algorithms for disengagement detection

Effects of Secondary Task Modality and Processing Code on Automation Trust and Utilization During Simulated Airline Luggage Screening

The purpose of this research was to examine the impact of environmental distractions on human trust and utilization of automation during the process of visual search. Participants performed a computer-simulated airline luggage screening task with the assistance of a 70% reliable automated decision aid (called DETECTOR) both with and without environmental distractions. The distraction was implemented as a secondary task in either a competing modality (visual) or non-competing modality (auditory). The secondary task processing code either competed with the luggage screening task (spatial code) or with the automation's textual directives (verbal code). We measured participants' system trust, perceived reliability of the system (when a target weapon was present and absent), compliance, reliance, and confidence when agreeing and disagreeing with the system under both distracted and undistracted conditions. Results revealed that system trust was lower in the visual-spatial and auditory-verbal conditions than in the visual-verbal and auditory-spatial conditions. Perceived reliability of the system (when the target was present) was significantly higher when the secondary task was visual rather than auditory. Compliance with the aid increased in all conditions except for the auditory-verbal condition, where it decreased. Similar to the pattern for trust, reliance on the automation was lower in the visual-spatial and auditory-verbal conditions than in the visual-verbal and auditory-spatial conditions. Confidence when agreeing with the system decreased with the addition of any kind of distraction; however, confidence when disagreeing increased with the addition of an auditory secondary task but decreased with the addition of a visual task. A model was developed to represent the research findings and demonstrate the relationship between secondary task modality, processing code, and automation use. Results suggest that the nature of environmental distractions influence interaction with automation via significant effects on trust and system utilization. These findings have implications for both automation design and operator training

Noninvasive Physiological Measures And Workload Transitions:an Investigation Of Thresholds Using Multiple Synchronized Sensors

The purpose of this study is to determine under what conditions multiple minimally intrusive physiological sensors can be used together and validly applied for use in areas which rely on adaptive systems including adaptive automation and augmented cognition. Specifically, this dissertation investigated the physiological transitions of operator state caused by changes in the level of taskload. Three questions were evaluated including (1) Do differences exist between physiological indicators when examined between levels of difficulty? (2) Are differences of physiological indicators (which may exist) between difficulty levels affected by spatial ability? (3) Which physiological indicators (if any) account for variation in performance on a spatial task with varying difficulty levels? The Modular Cognitive State Gauge model was presented and used to determine which basic physiological sensors (EEG, ECG, EDR and eye-tracking) could validly assess changes in the utilization of two-dimensional spatial resources required to perform a spatial ability dependent task. Thirty-six volunteers (20 female, 16 male) wore minimally invasive physiological sensing devices while executing a challenging computer based puzzle task. Specifically, participants were tested with two measures of spatial ability, received training, a practice session, an experimental trial and completed a subjective workload survey. The results of this experiment confirmed that participants with low spatial ability reported higher subjective workload and performed poorer when compared to those with high spatial ability. Additionally, there were significant changes for a majority of the physiological indicators between two difficulty levels and most importantly three measures (EEG, ECG and eye-tracking) were shown to account for variability in performance on the spatial task

Tunnélisation Attentionnelle : Définitions de métriques physiologiques et comportementales pour diagnostiquer la tunnélisation attentionnelle chez un opérateur humain

L’incapacité à détecter les alarmes visuelles est un problème crucial dans beaucoup de domaines comme l’automobile (Crundall, Underwood, et Chapman 1999) et l’aéronautique (Thomas et Wickens 2004). Différents modèles ont été proposés pour décrire ce phénomène comme étant une réduction du champ visuel, aussi connu sous le nom de « tunnélisation visuelle » (Williams 1985), ou à une dilution de l’attention visuelle (Crundall, Underwood, et Chapman 1999). D’autres études ont montré que, en fonction de la tâche expérimentale, la détection des stimuli visuels périphériques était détériorée si ces stimuli étaient placés entre 2,2 et 10 degrés du champ visuel fovéal (Plainis, Murray, et Chauhan 2001). Des auteurs s’appuyant sur le concept d’aveuglement attentionnel font l’hypothèse d’une explication purement attentionnelle du phénomène (Newby et Rock 1998) pour expliquer le rejet des stimuli inattendus dans le champ fovéal et ce malgré leur sallience (Simons et Chabris 1999). Ces auteurs postulent l’existence d’un processus d’inhibition sur le locus du stimulus inattendu, un mécanisme déjà décrit par Posner (1987). La « tunnélisation attentionnelle », un concept adjacent, a été opérationnalisé dans le contexte des facteurs humains par Wickens (Wickens 2005). Cet auteur qui propose de la définir comme « l’allocation de l’attention à un canal d’information particulier, à un diagnostic d’un jeu d’hypothèses ou à la réalisation d’une tâche dite objectif, pour une durée dépassant la durée optimale, étant donné les couts associés à la négligence des informations présentées par les autres canaux, ou à de nouvelles hypothèses non envisagées, ou encore à la non réalisation d’autres tâches. ». Des expériences réalisées en simulateur de vol démontrent que la tunnélisation attentionnelle peut conduire les pilotes à négliger des information critiques telles que des alarmes visuelles (Dehais, Tessier, et Chaudron 2003) et auditives (Dehais et al. 2010). Des travaux ont montré que des solutions existent pour lutter contre ce phénomène, telles que les contre-mesures cognitives (Dehais, Tessier, et Chaudron 2003), la régulation du niveau d’automatisation (Parasuraman et Wickens 2008) ou la modification du partage d’autorité (Dehais, Mercier, et Tessier 2009). Dès lors, un enjeu est de disposer de moyens de mesure pour détecter la tunnélisation attentionnelle dans le but d’adapter en temps réel l’interaction homme-machine selon les principes proposés précédemment

Using near infrared spectroscopy and heart rate variability to detect mental overload

Mental workload is a key factor influencing the occurrence of human error, especially during piloting and remotely operated vehicle (ROV) operations, where safety depends on the ability of pilots to act appropriately. In particular, excessively high or low mental workload can lead operators to neglect critical information. The objective of the present study is to investigate the potential of functional Near Infrared Spectroscopy (fNIRS) – a non-invasive method of measuring prefrontal cortex activity – in combination with measurements of heart rate variability (HRV), to predict mental workload during a simulated piloting task, with particular regard to task engagement and disengagement. Twelve volunteers performed a computer-based piloting task in which they were asked to follow a dynamic target with their aircraft, a task designed to replicate key cognitive demands associated with real life ROV operating tasks. In order to cover a wide range of mental workload levels, task difficulty was manipulated in terms of processing load and difficulty of control – two critical sources of workload associated with piloting and remotely operating a vehicle. Results show that both fNIRS and HRV are sensitive to different levels of mental workload; notably, lower prefrontal activation as well as a lower LF/HF ratio at the highest level of difficulty, suggest that these measures are suitable for mental overload detection. Moreover, these latter measurements point towards the existence of a quadratic model of mental workload

Mitigation of Conflicts with Automation: Use of Cognitive Countermeasures

Objective: The aim of this study was to empirically assess the efficacy of cognitive countermeasures based on the technique of information removal to enhance human operator attentional disengagement abilities when facing attentional tunneling. Background: Lessons learned from human factors studies suggest that conflict with automation leads to the degradation of operators’ performance by promoting excessive focusing on a single task to the detriment of the supervision of other critical parameters. Method: An experimental setup composed of a real unmanned ground vehicle and a ground station was developed to test the efficiency of the cognitive countermeasures.The scenario (with and without countermeasure) involved an authority conflict between the participants and the robot induced by a battery failure.The effects of the conflict and, in particular, the impact of cognitive countermeasures on the participants’ cognition and arousal were assessed through heart rate measurement and eye tracking techniques. Results: In the control group (i.e., no countermeasure), 8 out of 12 participants experienced attentional tunneling when facing the conflict, leading them to neglect the visual alarms displayed that would have helped them to understand the evolution of the tactical situation. Participants in the countermeasure group showed lower heart rates and enhanced attentional abilities, and 10 out of 11 participants made appropriate decisions. Conclusions: The use of cognitive countermeasures appeared to be an efficient means to mitigate excessive focus issues in the unmanned ground vehicle environment. Applications: The principle of cognitive counter- measures can be applied to a large domain of applications involving human operators interacting with critical systems

Formal Detection of Attentional Tunneling in Human Operator-Automation Interactions

The allocation of visual attention is a key factor for the humans when operating complex systems under time pressure with multiple information sources. In some situations, attentional tunneling is likely to appear and leads to excessive focus and poor decision making. In this study, we propose a formal approach to detect the occurrence of such an attentional impairment that is based on machine learning techniques. An experiment was conducted to provoke attentional tunneling during which psycho-physiological and oculomotor data from 23 participants were collected. Data from 18 participants were used to train an adaptive neuro-fuzzy inference system (ANFIS). From a machine learning point of view, the classification performance of the trained ANFIS proved the validity of this approach. Furthermore, the resulting classification rules were consistent with the attentional tunneling literature. Finally, the classifier was robust to detect attentional tunneling when performing over test data from four participants

Estimating pilots’ cognitive load from ocular parameters through simulation and in-flight studies

Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. This paper investigated use of eye gaze trackers in military aviation environment to automatically estimate pilot’s cognitive load from ocular parameters. We used a fixed base variable stability flight simulator with longitudinal tracking task and collected data from 14 military pilots. In a second study, we undertook three test flights with a BAES Hawk Trainer aircraft doing air to ground attack training missions and constant G level turn maneuvers up to +5G. Our study found that ocular parameters like rate of fixation is significantly different in different flying conditions and significantly correlate with altitude gradient during air to ground dive training task, normal load factor (G) of the aircraft during constant G level turn maneuvers and pilot’s control inceptor and tracking error in simulation tasks. Results from our studies can be used for real time estimation of pilots’ cognitive load, providing suitable warnings and alerts to the pilot in cockpit and training of military pilots on cognitive load management during operational missions

Reducing Human/Pilot Errors in Aviation Using Augmented Cognition and Automation Systems in Aircraft Cockpit

Human errors cause the majority of aviation accidents. Augmented cognition and automation systems enhance pilot performance by evaluating system limitations and flight precision and performance. This study examines the human-machine interface in cockpit design using the tenets of augmented cognition and automation systems theory in terms of task allocation, attentional resources, and situational awareness. The study compares how these principles apply to and interact with each other and with a human/pilot in a closed-loop system. We present a method for integrating augmented cognition systems into airplane flight management systems. We demonstrate systems enhancement with an experiment in which test pilots flew two simulated flights, once without and once with an augmented cognition system. We measured pilot and airplane performance, pilots’ situational awareness, workload management, pilots’ use of cockpit checklists, and flight precision along four axes: (1) altitude, (2) course, (3) radial/bearing and heading, and (4) airspeed

Predicting Cognitive Workload with Measures from Functional Near-Infrared Spectroscopy (fNIRS) and Heart Rate

The objective of this study was to assess low to high levels of Cognitive Workload by measuring heart rate and cortical blood flow in real-time. Four conditions were implemented into a within-subjects experimental design. Two conditions of difficulty and two conditions of trial order were used to illicit different levels of workload which will be analyzed with psychophysiological equipment. Functional Near-Infrared Spectroscopy (fNIRS) has become more prominent for measuring the blood oxygenation levels in the prefrontal cortex of individuals operating in hazardous work environments, students with learning disabilities, and in research for military training. This is due to the fNIR device being highly mobile, inexpensive, and able to produce a high-spatial resolution of the dorsolateral prefrontal cortex during executive functioning. Heart Rate will be measured by an Electrocardiogram, which will be used in concordance with fNIR oxygenation levels to predict if an individual is in a condition that produces low or high mental workload. Successfully utilizing heart rate and blood oxygenation data as predictors of cognitive workload may validate implementing multiple physiological devices together in real-time and may be a more accurate solution for preventing excessive workload