Prediction of Cognitive States During Flight Simulation Using Multimodal Psychophysiological Sensing

The Commercial Aviation Safety Team found the majority of recent international commercial aviation accidents attributable to loss of control inflight involved flight crew loss of airplane state awareness (ASA), and distraction was involved in all of them. Research on attention-related human performance limiting states (AHPLS) such as channelized attention, diverted attention, startle/surprise, and confirmation bias, has been recommended in a Safety Enhancement (SE) entitled "Training for Attention Management." To accomplish the detection of such cognitive and psychophysiological states, a broad suite of sensors was implemented to simultaneously measure their physiological markers during a high fidelity flight simulation human subject study. Twenty-four pilot participants were asked to wear the sensors while they performed benchmark tasks and motion-based flight scenarios designed to induce AHPLS. Pattern classification was employed to predict the occurrence of AHPLS during flight simulation also designed to induce those states. Classifier training data were collected during performance of the benchmark tasks. Multimodal classification was performed, using pre-processed electroencephalography, galvanic skin response, electrocardiogram, and respiration signals as input features. A combination of one, some or all modalities were used. Extreme gradient boosting, random forest and two support vector machine classifiers were implemented. The best accuracy for each modality-classifier combination is reported. Results using a select set of features and using the full set of available features are presented. Further, results are presented for training one classifier with the combined features and for training multiple classifiers with features from each modality separately. Using the select set of features and combined training, multistate prediction accuracy averaged 0.64 +/- 0.14 across thirteen participants and was significantly higher than that for the separate training case. These results support the goal of demonstrating simultaneous real-time classification of multiple states using multiple sensing modalities in high fidelity flight simulators. This detection is intended to support and inform training methods under development to mitigate the loss of ASA and thus reduce accidents and incidents

Exploring Cognitive States: Methods for Detecting Physiological Temporal Fingerprints

Cognitive state detection and its relationship to observable physiologically telemetry has been utilized for many human-machine and human-cybernetic applications. This paper aims at understanding and addressing if there are unique psychophysiological patterns over time, a physiological temporal fingerprint, that is associated with specific cognitive states. This preliminary work involves commercial airline pilots completing experimental benchmark task inductions of three cognitive states: 1) Channelized Attention (CA); 2) High Workload (HW); and 3) Low Workload (LW). We approach this objective by modeling these "fingerprints" through the use of Hidden Markov Models and Entropy analysis to evaluate if the transitions over time are complex or rhythmic/predictable by nature. Our results indicate that cognitive states do have unique complexity of physiological sequences that are statistically different from other cognitive states. More specifically, CA has a significantly higher temporal psychophysiological complexity than HW and LW in EEG and ECG telemetry signals. With regards to respiration telemetry, CA has a lower temporal psychophysiological complexity than HW and LW. Through our preliminary work, addressing this unique underpinning can inform whether these underlying dynamics can be utilized to understand how humans transition between cognitive states and for improved detection of cognitive states

Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation

Attention-related human performance limiting states (AHPLS) can cause pilots to lose airplane state awareness (ASA), and their detection is important to improving commercial aviation safety. The Commercial Aviation Safety Team found that the majority of recent international commercial aviation accidents attributable to loss of control inflight involved flight crew loss of airplane state awareness, and that distraction of various forms was involved in all of them. Research on AHPLS, including channelized attention, diverted attention, startle / surprise, and confirmation bias, has been recommended in a Safety Enhancement (SE) entitled "Training for Attention Management." To accomplish the detection of such cognitive and psychophysiological states, a broad suite of sensors has been implemented to simultaneously measure their physiological markers during high fidelity flight simulation human subject studies. Pilot participants were asked to perform benchmark tasks and experimental flight scenarios designed to induce AHPLS. Pattern classification was employed to distinguish the AHPLS induced by the benchmark tasks. Unimodal classification using pre-processed electroencephalography (EEG) signals as input features to extreme gradient boosting, random forest and deep neural network multiclass classifiers was implemented. Multi-modal classification using galvanic skin response (GSR) in addition to the same EEG signals and using the same types of classifiers produced increased accuracy with respect to the unimodal case (90 percent vs. 86 percent), although only via the deep neural network classifier. These initial results are a first step toward the goal of demonstrating simultaneous real time classification of multiple states using multiple sensing modalities in high-fidelity flight simulators. This detection is intended to support and inform training methods under development to mitigate the loss of ASA and thus reduce accidents and incidents

Using cardiovascular measures for adaptive automation

Adaptieve automatisering betekent zoveel als een technologisch ingerichte werkomgeving die zich aanpast aan de gebruiker. Binnen het onderzoeksgebied adaptieve automatisering worden technologische systemen ontwikkeld die flexibel zijn en zich kunnen aanpassen aan de specifieke behoeften en eisen van de individuele mens. Het idee is dat het hele systeem (mens en machine samen) het best functioneert als de werklast van de mens op een optimaal niveau wordt gehouden. Enerzijds moet voorkomen worden dat de hele taak geautomatiseerd wordt, zodat de menselijke bestuurder niet meer snel en adequaat kan ingrijpen als er iets mis gaat met het geautomatiseerde systeem. Anderzijds moet voorkomen worden dat oververmoeidheid en concentratieverlies optreden door een voortdurende (te) hoge werklast . Meten is weten. Een belangrijke eerste voorwaarde om de werklast op een adequaat niveau te houden is deze te kunnen meten. Een geschikte manier om mentale belasting te bepalen is met behulp van fysiologische methoden, in het bijzonder met behulp van hartslag, bloeddruk en ademhalingsmaten. In de literatuur is er nog enige onduidelijkheid over de relatie tussen mentale inspanning en de psychofysiologische reacties daarop. In dit proefschrift worden een aantal onduidelijkheden over deze relatie verklaard aan de hand van een onderscheid dat gemaakt wordt tussen toestand-gerelateerde (of compensatoire) effecten en korte termijn effecten die naar verwachting meer rechtstreeks verband houden met veranderingen in de taakeisen. De toestand gerelateerde effecten zijn pogingen van het lichaam om te herstellen van langdurige inspanning en terug te keren naar een evenwichtssituatie. De korte termijn effecten ontstaan doordat het lichaam reageert op momentane verschillen in werklast waar het lichaam energie voor moet leveren. In het beschreven onderzoek is een aantal experimenten in een gesimuleerde ambulancemeldkamer en in een rijsimulator uitgevoerd op basis waarvan een nieuwe methode is ontwikkeld die meer inzicht geeft in de momentane werklast van de mens. Deze methode leent zich ook uitstekend voor gebruik bij adaptieve automatisering. In het kort kan worden gezegd dat de toestandseffecten, met andere woorden het herstellen van inspanning, de effecten die door de huidige werklast worden veroorzaakt, in veel werksituaties overschaduwen. Hierdoor zijn de directe effecten van werklastveranderingen vaak niet zichtbaar. De oplossing die in dit proefschrift wordt beschreven is een keuze voor korte termijn cardiovasculaire maten. Daarmee zijn verschillen in hartslag, hartslagvariabiliteit en andere cardiovasculaire maten nog steeds zichtbaar in de korte termijn respons-patronen, ondanks de aanwezigheid van de toestandseffecten. De korte termijn analyse berust op een tijd-frequentie methode waarbij de variabiliteit van de cardiovasculaire maten wordt berekend in tijdsegmenten van 30 seconden. Deze methode is getoetst in experimenten die opnieuw in de ambulance meldkamer simulatie en rijsimulator zijn uitgevoerd. De conclusie is dat de verschillen in gevonden effecten in eerder beschreven onderzoek m.b.t. werkbelastingmaten in belangrijke mate verklaard kunnen worden uit de compenserende werking van het bloeddrukregulatie systeem. De compenserende werking van het bloeddrukregulatiesysteem nivelleert de directe effecten van mentale inspanning en maakt daarmee de interpretatie van de werklast-effecten op de cardiovasculaire maten moeilijker. Dit geldt met name voor de hartslag en hartslagvariabiliteit. Door onderscheid te maken tussen effecten die direct gerelateerd zijn aan de taakeisen en effecten die veroorzaakt worden door het bloeddruk regulatiesysteem, krijgen we meer zicht op de echte taakgerelateerde mentale inspanning. Dit heeft geleid tot de ontwikkeling van betere mentale werklast-maten die bruikbaar zijn bij adaptieve automatisering

Highly Automated Driving, Secondary Task Performance, and Driver State

Objective: a driving simulator study compared the effect of changes in workload on performance in manual and highly automated driving. Changes in driver state were also observed by examining variations in blink patterns. Background: With the addition of a greater number of advanced driver assistance systems in vehicles, the driver’s role is likely to alter in the future from an operator in manual driving to a supervisor of highly automated cars. Understanding the implications of such advancements on drivers and road safety is important. Method: a total of 50 participants were recruited for this study and drove the simulator in both manual and highly automated mode. As well as comparing the effect of adjustments in driving-related workload on performance, the effect of a secondary Twenty Questions Task was also investigated. Results: in the absence of the secondary task, drivers’ response to critical incidents was similar in manual and highly automated driving conditions. The worst performance was observed when drivers were required to regain control of driving in the automated mode while distracted by the secondary task. Blink frequency patterns were more consistent for manual than automated driving but were generally suppressed during conditions of high workload. Conclusion: highly automated driving did not have a deleterious effect on driver performance, when attention was not diverted to the distracting secondary task. Application: as the number of systems implemented in cars increases, an understanding of the implications of such automation on drivers’ situation awareness, workload, and ability to remain engaged with the driving task is important

Fuzzy Filtering: A Mathematical Theory and Applications in Life Science

Non

Tune in to your emotions: a robust personalized affective music player

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application

Noninvasive Physiological Measures And Workload Transitions:an Investigation Of Thresholds Using Multiple Synchronized Sensors

The purpose of this study is to determine under what conditions multiple minimally intrusive physiological sensors can be used together and validly applied for use in areas which rely on adaptive systems including adaptive automation and augmented cognition. Specifically, this dissertation investigated the physiological transitions of operator state caused by changes in the level of taskload. Three questions were evaluated including (1) Do differences exist between physiological indicators when examined between levels of difficulty? (2) Are differences of physiological indicators (which may exist) between difficulty levels affected by spatial ability? (3) Which physiological indicators (if any) account for variation in performance on a spatial task with varying difficulty levels? The Modular Cognitive State Gauge model was presented and used to determine which basic physiological sensors (EEG, ECG, EDR and eye-tracking) could validly assess changes in the utilization of two-dimensional spatial resources required to perform a spatial ability dependent task. Thirty-six volunteers (20 female, 16 male) wore minimally invasive physiological sensing devices while executing a challenging computer based puzzle task. Specifically, participants were tested with two measures of spatial ability, received training, a practice session, an experimental trial and completed a subjective workload survey. The results of this experiment confirmed that participants with low spatial ability reported higher subjective workload and performed poorer when compared to those with high spatial ability. Additionally, there were significant changes for a majority of the physiological indicators between two difficulty levels and most importantly three measures (EEG, ECG and eye-tracking) were shown to account for variability in performance on the spatial task

The use of multi-attribute task battery in mental workload studies: A scoping review

Multi-Attribute Task Battery (MATB) is a software that has been arguably utilized in many ergonomics/human factors studies, including in the topic of mental workload. However, the use of this well-known program in diverse investigations has not yet been systematically tracked. Furthermore, it may be argued that a critical appraisal of MATB is urgently needed so that future researchers and users can take several crucial factors into account when planning a study or experiment using MATB. The aim of this paper is to comprehensively identify and review the use of MATB software in published studies. This aim might be accomplished by achieving two goals: (1) systematic discovery of published papers in literature databases and (2) categorization of research according to pertinent topics. In this paper, thirty-one articles were included for analysis after carefully screening for their eligibility. Our scoping review finds that MATB is a beneficial program for creating multitasking environments in general, with aviation being the area where it has been used the most. The program was also extensively used for studies on mental workload, especially by producing various stimuli that ultimately result in varying degrees of task demand or difficulty. Moreover, to successfully use MATB, researchers must be aware of a few operational issues and criticisms