In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations

Neuroergonomics for aviation

Neuroergonomics is the study of how the brain functions in real-world situations with the goal of developing technology to enhance human performance. Neuroergonomics constitutes a paradigm shift away from the standard reductionist approach to neuroscience. The neuroergonomic approach maintains that an understanding of neural processes underlying human behavior can best be understood by investigating the underlying interacting brain networks in the context of carrying out various real-world tasks under investigation, rather than under reduced isolated conditions that only occur in the laboratory. In this chapter we discuss why aerospace cerebral experimental sciences (ACES) is an ideal paradigm to implement this neuroergonomic approach. By using a combination of high resolution and lower resolution portable brain imaging techniques as well as non-invasive brain stimulation the goal of ACES is to determine brain processes underlying complex behavior during aviation and space operations such that neuroergonomic technology can be developed to improve human performance

Applications of Optical Brain Imaging Methods in Aviation Neuroergonomics

Pilotların, insansız hava aracı operatörlerinin, hava trafik kontrolörlerinin eğitim ve uçuş faaliyetleri sırasında bilişsel durumlarının takibini sağlayacak nesnel yöntemlerin geliştirilmesi uçuş emniyetinin sağlanması, eğitim süreçlerinin optimizasyonu ve yenilikçi insan-makine arayüzlerinin tasarımı bakımından kritik önem taşımaktadır. İşlevsel Yakın-Kızılötesi Tayfölçümü (functional near infrared spectroscopy – fNIRS) optik beyin görüntüleme teknolojisi gibi saha kullanımına uygun, portatif ve güvenilir nörofizyolojik ölçüm yöntemleri bu ihtiyaçlara yönelik bazı önemli avantajlar sunmaktadır. Bu derlemede fNIRS teknolojisinin dayandığı bilimsel temeller ve bu teknolojiyle gerçekleştirilmiş pilot/operatör bilişsel işyükü takibi, kontrol arayüzü değerlendirmesi, G-LoC/hipoksi kestirimi gibi öncü havacılık uygulamalarından örnekler sunularak fNIRS yönteminin havacılık tıbbı ve ergonomisi alanları için sunduğu imkanların özetlenmesi amaçlanmıştır.The development of objective methods that enable monitoring of the cognitive status of pilots, unmanned aerial vehicle operators, and air traffic controllers is critically important in aviation for improving flight safety, optimizing pilot/operator training and developing innovative man-machine interfaces. Functional near-infrared spectroscopy (fNIRS) optical brain imaging technology offers significant advantages for this purpose by providing portable, rugged sensors that can be employed in the field to monitor neurophysiological markers during flight operations. This article reviews studies that employ fNIRS technology for cognitive workload assessment, operator interface evaluation, and G-LoC/hypoxia prediction in aviation to document the potential of neurophysiological measurement modalities like fNIRS for aviation medicine and ergonomics

Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs

Mixed-reality for unmanned aerial vehicle operations in near earth environments

Future applications will bring unmanned aerial vehicles (UAVs) to near Earth environments such as urban areas, causing a change in the way UAVs are currently operated. Of concern is that UAV accidents still occur at a much higher rate than the accident rate for commercial airliners. A number of these accidents can be attributed to a UAV pilot's low situation awareness (SA) due to the limitations of UAV operating interfaces. The main limitation is the physical separation between the vehicle and the pilot. This eliminates any motion and exteroceptive sensory feedback to the pilot. These limitation on top of a small eld of view from the onboard camera results in low SA, making near Earth operations di cult and dangerous. Autonomy has been proposed as a solution for near Earth tasks but state of the art arti cial intelligence still requires very structured and well de ned goals to allow safe autonomous operations. Therefore, there is a need to better train pilots to operate UAVs in near Earth environments and to augment their performance for increased safety and minimization of accidents.In this work, simulation software, motion platform technology, and UAV sensor suites were integrated to produce mixed-reality systems that address current limitations of UAV piloting interfaces. The mixed reality de nition is extended in this work to encompass not only the visual aspects but to also include a motion aspect. A training and evaluation system for UAV operations in near Earth environments was developed. Modi cations were made to ight simulator software to recreate current UAV operating modalities (internal and external). The training and evaluation system has been combined with Drexel's Sensor Integrated Systems Test Rig (SISTR) to allow simulated missions while incorporating real world environmental e ects andUAV sensor hardware.To address the lack of motion feedback to a UAV pilot, a system was developed that integrates a motion simulator into UAV operations. The system is designed such that during ight, the angular rate of a UAV is captured by an onboard inertial measurement unit (IMU) and is relayed to a pilot controlling the vehicle from inside the motion simulator.Efforts to further increase pilot SA led to the development of a mixed reality chase view piloting interface. Chase view is similar to a view of being towed behind the aircraft. It combines real world onboard camera images with a virtual representation of the vehicle and the surrounding operating environment. A series of UAV piloting experiments were performed using the training and evaluation systems described earlier. Subjects' behavioral performance while using the onboard camera view and the mixed reality chase view interface during missions was analyzed. Subjects' cognitive workload during missions was also assessed using subjective measures such as NASA task load index and non-subjective brain activity measurements using a functional Infrared Spectroscopy (fNIR) system. Behavioral analysis showed that the chase view interface improved pilot performance in near Earth ights and increased their situational awareness. fNIR analysis showed that a subjects cognitive workload was signi cantly less while using the chase view interface. Real world ight tests were conducted in a near Earth environment with buildings and obstacles to evaluate the chase view interface with real world data. The interface performed very well with real world, real time data in close range scenarios.The mixed reality approaches presented follow studies on human factors performance and cognitive loading. The resulting designs serve as test beds for studying UAV pilot performance, creating training programs, and developing tools to augment UAV operations and minimize UAV accidents during operations in near Earth environments.Ph.D., Mechanical Engineering -- Drexel University, 201

Multimodal Neuroergonomic Approaches to Human Behavior and Cognitive Workload in Complex High-Risk Semantically Rich Environments: A Case Study of Local & En-Route Air Traffic Controllers

Fast-paced technology advancements have enabled us to create ecologically valid simulations of high risk, complex, and semantically rich environments in which human interaction and decision-making are the keys to increase system performance. These advances have improved our capabilities of exploring, quantifying, and measuring the underlying mechanisms that guide human behavior using sophisticated neuroergonomic devices; and in turn, improve human performance and reduce human errors. In this thesis, multimodal approaches consisted of a self-report analysis, eye-tracking analysis, and functional near-infrared spectroscopy analysis were used to investigate how veteran local & en-route air traffic controllers carry out their operational tasks. Furthermore, the correlations among the cognitive workload and physiological measures (i.e. eye movement characteristics and brain activities) were investigated. Combining the results of these experiments, we can observe that the multimodal approaches show promise on exploring the underlying mechanisms of workload and human interaction in a complex, high-risk, and semantically rich environment. This is because cognitive workload can be considered as a multidimensional construct and different devices or approaches might be more effective in sensing changes in either the task difficulty or complexity. The results can be used to find ways to better train the novices

Development of a System for the Training Assessment and Mental Workload Evaluation

Several studies have demonstrated that the main cause of accidents are due to Human Factor (HF) failures. Humans are the least and last controllable factor in the activity workflows, and the availability of tools able to provide objective information about the user’s cognitive state should be very helpful in maintain proper levels of safety. To overcome these issues, the objectives of the PhD covered three topics. The first phase was focused on the study of machine-learning techniques to evaluate the user’s mental workload during the execution of a task. In particular, the methodology was developed to address two important limitations: i) over-time reliability (no recalibration of the algorithm); ii) automatic brain features selection to avoid both the underfitting and overfitting problems. The second phase was dedicated to the study of the training assessment. In fact, the standard training evaluation methods do not provide any objective information about the amount of brain activation\resources required by the user, neither during the execution of the task, nor across the training sessions. Therefore, the aim of this phase was to define a neurophysiological methodology able to address such limitation. The third phase of the PhD consisted in overcoming the lack of neurophysiological studies regarding the evaluation of the cognitive control behaviour under which the user performs a task. The model introduced by Rasmussen was selected to seek neurometrics to characterize the skill, rule and knowledge behaviours by means of the user’s brain activity. The experiments were initially ran in controlled environments, whilst the final tests were carried out in realistic environments. The results demonstrated the validity of the developed algorithm and methodologies (2 patents pending) in solving the issues quoted initially. In addition, such results brought to the submission of a H2020-SMEINST project, for the realization of a device based on such results