In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations

Using near infrared spectroscopy and heart rate variability to detect mental overload

Mental workload is a key factor influencing the occurrence of human error, especially during piloting and remotely operated vehicle (ROV) operations, where safety depends on the ability of pilots to act appropriately. In particular, excessively high or low mental workload can lead operators to neglect critical information. The objective of the present study is to investigate the potential of functional Near Infrared Spectroscopy (fNIRS) – a non-invasive method of measuring prefrontal cortex activity – in combination with measurements of heart rate variability (HRV), to predict mental workload during a simulated piloting task, with particular regard to task engagement and disengagement. Twelve volunteers performed a computer-based piloting task in which they were asked to follow a dynamic target with their aircraft, a task designed to replicate key cognitive demands associated with real life ROV operating tasks. In order to cover a wide range of mental workload levels, task difficulty was manipulated in terms of processing load and difficulty of control – two critical sources of workload associated with piloting and remotely operating a vehicle. Results show that both fNIRS and HRV are sensitive to different levels of mental workload; notably, lower prefrontal activation as well as a lower LF/HF ratio at the highest level of difficulty, suggest that these measures are suitable for mental overload detection. Moreover, these latter measurements point towards the existence of a quadratic model of mental workload

Functional Near Infrared Spectroscopy: Watching the Brain in Flight

Functional Near Infrared Spectroscopy (fNIRS) is an emerging neurological sensing technique applicable to optimizing human performance in transportation operations, such as commercial aviation. Cognitive state can be determined via pattern classification of functional activations measured with fNIRS. Operational application calls for further development of algorithms and filters for dynamic artifact removal. The concept of using the frequency domain phase shift signal to tune a Kalman filter is introduced to improve the quality of fNIRS signals in realtime. Hemoglobin concentration and phase shift traces were simulated for four different types of motion artifact to demonstrate the filter. Unwanted signal was reduced by at least 43%, and the contrast of the filtered oxygenated hemoglobin signal was increased by more than 100% overall. This filtering method is a good candidate for qualifying fNIRS signals in real time without auxiliary sensor

Real-Time State Estimation in a Flight Simulator Using fNIRS

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot’s instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot’s mental state matched significantly better than chance with the pilot’s real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development

Using functional near infrared spectroscopy to assess cognitive performance of UAV sensor operators during route scanning

The composition of UAV (Unmanned Aerial Vehicle) crew will sometimes define roles specific to tasks associated with the Ground Control Station (GCS). The sensor operator task is specific to both the type of platform and GCS they are operating, but in many instances the role of this operator is critical in determining mission success. In order to assess mission effectiveness we applied human performance measures that focussed on neurological brain imaging techniques and other physiological biomarkers in conjunction with behavioral data acquired from the sensor operator task. In the execution of the experiment, this included such tasks as route scanning, target detection and positive identification, and the tracking of identified targets. Within the scope of this paper, we reported the preliminary results for the route scanning task. Over the duration of three trials brain activity measures from the prefrontal cortex region were acquired via functional near infrared spectroscopy (fNIRS) in this research study. As the trials progressed, there was a significant difference between low and high performers on the route scanning task as determined by specific biomarkers, namely oxygenated haemoglobin. These findings support previous studies and indicates the benefits of applying neurophysiological measures in order to gain further objective insight into human cognitive performance. The use of fNIRS in this context is also discussed in terms of providing a key benefit in dynamically evaluating human performance in parallel with personalized training for UAV operators

UAV Operator mental workload:A neurophysiological comparison of mental workload and vigilance

Human Factors can offer insights into the nature of human performance across many different domains. The steady increase of unmanned systems presents not only a unique challenge in terms of defining the nature of human-system interaction, but also the demand for providing decision support systems to assist the human operate multiple of these systems, or indeed operate beyond line of visual sight. The nature of cognitive performance can involve a high degree of complexity and in many instances result in disagreement over what it is that is actually being measured. The main cognitive processes that tend to be discussed in terms of operating UAVs tends to focus on mental workload and situation awareness. However, other constructs, such as vigilance, may be considered as important when we examine the task of commanding a UAV – more so when a single operator is supervising multiple UAVs. This paper presents the findings of a study whereby participants were asked to perform tasks involving the control of a UAV. Neurophysiological assessment was carried out by application of functional near infra-red spectroscopy, and results are discussed in relation to how this technique can provide insight into higher cognitive functions related to UAV operator state

Using Functional Near Infrared Spectroscopy to Assess Cognitive Workload

Quantification of mental workload is a significant aspect of monitoring and adaptive aiding systems that are intended to improve the efficiency and safety of human–machine systems. Functional near Infrared (fNIR) spectroscopy is a field-deployable brain monitoring device that provides a measures of cerebral hemodynamic within the prefrontal cortex. The purpose of this study was to assess the cognitive load by using Performance (reaction time), Behavioral metrics (NASA TLX) and Neuro-Cognitive Measures (Hemodynamic response). To observe the activation in prefrontal cortex, we employed Functional Near Infrared (fNIR) Spectroscopy with a Standard Stroop task. A total of 25 healthy participants (N 18 Male and N 07 Female, M Age 25.5 SD 7.6), participated in the study. For statistical analysis, a repeated measure t-test was computed to compare the Oxy (Δ[HbO2]) and De-Oxy (Δ[hHb]) changes under Congruent and In-Congruent task conditions. For Classification, Binary logistic regression model applied to identify how accurately classifying the varied workload conditions. The finding shows that fNIR measures had adequate predictive power for estimating task performance in workload conditions. In this paper, we have found evidence that fNIR can be used as indicator of cognitive load which is important for optimal human performance

Aerospace Medicine and Biology: A continuing bibliography with indexes, supplement 217, March 1981

Approximately 130 reports, articles, and other documents introduced into the NASA scientific and technical information system in February 1981 are included in this bibliography. Topics include aerospace medicine and biology

Toward Adaptation of fNIRS Instrumentation to Airborne Environments

The paper reviews potential applications of functional Near-Infrared Spectroscopy (fNIRS), a well-known medical diagnostic technique, to monitoring the cognitive state of pilots with a focus on identifying ways to adopt this technique to airborne environments. We also discuss various fNIRS techniques and the direction of technology maturation of associated hardware in view of their potential for miniaturization, maximization of data collection capabilities, and user friendliness

Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs