Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs

A Multi-Task Learning Approach for Meal Assessment

Key role in the prevention of diet-related chronic diseases plays the balanced nutrition together with a proper diet. The conventional dietary assessment methods are time-consuming, expensive and prone to errors. New technology-based methods that provide reliable and convenient dietary assessment, have emerged during the last decade. The advances in the field of computer vision permitted the use of meal image to assess the nutrient content usually through three steps: food segmentation, recognition and volume estimation. In this paper, we propose a use one RGB meal image as input to a multi-task learning based Convolutional Neural Network (CNN). The proposed approach achieved outstanding performance, while a comparison with state-of-the-art methods indicated that the proposed approach exhibits clear advantage in accuracy, along with a massive reduction of processing time

Pre-stimulus antero-posterior EEG connectivity predicts performance in a UAV monitoring task

Long monitoring tasks without regular actions, are becoming increasingly common from aircraft pilots to train conductors as these systems grow more automated. These task contexts are challenging for the human operator because they require inputs at irregular and highly interspaced moments even though these actions are often critical. It has been shown that such conditions lead to divided and distracted attentional states which in turn reduce the processing of external stimuli (e.g. alarms) and may lead to miss critical events. In this study we explored to which extent it is possible to predict an operator’s behavioural performance in a Unmanned Aerial Vehicle (UAV) monitoring task using electroencephalographic (EEG) activity. More specifically we investigated the relevance of large-scale EEG connectivity for performance prediction by correlating relative coherence with reaction times (RT). We show that long-range EEG relative coherence, i.e. between occipital and frontal electrodes, is significantly correlated with RT and that different frequency bands exhibit opposite effects. More specifically we observed that coherence between occipital and frontal electrodes was: negatively correlated with RT at 6Hz (theta band), more coherence leading to better performance, and positively correlated with RT at 8Hz (lower alpha band), more coherence leading to worse performance. Our results suggest that EEG connectivity measures could be useful in predicting an operator’s attentional state and her/his performances in ecological settings. Hence these features could potentially be used in a neuro-adaptive interface to improve operator-system interaction and safety in critical systems

Spectral EEG-based classification for operator dyads workload and cooperation level estimation

There is a growing momentum to design online tools to measure mental workload for neuroergonomic purposes. Most of the research focuses on the monitoring of a single human operator. However, in real-life situations, human operators work in cooperation to optimize safety and performance. This is particularly the case in aviation whereby crews are composed of a pilot flying and a pilot monitoring. The motivation of this study is to evaluate the possibility to apply an hyperscanning approach to estimate the mental workload of crews composed of two operators. We designed an experimental protocol in which ten crews (i.e. 20 subjects) had to perform a modified version of the NASA MATBII during 8 five-minute blocks (i.e. 4 mental workload level configurations * 2 cooperation v. non cooperation conditions). Mental workload and cooperation level were classified using a traditional passive brain-computer interface pipeline that includes a spatial filtering step on frequency features. Our results disclosed that all mental states’ estimations were significantly above chance level. Intra-subject classification accuracy for mental workload (2 classes) was 63% for the pilot flying and 58% for the pilot monitoring. As for cooperation level, the binary classification reached 57% for the pilot flying and 60% for the pilot monitoring. Regarding the team, intra-team classification accuracy of the workload configuration of the team (4-class) reached 35%. As for the team cooperation level, the binary classifier reached 60% of accuracy. The results are discussed in terms of hyperscanning applications

Biocybernetic Adaptation Strategies: Machine awareness of human state for improved operational performance

Human operators interacting with machines or computers continually adapt to the needs of the system ideally resulting in optimal performance. In some cases, however, deteriorated performance is an outcome. Adaptation to the situation is a strength expected of the human operator which is often accomplished by the human through self-regulation of mental state. Adaptation is at the core of the human operator’s activity, and research has demonstrated that the implementation of a feedback loop can enhance this natural skill to improve training and human/machine interaction. Biocybernetic adaptation involves a “loop upon a loop,” which may be visualized as a superimposed loop which senses a physiological signal and influences the operator’s task at some point. Biocybernetic adaptation in, for example, physiologically adaptive automation employs the “steering” sense of “cybernetic,” and serves a transitory adaptive purpose – to better serve the human operator by more fully representing their responses to the system. The adaptation process usually makes use of an assessment of transient cognitive state to steer a functional aspect of a system that is external to the operator’s physiology from which the state assessment is derived. Therefore, the objective of this paper is to detail the structure of biocybernetic systems regarding the level of engagement of interest for adaptive systems, their processing pipeline, and the adaptation strategies employed for training purposes, in an effort to pave the way towards machine awareness of human state for self-regulation and improved operational performance

Physiological synchrony revealed by delayed coincidence count: Application to a cooperative complex environment

Synchrony at the physiological level is an objective measure that can be used to investigate cooperation between human agents. This physiological synchrony has been experimentally observed in different dyadic contexts through measures of the autonomous system such as cardiac measures. Various metrics were used to characterize synchrony between participants such as cross-correlation, weighted coherence or cross recurrence quantification analysis (CRQA) and with a wide variety of paradigms. We propose the delayed coincidence count as a new method for assessing cardiac synchrony. Delayed coincidence count has already been used to characterize synchrony in firing neurons populations. While being straightforward and computationally light, this method has already been formally proven to be statistically robust. A complex dynamic micro-world was designed with two difficulty levels and two cooperation conditions. Forty participants, i.e. 20 teams, voluntarily underwent the experiment. The delayed coincidence count method (with a coincidence threshold delta of 20 ms) revealed a significant synchrony (p < .01) during the cooperative and high difficulty condition only, while the other methods did not. The results are interpreted in terms of interaction intensity in accordance with recent literature

Ontology-Driven Food Category Classification in Images

The self-management of chronic diseases related to dietary habits includes the necessity of tracking what people eat. Most of the approaches proposed in the literature classify food pictures by labels describing the whole recipe. The main drawback of this kind of strategy is that a wrong prediction of the recipe leads to a wrong prediction of any ingredient of such a recipe. In this paper we present a multi-label food classification approach, exploiting deep neural networks, where each food picture is classified with labels describing the food categories of the ingredients in each recipe. The aim of our approach is to support the detection of food categories in order to detect which one might be dangerous for a user affected by chronic disease. Our approach relies on background knowledge where recipes, food categories, and their relatedness with chronic diseases are modeled within a state-of-the-art ontology. Experiments conducted on a new publicly released dataset demonstrated the effectiveness of the proposed approach with respect to state-of-the-art classification strategies

Cumulative incidence and risk factors for radiation induced leukoencephalopathy in high grade glioma long term survivors

The incidence and risk factors associated with radiation-induced leukoencephalopathy (RIL) in long-term survivors of high-grade glioma (HGG) are still poorly investigated. We performed a retrospective research in our institutional database for patients with supratentorial HGG treated with focal radiotherapy, having a progression-free overall survival > 30 months and available germline DNA. We reviewed MRI scans for signs of leukoencephalopathy on T2/FLAIR sequences, and medical records for information on cerebrovascular risk factors and neurological symptoms. We investigated a panel of candidate single nucleotide polymorphisms (SNPs) to assess genetic risk. Eighty-one HGG patients (18 grade IV and 63 grade III, 50M/31F) were included in the study. The median age at the time of radiotherapy was 48 years old (range 18–69). The median follow-up after the completion of radiotherapy was 79 months. A total of 44 patients (44/81, 54.3%) developed RIL during follow-up. Twenty-nine of the 44 patients developed consistent symptoms such as subcortical dementia (n = 28), gait disturbances (n = 12), and urinary incontinence (n = 9). The cumulative incidence of RIL was 21% at 12 months, 42% at 36 months, and 48% at 60 months. Age > 60 years, smoking, and the germline SNP rs2120825 (PPARg locus) were associated with an increased risk of RIL. Our study identified potential risk factors for the development of RIL (age, smoking, and the germline SNP rs2120825) and established the rationale for testing PPARg agonists in the prevention and management of late-delayed radiation-induced neurotoxicity

Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic environments. Recent technological progress has allowed the development of new generations of brain imaging systems such as dry electrodes electroencephalography (EEG) and functional near infrared spec- troscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. These highly portable brain imaging devices offer interesting prospects to implement passive brain computer interfaces (pBCI) and neuroadaptive technology. We developed a fNIRS-EEG based pBCI to monitor cognitive fatigue using engagement related features (EEG engagement ratio and wavelet coherence fNIRS based metrics). This mental state is known to impair cognitive performance and can jeopardize flight safety. In this preliminary study, four participants were asked to perform four identical traffic patterns along with a secondary auditory task in a flight simulator and in an actual light aircraft. The two first traffic patterns were considered as the low cognitive fatigue class, whereas the two last traffic patterns were considered as the high cognitive fatigue class. As expected, the pilots missed more auditory targets in the second part than in the first part of the experiment. Classification accuracy reached 87.2% in the flight simulator condition and 87.6% in the actual flight conditions when combining the two modalities. This study demonstrates that fNIRS and EEG-based pBCIs can monitor mental states in operational and noisy environments

An End-to-End Semantic Platform for Nutritional Diseases Management

The self-management of nutritional diseases requires a system that combines food tracking with the potential risks of food categories on people’s health based on their personal health records (PHRs). The challenges range from the design of an effective food image classification strategy to the development of a full-fledged knowledge-based system. This maps the results of the classification strategy into semantic information that can be exploited for reasoning. However, current works mainly address the single challenges separately without their integration into a whole pipeline. In this paper, we propose a new end-to-end semantic platform where: (i) the classification strategy aims to extract food categories from food pictures; (ii) an ontology is used for detecting the risk factors of food categories for specific diseases; (iii) the Linked Open Data (LOD) Cloud is queried for extracting information concerning related diseases and comorbidities; and, (iv) information from the users’ PHRs are exploited for generating proper personal feedback. Experiments are conducted on a new publicly released dataset. Quantitative and qualitative evaluations, from two living labs, demonstrate the effectiveness and the suitability of the proposed approach