Using Noninvasive Brain Measurement to Explore the Psychological Effects of Computer Malfunctions on Users during Human-Computer Interactions

In today’s technologically driven world, there is a need to better understand the ways that common computer malfunctions affect computer users. These malfunctions may have measurable influences on computer user’s cognitive, emotional, and behavioral responses. An experiment was conducted where participants conducted a series of web search tasks while wearing functional nearinfrared spectroscopy (fNIRS) and galvanic skin response sensors. Two computer malfunctions were introduced during the sessions which had the potential to influence correlates of user trust and suspicion. Surveys were given after each session to measure user’s perceived emotional state, cognitive load, and perceived trust. Results suggest that fNIRS can be used to measure the different cognitive and emotional responses associated with computer malfunctions. These cognitive and emotional changes were correlated with users’ self-report levels of suspicion and trust, and they in turn suggest future work that further explores the capability of fNIRS for the measurement of user experience during human-computer interactions

Measuring cognitive load and cognition: metrics for technology-enhanced learning

This critical and reflective literature review examines international research published over the last decade to summarise the different kinds of measures that have been used to explore cognitive load and critiques the strengths and limitations of those focussed on the development of direct empirical approaches. Over the last 40 years, cognitive load theory has become established as one of the most successful and influential theoretical explanations of cognitive processing during learning. Despite this success, attempts to obtain direct objective measures of the theory's central theoretical construct – cognitive load – have proved elusive. This obstacle represents the most significant outstanding challenge for successfully embedding the theoretical and experimental work on cognitive load in empirical data from authentic learning situations. Progress to date on the theoretical and practical approaches to cognitive load are discussed along with the influences of individual differences on cognitive load in order to assess the prospects for the development and application of direct empirical measures of cognitive load especially in technology-rich contexts

Prefrontal cortex activation upon a demanding virtual hand-controlled task: A new frontier for neuroergonomics

open9noFunctional near-infrared spectroscopy (fNIRS) is a non-invasive vascular-based functional neuroimaging technology that can assess, simultaneously from multiple cortical areas, concentration changes in oxygenated-deoxygenated hemoglobin at the level of the cortical microcirculation blood vessels. fNIRS, with its high degree of ecological validity and its very limited requirement of physical constraints to subjects, could represent a valid tool for monitoring cortical responses in the research field of neuroergonomics. In virtual reality (VR) real situations can be replicated with greater control than those obtainable in the real world. Therefore, VR is the ideal setting where studies about neuroergonomics applications can be performed. The aim of the present study was to investigate, by a 20-channel fNIRS system, the dorsolateral/ventrolateral prefrontal cortex (DLPFC/VLPFC) in subjects while performing a demanding VR hand-controlled task (HCT). Considering the complexity of the HCT, its execution should require the attentional resources allocation and the integration of different executive functions. The HCT simulates the interaction with a real, remotely-driven, system operating in a critical environment. The hand movements were captured by a high spatial and temporal resolution 3-dimensional (3D) hand-sensing device, the LEAP motion controller, a gesture-based control interface that could be used in VR for tele-operated applications. Fifteen University students were asked to guide, with their right hand/forearm, a virtual ball (VB) over a virtual route (VROU) reproducing a 42 m narrow road including some critical points. The subjects tried to travel as long as possible without making VB fall. The distance traveled by the guided VB was 70.2 ± 37.2 m. The less skilled subjects failed several times in guiding the VB over the VROU. Nevertheless, a bilateral VLPFC activation, in response to the HCT execution, was observed in all the subjects. No correlation was found between the distance traveled by the guided VB and the corresponding cortical activation. These results confirm the suitability of fNIRS technology to objectively evaluate cortical hemodynamic changes occurring in VR environments. Future studies could give a contribution to a better understanding of the cognitive mechanisms underlying human performance either in expert or non-expert operators during the simulation of different demanding/fatiguing activities.openCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; Basso Moro, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, ValentinaCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; BASSO MORO, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, Valentin

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations

Machine Learning Methods for functional Near Infrared Spectroscopy

Identification of user state is of interest in a wide range of disciplines that fall under the umbrella of human machine interaction. Functional Near Infra-Red Spectroscopy (fNIRS) device is a relatively new device that enables inference of brain activity through non-invasively pulsing infra-red light into the brain. The fNIRS device is particularly useful as it has a better spatial resolution than the Electroencephalograph (EEG) device that is most commonly used in Human Computer Interaction studies under ecologically valid settings. But this key advantage of fNIRS device is underutilized in current literature in the fNIRS domain. We propose machine learning methods that capture this spatial nature of the human brain activity using a novel preprocessing method that uses `Region of Interest\u27 based feature extraction. Experiments show that this method outperforms the F1 score achieved previously in classifying `low\u27 vs `high\u27 valence state of a user. We further our analysis by applying a Convolutional Neural Network (CNN) to the fNIRS data, thus preserving the spatial structure of the data and treating the data similar to a series of images to be classified. Going further, we use a combination of CNN and Long Short-Term Memory (LSTM) to capture the spatial and temporal behavior of the fNIRS data, thus treating it similar to a video classification problem. We show that this method improves upon the accuracy previously obtained by valence classification methods using EEG or fNIRS devices. Finally, we apply the above model to a problem in classifying combined task-load and performance in an across-subject, across-task scenario of a Human Machine Teaming environment in order to achieve optimal productivity of the system

Physiological Indicators of Task Demand, Fatigue, and Cognition in Future Digital Manufacturing Environments

As Digital Manufacturing transforms traditionally physical work into more system-monitoring tasks, new methods are required for understanding people's mental workload and prolonged capacity for focused attention. Many physiological measures have shown promise for detecting changes in cognitive state, and recent advances in sensor technology offer minimally-invasive ways to monitor our cognitive activity. Previous research in functional near-infrared spectroscopy, for example, has observed changes in cerebral hemodynamic response during periods of high demand within tasks. This work investigated the relationships among task demand, fatigue, and attention degradation in a sustained attention task, and their effect on heart rate, breathing rate, nose temperature and hemodynamic response in the prefrontal cortex and middle temporal gyrus. Analysis revealed a small but significant effect of fatigue on heart rate relative to baseline, breathing rate and hemodynamic response. Task demand had a small but significant effect on breathing rate and nose temperature, both relative to baseline, but no difference between levels of demand was observed in heart rate or hemodynamic response. Our results provide insight into what physiological data can tell us about cognitive state, ability to focus, and the impact of fatigue over time

Exploring Machine Learning Approaches for Classifying Mental Workload using fNIRS Data from HCI Tasks

Functional Near-Infrared Spectroscopy (fNIRS) has shown promise for being potentially more suitable (than e.g. EEG) for brain-based Human Computer Interaction (HCI). While some machine learning approaches have been used in prior HCI work, this paper explores different approaches and configurations for classifying Mental Workload (MWL) from a continuous HCI task, to identify and understand potential limitations and data processing decisions. In particular, we investigate three overall approaches: a logistic regression method, a supervised shallow method (SVM), and a supervised deep learning method (CNN). We examine personalised and gen-eralised models, as well as consider different features and ways of labelling the data. Our initial explorations show that generalised models can perform as well as personalised ones and that deep learning can be a suitable approach for medium size datasets. To provide additional practical advice for future brain-computer interaction systems, we conclude by discussing the limitations and data-preparation needs of different machine learning approaches. We also make recommendations for avenues of future work that are most promising for the machine learning of fNIRS data

Validation of fNIRS System as a Technique to Monitor Cognitive Workload

CognitiveWorkload (CW) is a key factor in the human learning context. Knowing the optimal amount of CW is essential to maximise cognitive performance, emerging as an important variable in e-learning systems and Brain-Computer Interfaces (BCI) applications. Functional Near-Infrared Spectroscopy (fNIRS) has emerged as a promising avenue of brain discovery because of its easy setup and robust results. It is, in fact, along with Electroencephalography (EEG), an encouraging technique in the context of BCI. Brain- Computer Interfaces, by tracking the user’s cognitive state, are suitable for educational systems. Thus, this work sought to validate the fNIRS technique for monitoring different CW stages. For this purpose, we acquired the fNIRS and EEG signals when performing cognitive tasks, which included a progressive increase of difficulty and simulation of the learning process. We also used the breathing sensor and the participants’ facial expressions to assess their cognitive status. We found that both visual inspections of fNIRS signals and power spectral analysis of EEG bands are not sufficient for discriminating cognitive states, nor quantify CW. However, by applying machine learning (ML) algorithms, we were able to distinguish these states with mean accuracies of 79.8%, reaching a value of 100% in one specific case. Our findings provide evidence that fNIRS technique has the potential to monitor different levels of CW. Furthermore, our results suggest that this technique allied with the EEG and combined via ML algorithms is a promising tool to be used in the e-learning and BCI fields for its skill to discriminate and characterize cognitive states.O esforço cognitivo (CW) é um factor relevante no contexto da aprendizagem humana. Conhecer a quantidade óptima de CW é essencial para maximizar o desempenho cognitivo, surgindo como uma variável importante em sistemas de e-learning e aplicações de Interfaces Cérebro-Computador (BCI). A Espectroscopia Funcional de Infravermelho Próximo (fNIRS) emergiu como uma via de descoberta do cérebro devido à sua fácil configuração e resultados robustos. É, de facto, juntamente com a Electroencefalografia (EEG), uma técnica encorajadora no contexto de BCI. As interfaces cérebro-computador, ao monitorizar o estado cognitivo do utilizador, são adequadas para sistemas educativos. Assim, este trabalho procurou validar o sistema de fNIRS como uma técnica de monitorização de CW. Para este efeito, adquirimos os sinais fNIRS e EEG aquando da execução de tarefas cognitivas, que incluiram um aumento progressivo de dificuldade e simulação do processo de aprendizagem. Utilizámos, ainda, o sensor de respiração e as expressões faciais dos participantes para avaliar o seu estado cognitivo. Verificámos que tanto a inspeção visual dos sinais de fNIRS como a análise espectral dos sinais de EEG não são suficientes para discriminar estados cognitivos, nem para quantificar o CW. No entanto, aplicando algoritmos de machine learning (ML), fomos capazes de distinguir estes estados com exatidões médias de 79.8%, chegando a atingir o valor de 100% num caso específico. Os nossos resultados fornecem provas da prospecção da técnica fNIRS para supervisionar diferentes níveis de CW. Além disso, os nossos resultados sugerem que esta técnica aliada à de EEG e combinada via algoritmos ML é uma ferramenta promissora a ser utilizada nos campos do e-learning e de BCI, pela sua capacidade de discriminar e caracterizar estados cognitivos

Neuroergonomic Assessment of Wheelchair Control Using Mobile fNIRS

For over two centuries, the wheelchair has been one of the most common assistive devices for individuals with locomotor impairments without many modifications. Wheelchair control is a complex motor task that increases both the physical and cognitive workload. New wheelchair interfaces, including Power Assisted devices, can further augment users by reducing the required physical effort, however little is known on the mental effort implications. In this study, we adopted a neuroergonomic approach utilizing mobile and wireless functional near infrared spectroscopy (fNIRS) based brain monitoring of physically active participants. 48 volunteers (30 novice and 18 experienced) self-propelled on a wheelchair with and without a PowerAssist interface in both simple and complex realistic environments. Results indicated that as expected, the complex more difficult environment led to lower task performance complemented by higher prefrontal cortex activity compared to the simple environment. The use of the PowerAssist feature had significantly lower brain activation compared to traditional manual control only for novices. Expertise led to a lower brain activation pattern within the middle frontal gyrus, complemented by performance metrics that involve lower cognitive workload. Results here confirm the potential of the Neuroergonomic approach and that direct neural activity measures can complement and enhance task performance metrics. We conclude that the cognitive workload benefits of PowerAssist are more directed to new users and difficult settings. The approach demonstrated here can be utilized in future studies to enable greater personalization and understanding of mobility interfaces within real-world dynamic environments

Measuring mental workload with EEG+fNIRS

We studied the capability of a Hybrid functional neuroimaging technique to quantify human mental workload (MWL). We have used electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) as imaging modalities with 17 healthy subjects performing the letter n-back task, a standard experimental paradigm related to working memory (WM). The level of MWL was parametrically changed by variation of n from 0 to 3. Nineteen EEG channels were covering the whole-head and 19 fNIRS channels were located on the forehead to cover the most dominant brain region involved in WM. Grand block averaging of recorded signals revealed specific behaviors of oxygenated-hemoglobin level during changes in the level of MWL. A machine learning approach has been utilized for detection of the level of MWL. We extracted different features from EEG, fNIRS, and EEG+fNIRS signals as the biomarkers of MWL and fed them to a linear support vector machine (SVM) as train and test sets. These features were selected based on their sensitivity to the changes in the level of MWL according to the literature. We introduced a new category of features within fNIRS and EEG+fNIRS systems. In addition, the performance level of each feature category was systematically assessed. We also assessed the effect of number of features and window size in classification performance. SVM classifier used in order to discriminate between different combinations of cognitive states from binary- and multi-class states. In addition to the cross-validated performance level of the classifier other metrics such as sensitivity, specificity, and predictive values were calculated for a comprehensive assessment of the classification system. The Hybrid (EEG+fNIRS) system had an accuracy that was significantly higher than that of either EEG or fNIRS. Our results suggest that EEG+fNIRS features combined with a classifier are capable of robustly discriminating among various levels of MWL. Results suggest that EEG+fNIRS should be preferred to only EEG or fNIRS, in developing passive BCIs and other applications which need to monitor users' MWL