A Mental Workload Estimation Model for Visualization Using EEG

Various visualization design guides have been proposed and evaluated through quantitative methods that compare the response accuracy and time for completing visualization tasks. However, accuracy and time do not always represent the mental workload. Since quantitative approaches do not fully mirror mental workload, questionnaires and biosignals have been employed to measure mental workload in visualization assessments. The EEG as biosignal is one of the indicators frequently utilized to measure mental workload. Nevertheless, many studies have not applied the EEG for mental workload measurement in the visualization evaluation. In this work, we study the EEG to measure mental workload for visualization evaluation. We examine whether there is a difference in mental workload for the visualization designs suggested by the previously proposed visualization design guides. Besides, we propose a mental workload estimation model using EEG data specialized for each individual to evaluate visualization designs

Detection, recuperation and cross-subject classification of mental fatigue

La fatigue mentale est un état complexe qui résulte d'une activité cognitive prolongée. Les symptômes de la fatigue mentale inclus des changements d'humeur, de motivation et une détérioration temporaire de diverses fonctions cognitives. Plusieurs recherches approfondies ont été menées pour développer des méthodes de reconnaissance des signes physiologiques et psychophysiologiques de la fatigue mentale. Les signes psychophysiologiques concernent principalement signaux d'activité cérébrale et leur relation avec la psychologie et la cognition. Celles-ci ont permise le développement de nombreux modèles basés sur l'IA pour classer différents niveaux de fatigue, en utilisant des données extraites d'un appareil eye-tracking, d'un électroencéphalogramme (EEG) pour mesurer l’activité cérébrale ou d'un électrocardiogramme (ECG) pour mesurer l’activité cérébrale. Dans cette mémoire, nous présentons le protocole expérimental et développé par mes directeurs de recherche et moi-même, qui vise à la fois à générer et mesurer la fatigue mentale, et à proposer des stratégies efficaces de récupération via des séances de réalité virtuelle couplées à des dispositifs EEG et eye tracking. Réussir à générer de la fatigue mentale est nécessaire pour générer un ensemble de données suivant l’évolution de la fatigue et de la récupération au cours de l’expérience, et sera également utilisé pour classer différents niveaux de fatigue à l’aide de l’apprentissage automatique. Cette mémoire fournit d'abord un état de l'art complet des facteurs prédictifs de la fatigue mentale, des méthodes de mesure et des stratégies de récupération. Ensuite, l'article présente un protocole expérimental résultant de l'état de l'art pour (1) générer et mesurer la fatigue mentale et (2) évaluer l'efficacité de la thérapie virtuelle pour la récupération de la fatigue, (3) entrainer un algorithme d'apprentissage automatique sur les données EEG pour classer 3 niveaux de fatigue différents en utilisant un environnement simulé de réalité virtuelle (VR). La thérapie virtuelle est une technique favorisant la relaxation dans un environnement simulé virtuel et interactif qui vise à réduire le stress. Dans notre travail, nous avons réussi à générer de la fatigue mentale en accomplissant des tâches cognitives dans un environnement virtuel. Les participants ont montré une diminution significative du diamètre de la pupille et du score thêta/alpha au cours des différentes tâches cognitives. Le score alpha/thêta est un indice EEG qui suit les fluctuations de la charge cognitiveet de la fatigue mentale. Divers algorithmes d'apprentissage automatique ont été formés et testés sur des segments de données EEG afin de sélectionner le modèle qui s'ajuste le mieux à ces données en ce qui concerne la métrique d'évaluation "précision équilibrée" et "f1". Parmi les 8 différents classificateurs, le SVM RBF a montré les meilleures performances avec une précision équilibrée de 95 % et une valeur de mesure f de 0,82. La précision équilibrée fournit une mesure précise de la performance dans le cas de jeu de données déséquilibrées, en tenant compte de la sensibilité et de la spécificité, et le f-score est une mesure d'évaluation qui combine les scores de précision et de rappel. Finalement, nos résultats montrent que le temps alloué à la thérapie virtuelle n'a pas amélioré le diamètre pupillaire en période post-relaxation. D'autres recherches sur l'impact de la thérapie devraient consacrer un temps plus proche du temps de récupération standard de 60 min.Mental fatigue is a complex state that results from prolonged cognitive activity. Symptoms of mental fatigue can include change in mood, motivation, and temporary deterioration of various cognitive functions involved in goal-directed behavior. Extensive research has been done to develop methods for recognizing physiological and psychophysiological signs of mental fatigue. Psychophysiological signs are mostly concern with patterns of brain activity and their relation to psychology and cognition. This has allowed the development of many AI-based models to classify different levels of fatigue, using data extracted from eye-tracking devices, electroencephalogram (EEG) measuring brain activity, or electrocardiogram (ECG) measuring cardiac activity. In this thesis, we present the experimental protocol developed by my research directors and I, which aims to both generate/measure mental fatigue and provide effective strategies for recuperation via VR sessions paired with EEG and eye-tracking devices. Successfully generating mental fatigue is crucial to generate a time-series dataset tracking the evolution of fatigue and recuperation during the experiment and will also be used to classify different levels of fatigue using machine learning. This thesis first provides a state-of-the-art of mental fatigue predictive factors, measurement methods, and recuperation strategies. The goal of this protocol is to (1) generate and measure mental fatigue, (2) evaluate the effectiveness of virtual therapy for fatigue recuperation, using a virtual reality (VR) simulated environment and (3) train a machine learning algorithm on EEG data to classify 3 different levels of fatigue. Virtual therapy is relaxation promoting technique in a virtual and interactive simulated environment which aims to reduce stress. In our work, we successfully generated mental fatigue through completion of cognitive tasks in a virtual simulated environment. Participants showed significant decline in pupil diameter and theta/alpha score during the various cognitive tasks. The alpha/theta score is an EEG index tracking fluctuations in cognitive load and mental fatigue. Various machine learning algorithm candidates were trained and tested on EEG data segments in order to select the classifier that best fits EEG data with respect to evaluation metric ‘balanced accuracy’ and 'f1-measures'. Among the 8 different classifier candidates, RBF SVM showed the best performance with 95% balanced accuracy 0.82 f-score value and on the validation set, and 92% accuracy and 0.90 f-score on test set. Balanced accuracy provides an accurate measure of performance in the case of imbalanced data, considering sensitivity and specificity and f-score is an evaluation metric which combines precision and recall scores. Finally, our results show that the time allocated for virtual therapy did not improve pupil diameter in the post-relaxation period. Further research on the impact of relaxation therapy should allocate time closer to the standard recovery time of 60 min

Validation of fNIRS System as a Technique to Monitor Cognitive Workload

CognitiveWorkload (CW) is a key factor in the human learning context. Knowing the optimal amount of CW is essential to maximise cognitive performance, emerging as an important variable in e-learning systems and Brain-Computer Interfaces (BCI) applications. Functional Near-Infrared Spectroscopy (fNIRS) has emerged as a promising avenue of brain discovery because of its easy setup and robust results. It is, in fact, along with Electroencephalography (EEG), an encouraging technique in the context of BCI. Brain- Computer Interfaces, by tracking the user’s cognitive state, are suitable for educational systems. Thus, this work sought to validate the fNIRS technique for monitoring different CW stages. For this purpose, we acquired the fNIRS and EEG signals when performing cognitive tasks, which included a progressive increase of difficulty and simulation of the learning process. We also used the breathing sensor and the participants’ facial expressions to assess their cognitive status. We found that both visual inspections of fNIRS signals and power spectral analysis of EEG bands are not sufficient for discriminating cognitive states, nor quantify CW. However, by applying machine learning (ML) algorithms, we were able to distinguish these states with mean accuracies of 79.8%, reaching a value of 100% in one specific case. Our findings provide evidence that fNIRS technique has the potential to monitor different levels of CW. Furthermore, our results suggest that this technique allied with the EEG and combined via ML algorithms is a promising tool to be used in the e-learning and BCI fields for its skill to discriminate and characterize cognitive states.O esforço cognitivo (CW) é um factor relevante no contexto da aprendizagem humana. Conhecer a quantidade óptima de CW é essencial para maximizar o desempenho cognitivo, surgindo como uma variável importante em sistemas de e-learning e aplicações de Interfaces Cérebro-Computador (BCI). A Espectroscopia Funcional de Infravermelho Próximo (fNIRS) emergiu como uma via de descoberta do cérebro devido à sua fácil configuração e resultados robustos. É, de facto, juntamente com a Electroencefalografia (EEG), uma técnica encorajadora no contexto de BCI. As interfaces cérebro-computador, ao monitorizar o estado cognitivo do utilizador, são adequadas para sistemas educativos. Assim, este trabalho procurou validar o sistema de fNIRS como uma técnica de monitorização de CW. Para este efeito, adquirimos os sinais fNIRS e EEG aquando da execução de tarefas cognitivas, que incluiram um aumento progressivo de dificuldade e simulação do processo de aprendizagem. Utilizámos, ainda, o sensor de respiração e as expressões faciais dos participantes para avaliar o seu estado cognitivo. Verificámos que tanto a inspeção visual dos sinais de fNIRS como a análise espectral dos sinais de EEG não são suficientes para discriminar estados cognitivos, nem para quantificar o CW. No entanto, aplicando algoritmos de machine learning (ML), fomos capazes de distinguir estes estados com exatidões médias de 79.8%, chegando a atingir o valor de 100% num caso específico. Os nossos resultados fornecem provas da prospecção da técnica fNIRS para supervisionar diferentes níveis de CW. Além disso, os nossos resultados sugerem que esta técnica aliada à de EEG e combinada via algoritmos ML é uma ferramenta promissora a ser utilizada nos campos do e-learning e de BCI, pela sua capacidade de discriminar e caracterizar estados cognitivos

A Novel Analysis of Performance Classification and Workload Prediction Using Electroencephalography (EEG) Frequency Data

Across the DOD each task an operator is presented with has some level of difficulty associated with it. This level of difficulty over the course of the task is also known as workload, where the operator is faced with varying levels of workload as he or she attempts to complete the task. The focus of the research presented in this thesis is to determine if those changes in workload can be predicted and to determine if individuals can be classified based on performance in order to prevent an increase in workload that would cause a decline in performance in a given task. Despite many efforts to predict workload and classify individuals with machine learning, the classification and predictive ability of Electroencephalography (EEG) frequency data has not been explored at the individual EEG Frequency band level. In a 711th HPW/RCHP Human Universal Measurement and Assessment Network (HUMAN) Lab study, 14 Subjects were asked to complete two tasks over 16 scenarios, while their physiological data, including EEG frequency data, was recorded to capture the physiological changes their body went through over the course of the experiment. The research presented in this thesis focuses on EEG frequency data, and its ability to predict task performance and changes in workload. Several machine learning techniques are explored in this thesis before a final technique was chosen. This thesis contributes research to the medical and machine learning fields regarding the classification and workload prediction efficacy of EEG frequency data. Specifically, it presents a novel investigation of five EEG frequencies and their individual abilities to predict task performance and workload. It was discovered that using the Gamma EEG frequency and all EEG frequencies combined to predict task performance resulted in average classification accuracies of greater than 90%

SAFECAR: A Brain–Computer Interface and intelligent framework to detect drivers’ distractions

As recently reported by the World Health Organization (WHO), the high use of intelligent devices such as smartphones, multimedia systems, or billboards causes an increase in distraction and, consequently, fatal accidents while driving. The use of EEG-based Brain–Computer Interfaces (BCIs) has been proposed as a promising way to detect distractions. However, existing solutions are not well suited for driving scenarios. They do not consider complementary data sources, such as contextual data, nor guarantee realistic scenarios with real-time communications between components. This work proposes an automatic framework for detecting distractions using BCIs and a realistic driving simulator. The framework employs different supervised Machine Learning (ML)-based models on classifying the different types of distractions using Electroencephalography (EEG) and contextual driving data collected by car sensors, such as line crossings or objects detection. This framework has been evaluated using a driving scenario without distractions and a similar one where visual and cognitive distractions are generated for ten subjects. The proposed framework achieved 83.9% -score with a binary model and 73% with a multiclass model using EEG, improving 7% in binary classification and 8% in multi-class classification by incorporating contextual driving into the training dataset. Finally, the results were confirmed by a neurophysiological study, which revealed significantly higher voltage in selective attention and multitasking