EEG-Based Multi-Class Workload Identification Using Feature Fusion and Selection

The effectiveness of workload identification is one of the critical aspects in a monitoring instrument of mental state. In this field, the workload is usually recognised as binary classes. There are scarce studies towards multi-class workload identification because the challenge of the success of workload identification is much tough, even though one more workload class is added. Besides, most of the existing studies only utilized spectral power features from individual channels but ignoring abundant inter-channel features that represent the interactions between brain regions. In this study, we utilized features representing intra-channel information and inter-channel information to classify multiple classes of workload based on EEG. We comprehensively compared each category of features contributing to workload identification and elucidated the roles of feature fusion and feature selection for the workload identification. The results demonstrated that feature combination (83.12% in terms of accuracy) enhanced the classification performance compared to individual feature categories (i.e., band power features, 75.90%; connection features, 81.72%, in terms of accuracy). With the F-score feature selection, the classification accuracy was further increased to 83.47%. When the features of graph metric were fused, the accuracy was reached to 84.34%. Our study provided comprehensive performance comparisons between methods and feature categories for the multi-class workload identification and demonstrated that feature selection and fusion played an important role in the enhancement of workload identification. These results could facilitate further studies of multi-class workload identification and practical application of workload identification

The use of multi-attribute task battery in mental workload studies: A scoping review

Multi-Attribute Task Battery (MATB) is a software that has been arguably utilized in many ergonomics/human factors studies, including in the topic of mental workload. However, the use of this well-known program in diverse investigations has not yet been systematically tracked. Furthermore, it may be argued that a critical appraisal of MATB is urgently needed so that future researchers and users can take several crucial factors into account when planning a study or experiment using MATB. The aim of this paper is to comprehensively identify and review the use of MATB software in published studies. This aim might be accomplished by achieving two goals: (1) systematic discovery of published papers in literature databases and (2) categorization of research according to pertinent topics. In this paper, thirty-one articles were included for analysis after carefully screening for their eligibility. Our scoping review finds that MATB is a beneficial program for creating multitasking environments in general, with aviation being the area where it has been used the most. The program was also extensively used for studies on mental workload, especially by producing various stimuli that ultimately result in varying degrees of task demand or difficulty. Moreover, to successfully use MATB, researchers must be aware of a few operational issues and criticisms

생리학적 데이터의 조합을 바탕으로 한 아동의 집중력, 기억력 예측 알고리즘

학위논문 (석사)-- 서울대학교 대학원 농업생명과학대학 농생명공학부, 2017. 8. 이기원.Good performance is regarded as important element not only in workplace but also in daily activities. Performance of the human depends on the mental capacity and mental workload. Performance declines when the mental workload exceeds mental capacity. The point the mental capacity is exceeded by mental workload is regarded to as the cognitive redline of workload. Performance declines faster at this point, as task demand is greater than the mental capacity. Few studies of the cognitive redline of workload have been done. In addition, for good performance, mental workload is regarded as more important than physical workload. Especially, according to piagets cognitive development theory, children in concrete operational stage is critical for further learning ability that they develop their ability to distinguish between quality and quantity. However, the reason that mental workload is difficult to quantify through physiological measures, makes it more complicated to demonstrate the cognitive redline. When it comes to childrens development, physical change is visible and easy to identify but mental change is not. Moreover, EEG which is one of the representative measuring tool of physiological data requires accurate process of measuring and analyzing with the expert. HRV is relatively easy to measure but has limitation because it is indirect way of measuring brain signal. Above all things, many researches of real-time indicator measuring physiological data such as heart rate variability (HRV), skin conductance response (SCR) have been done sporadically but not integrated. Therefore, In this study I tried to demonstrate if I can predict the mental capacity (attention and memory ability) not mental workload with the EEG. In addition, with the combination of EEG and HRV, I tried to overcome disadvantages of physiological tool and tried to develop advanced algorithm which predicts mental capacity. Attention ability was measured with Stroop task, and memory ability was measured with digit span task. Elementary school students aged 6-13 were participated, whose brain development is in important phase according to Piaget theory. In conclusion, right-temporal EEG data significantly predicts attention score, and occipital EEG data significantly predicts memory score. I also analyzed brain wave EEG model, and found out beta EEG power significantly predicted attention score but not memory score. I also analyzed HRV data with all other physiological data to earn more predictable algorithms model. These data can be used as daily performance parameter of attention and memory ability. However, in the further study more number of population are needed to increase the accuracy of the model. Moreover, Application which can collect and analyze physiological data needs to be more sophisticated and needs to be properly connected to wearable devices.Ⅰ. Introduction. 1 1.1. Mental capacity and mental Workload 1 1.2. Physiological measurements as a real time indicator 2 1.2.1 Physiological measurements: Electroencephalography (EEG) 4 1.2.2 Physiological measurements: Heart rate variability (HRV) 5 1.3. Limitation of current physiological researches. 6 1.4. Cognitive development in children. 8 1.5. My hypothesis . 8 Ⅱ. Materials and methods. 10 2.1. Procedures 10 2.2. Participants. 11 2.3. Physiological measurements 16 2.3.1 Resting state EEG recordings. 16 2.3.2 Heart rate variability measurement. 17 2.4. Stroop task 18 2.5. Digit Span task. 19 2.6. Data analysis 20 Ⅲ. Results 23 3.1. Most predictable attention and memory ability algorithm model based on ROI (EEG) analysis 23 3.1.1. Right temporal (F8) is the most predictable attention ability algorithm model and Addition of HRV data to attention ability algorithm increases prediction accuracy 23 3.1.2 Occipital (O2) is the most predictable memory ability algorithm model and Addition of HRV data to memory ability algorithm increases prediction accuracy. 26 3.2. Most predictable attention and memory ability algorithm model based on Brain wave (EEG) analysis. 28 3.2.1 Beta wave is the most predictable attention ability algorithm model and Addition of HRV data to attention ability algorithm increases prediction accuracy. 28 3.2.2 There was no brain wave which can predict memory ability However, with the addition of HRV data, there was increase in predictability in the model. 29 Ⅳ. Discussion 32 4.1. With the specific area (temporal, F8) and specific brainwave (beta wave), Attention can be predictable. Moreover, with the addition of HRV data, predictability increased. As a result, temporal (F8) and beta wave predicts attention mostly 32 4.2. With the specific area (occipital, O2), Memory can be predictable. Moreover, with the addition of HRV data, predictability increased. As a result, Occipital (O2) predicts attention mostly. 34 4.3. Physiological data as real-time indicator in daily life. 38 Ⅴ. References 40 Ⅵ. 국문 초록 46Maste

Cross-Participant EEG-Based Assessment of Cognitive Workload Using Multi-Path Convolutional Recurrent Neural Networks

Applying deep learning methods to electroencephalograph (EEG) data for cognitive state assessment has yielded improvements over previous modeling methods. However, research focused on cross-participant cognitive workload modeling using these techniques is underrepresented. We study the problem of cross-participant state estimation in a non-stimulus-locked task environment, where a trained model is used to make workload estimates on a new participant who is not represented in the training set. Using experimental data from the Multi-Attribute Task Battery (MATB) environment, a variety of deep neural network models are evaluated in the trade-space of computational efficiency, model accuracy, variance and temporal specificity yielding three important contributions: (1) The performance of ensembles of individually-trained models is statistically indistinguishable from group-trained methods at most sequence lengths. These ensembles can be trained for a fraction of the computational cost compared to group-trained methods and enable simpler model updates. (2) While increasing temporal sequence length improves mean accuracy, it is not sufficient to overcome distributional dissimilarities between individuals’ EEG data, as it results in statistically significant increases in cross-participant variance. (3) Compared to all other networks evaluated, a novel convolutional-recurrent model using multi-path subnetworks and bi-directional, residual recurrent layers resulted in statistically significant increases in predictive accuracy and decreases in cross-participant variance

Exploring Physiological Measures for Prediction and Identification of the Redline of Cognitive Workload

Research suggests that physiological measures such as breath rate (BR), heart rate (HR), heart rate variability (HRV), skin conductance response (SCR), and electroencephalography (EEG) tend to be real-time indicators of mental workload, which are related with increases in the sympathetic nervous system. With increased cognitive workload, these physiological measures tend to change, until a plateau is reached. At this point, performance will decrease, as the workload imposed on the user exceeds their mental capacity to perform the task. This occurs when the user reaches their cognitive redline of workload. Performance will start to decline or decline more steeply at this point, as task demand imposed by the tasks is greater than the mental capacity. This thesis seeks to understand the underlying patterns reflected in the physiological data that can potentially be used as real-time indicators of the cognitive redline of workload. The study involved use of the Multi-Attribute Task Battery II (MATB-II) to manipulate workload. Subjective measures and performance were taken at the end of every scenario, while physiological measures (BR, HR, HRV, SCR, and EEG), and performance were analyzed to determine the cognitive redline. Results found subjective measures to be responsive to workload change, while heart rate variability seems to be the best physiological measure to respond to mental workload. EEG and SCR proved to also be reliable predictors

Machine Learning Methods for functional Near Infrared Spectroscopy

Identification of user state is of interest in a wide range of disciplines that fall under the umbrella of human machine interaction. Functional Near Infra-Red Spectroscopy (fNIRS) device is a relatively new device that enables inference of brain activity through non-invasively pulsing infra-red light into the brain. The fNIRS device is particularly useful as it has a better spatial resolution than the Electroencephalograph (EEG) device that is most commonly used in Human Computer Interaction studies under ecologically valid settings. But this key advantage of fNIRS device is underutilized in current literature in the fNIRS domain. We propose machine learning methods that capture this spatial nature of the human brain activity using a novel preprocessing method that uses `Region of Interest\u27 based feature extraction. Experiments show that this method outperforms the F1 score achieved previously in classifying `low\u27 vs `high\u27 valence state of a user. We further our analysis by applying a Convolutional Neural Network (CNN) to the fNIRS data, thus preserving the spatial structure of the data and treating the data similar to a series of images to be classified. Going further, we use a combination of CNN and Long Short-Term Memory (LSTM) to capture the spatial and temporal behavior of the fNIRS data, thus treating it similar to a video classification problem. We show that this method improves upon the accuracy previously obtained by valence classification methods using EEG or fNIRS devices. Finally, we apply the above model to a problem in classifying combined task-load and performance in an across-subject, across-task scenario of a Human Machine Teaming environment in order to achieve optimal productivity of the system

Breaking Down the Barriers To Operator Workload Estimation: Advancing Algorithmic Handling of Temporal Non-Stationarity and Cross-Participant Differences for EEG Analysis Using Deep Learning

This research focuses on two barriers to using EEG data for workload assessment: day-to-day variability, and cross- participant applicability. Several signal processing techniques and deep learning approaches are evaluated in multi-task environments. These methods account for temporal, spatial, and frequential data dependencies. Variance of frequency- domain power distributions for cross-day workload classification is statistically significant. Skewness and kurtosis are not significant in an environment absent workload transitions, but are salient with transitions present. LSTMs improve day- to-day feature stationarity, decreasing error by 59% compared to previous best results. A multi-path convolutional recurrent model using bi-directional, residual recurrent layers significantly increases predictive accuracy and decreases cross-participant variance. Deep learning regression approaches are applied to a multi-task environment with workload transitions. Accounting for temporal dependence significantly reduces error and increases correlation compared to baselines. Visualization techniques for LSTM feature saliency are developed to understand EEG analysis model biases

Adaptive Cognitive Interaction Systems

Adaptive kognitive Interaktionssysteme beobachten und modellieren den Zustand ihres Benutzers und passen das Systemverhalten entsprechend an. Ein solches System besteht aus drei Komponenten: Dem empirischen kognitiven Modell, dem komputationalen kognitiven Modell und dem adaptiven Interaktionsmanager. Die vorliegende Arbeit enthält zahlreiche Beiträge zur Entwicklung dieser Komponenten sowie zu deren Kombination. Die Ergebnisse werden in zahlreichen Benutzerstudien validiert

Étude de corrélats électrophysiologiques pour la discrimination d'états de fatigue et de charge mentale : apports pour les interfaces cerveau-machine passives

Mental state estimation on the basis of cerebral activity and its resulting physiological activities has become a challenge for passive Brain-Computer Interfaces (BCI), in particular to address a need in neuroergonomics. This thesis work focuses on mental fatigue and workload estimation. Its purpose is to provide efficient and realistic processing chains. Thus, one issue was the modulation of workload markers as well as classification performance robustness depending on time-on-task (TOT). The impact of workload and TOT on attentional state markers was also assessed. For those purposes, an experimental protocol was implemented to collect the electroencephalographic (EEG), cardiac (ECG) and ocular (EOG) signals from healthy volunteers as they performed for a prolonged period of time a task that mixes working memory load and selective attention. Efficient signal processing chains that include spatial filtering and classification steps were designed in order to better estimate these mental states. The relevance of several electrophysiological markers was compared, among which spontaneous EEG activity and event-related potentials (ERPs), as well as various preprocessing steps such as spatial filtering methods for ERPs. Interaction effects between mental states were brought to light. In particular, TOT negatively impacted mental workload estimation when using power features. However, the chain based on ERPs was robust to this effect. A comparison of the type of stimuli that can be used to elicit the ERPs revealed that task-independent probes still allow very high performance, which shows their relevance for real-life implementation. Lastly, ongoing work that aims at assessing task-robust workload markers, as well as the usefulness of auditory ERPs in a single-stimulus paradigm will be presented as prospects.L'estimation de l'état mental d'un individu sur la base de son activité cérébrale et de ses activités physiologiques résultantes est devenue l'un des challenges des interfaces cerveau-machine (ICM) dites passives, dans le but notamment de répondre à un besoin en neuroergonomie. Ce travail de thèse se focalise sur l'estimation des états de fatigue et de charge mentale. Son objectif est de proposer des chaines de traitement efficaces et réalistes dans leur mise en œuvre. Ainsi, un des points à l'étude a été la modulation des indicateurs de charge ainsi que la robustesse des performances de classification en fonction du temps passé sur une tâche (TPT). L'impact de la charge et du TPT sur les marqueurs d'état attentionnel a aussi été évalué. Pour ce faire, un protocole expérimental a été mis en œuvre afin de recueillir les signaux électro-encéphalographiques (EEG), cardiaques (ECG) et oculaires (EOG) de participants volontaires sains lors de la réalisation prolongée d'une tâche combinant charge en mémoire de travail et attention sélective. Des chaînes de traitement performantes incluant une étape de filtrage spatial et une classification supervisée ont été mises en place afin de classer au mieux ces états. La pertinence de plusieurs marqueurs électrophysiologiques a été comparée, notamment l'activité EEG spontanée et les potentiels évoqués (PEs), ainsi que différentes étapes de prétraitement dont les méthodes de filtrage spatial pour PEs. Des effets d'interactions ont été mis au jour entre les différents états mentaux, dont un effet négatif du TPT sur les performances en classification de la charge mentale lorsque l'on utilise des marqueurs mesurant la puissance moyenne de l'EEG dans des bandes de fréquence d'intérêt. La chaîne basée sur les PEs est en revanche robuste à cet effet. Une comparaison du type de stimuli utilisables pour éliciter les PEs a révélé que des stimuli tâche-indépendants permettent tout de même d'obtenir des performances très élevées, ce qui montre leur pertinence pour une implémentation en situation réelle. En perspective seront présentés des travaux en cours visant à mettre en évidence des marqueurs de charge mentale robustes à la tâche, ainsi que l'utilité des potentiels évoqués auditifs en paradigme de simple stimulus