Toward Mental Effort Measurement Using Electrodermal Activity Features

The ability to monitor mental effort during a task using a wearable sensor may improve productivity for both work and study. The use of the electrodermal activity (EDA) signal for tracking mental effort is an emerging area of research. Through analysis of over 92 h of data collected with the Empatica E4 on a single participant across 91 different activities, we report on the efficacy of using EDA features getting at signal intensity, signal dispersion, and peak intensity for prediction of the participant\u27s self-reported mental effort. We implemented the logistic regression algorithm as an interpretable machine learning approach and found that features related to signal intensity and peak intensity were most useful for the prediction of whether the participant was in a self-reported high mental effort state; increased signal and peak intensity were indicative of high mental effort. When cross-validated by activity moderate predictive efficacy was achieved (AUC = 0.63, F1 = 0.63, precision = 0.64, recall = 0.63) which was significantly stronger than using the model bias alone. Predicting mental effort using physiological data is a complex problem, and our findings add to research from other contexts showing that EDA may be a promising physiological indicator to use for sensor-based self-monitoring of mental effort throughout the day. Integration of other physiological features related to heart rate, respiration, and circulation may be necessary to obtain more accurate predictions

Classification of a Driver's cognitive workload levels using artificial neural network on ECG signals

An artificial neural network (ANN) model was developed in the present study to classify the level of a driver's cognitive workload based on electrocardiography (ECG). ECG signals were measured on 15 male participants while they performed a simulated driving task as a primary task with/without an N-back task as a secondary task. Three time-domain ECG measures (mean inter-beat interval (IBI), standard deviation of IBIs, and root mean squared difference of adjacent IBIs) and three frequencydomain ECG measures (power in low frequency, power in high frequency, and ratio of power in low and high frequencies) were calculated. To compensate for individual differences in heart response during the driving tasks, a three-step data processing procedure was performed to ECG signals of each participant: (1) selection of two most sensitive ECG measures, (2) definition of three (low, medium, and high) cognitive workload levels, and (3) normalization of the selected ECG measures. An ANN model was constructed using a feed-forward network and scaled conjugate gradient as a back-propagation learning rule. The accuracy of the ANN classification model was found satisfactory for learning data (95%) and testing data (82%). ? 2016 Elsevier Ltd115Nsciessciscopu

EEG-Based Multi-Class Workload Identification Using Feature Fusion and Selection

The effectiveness of workload identification is one of the critical aspects in a monitoring instrument of mental state. In this field, the workload is usually recognised as binary classes. There are scarce studies towards multi-class workload identification because the challenge of the success of workload identification is much tough, even though one more workload class is added. Besides, most of the existing studies only utilized spectral power features from individual channels but ignoring abundant inter-channel features that represent the interactions between brain regions. In this study, we utilized features representing intra-channel information and inter-channel information to classify multiple classes of workload based on EEG. We comprehensively compared each category of features contributing to workload identification and elucidated the roles of feature fusion and feature selection for the workload identification. The results demonstrated that feature combination (83.12% in terms of accuracy) enhanced the classification performance compared to individual feature categories (i.e., band power features, 75.90%; connection features, 81.72%, in terms of accuracy). With the F-score feature selection, the classification accuracy was further increased to 83.47%. When the features of graph metric were fused, the accuracy was reached to 84.34%. Our study provided comprehensive performance comparisons between methods and feature categories for the multi-class workload identification and demonstrated that feature selection and fusion played an important role in the enhancement of workload identification. These results could facilitate further studies of multi-class workload identification and practical application of workload identification

Modeling engagement with multimodal multisensor data: the continuous performance test as an objective tool to track flow

Engagement is one of the most important factors in determining successful outcomes and deep learning in students. Existing approaches to detect student engagement involve periodic human observations that are subject to inter-rater reliability. Our solution uses real-time multimodal multisensor data labeled by objective performance outcomes to infer the engagement of students. The study involves four students with a combined diagnosis of cerebral palsy and a learning disability who took part in a 3-month trial over 59 sessions. Multimodal multisensor data were collected while they participated in a continuous performance test. Eye gaze, electroencephalogram, body pose, and interaction data were used to create a model of student engagement through objective labeling from the continuous performance test outcomes. In order to achieve this, a type of continuous performance test is introduced, the Seek-X type. Nine features were extracted including high-level handpicked compound features. Using leaveone-out cross-validation, a series of different machine learning approaches were evaluated. Overall, the random forest classification approach achieved the best classification results. Using random forest, 93.3% classification for engagement and 42.9% accuracy for disengagement were achieved. We compared these results to outcomes from different models: AdaBoost, decision tree, k-Nearest Neighbor, naïve Bayes, neural network, and support vector machine. We showed that using a multisensor approach achieved higher accuracy than using features from any reduced set of sensors. We found that using high-level handpicked features can improve the classification accuracy in every sensor mode. Our approach is robust to both sensor fallout and occlusions. The single most important sensor feature to the classification of engagement and distraction was shown to be eye gaze. It has been shown that we can accurately predict the level of engagement of students with learning disabilities in a real-time approach that is not subject to inter-rater reliability, human observation or reliant on a single mode of sensor input. This will help teachers design interventions for a heterogeneous group of students, where teachers cannot possibly attend to each of their individual needs. Our approach can be used to identify those with the greatest learning challenges so that all students are supported to reach their full potential

Assessing cognitive workload using cardiovascular measures and voice

Monitoring cognitive workload has the potential to improve both the performance and fidelity of human decision making. However, previous efforts towards discriminating further than binary levels (e.g., low/high or neutral/high) in cognitive workload classification have not been successful. This lack of sensitivity in cognitive workload measurements might be due to individual differences as well as inadequate methodology used to analyse the measured signal. In this paper, a method that combines the speech signal with cardiovascular measurements for screen and heartbeat classification is introduced. For validation, speech and cardiovascular signals from 97 university participants and 20 airline pilot participants were collected while cognitive stimuli of varying difficulty level were induced with the Stroop colour/word test. For the trinary classification scheme (low, medium, high cognitive workload) the prominent result using classifiers trained on each participant achieved 15.17 ± 0.79% and 17.38 ± 1.85% average misclassification rates indicating good discrimination at three levels of cognitive workload. Combining cardiovascular and speech measures synchronized to each heartbeat and consolidated with short-term dynamic measures might therefore provide enhanced sensitivity in cognitive workload monitoring. The results show that the influence of individual differences is a limiting factor for a generic classification and highlights the need for research to focus on methods that incorporate individual differences to achieve even better results. This method can potentially be used to measure and monitor workload in real time in operational environments

A Quantitative analysis of the mental workload demands of MRAP vehicle drivers using physiological, subjective, and performance assessments

United States Special Operations Command (USSOCOM) Operators and vehicle Commanders are specially trained United States military Warfighters that have the demanding task of operating or working onboard Mine Resistant Ambush Protected (MRAP) All Terrain Vehicles (M-ATVs). Their missions encounter significant mental demands resulting from fatigue, highly stressful situations, and interactions with Government Furnished Equipment (GFE). Excessive mental demands can be the primary factor leading to compromised vehicle communication, missed improvised explosive device (IED) detection, and increased incidents of vehicle roll-over. Research has demonstrated the consequences of mental overloading including increased errors, performance decrements, distraction, cognitive tunneling and inadequate time to appropriately process information. The objectives of this thesis were to evaluate the extent to which task-related factors impact the mental workload of Warfighters and to evaluate the consistency among the three categories of mental workload metrics. The 14 participants studied in this research were Marine Corps personnel who had heavy vehicle driving experience. Physiological, subjective and performance measures were collected during a four-segment course that progressed in difficulty and analyzed across all participants to assess changes in mental workload. It was found that task-related factors impacted the mental workload of Warfighters. The subjective metric was able to capture changes in workload more accurately than biosignals. Due to technical problems with the biosignal data, comparison of consistency across metrics was inconclusive. The subjective workload ratings were significantly different between course segments and experience levels. The experiment resulted in workload ratings that increased by as much as 94% between segments and were 18% higher among novice drivers. This study showed that mental workload fluctuates while driving in a stressful situation, despite training and experience, and consequently, detection performance will be impacted which could have very adverse consequences. There is the need for additional research to have a better understanding of the true impact of mental workload on MRAP vehicle drivers, especially in an operational environment

Proceedings of the Seventeenth Annual Conference on Manual Control

Manual control is considered, with concentration on perceptive/cognitive man-machine interaction and interface