Measuring cognitive load and cognition: metrics for technology-enhanced learning

This critical and reflective literature review examines international research published over the last decade to summarise the different kinds of measures that have been used to explore cognitive load and critiques the strengths and limitations of those focussed on the development of direct empirical approaches. Over the last 40 years, cognitive load theory has become established as one of the most successful and influential theoretical explanations of cognitive processing during learning. Despite this success, attempts to obtain direct objective measures of the theory's central theoretical construct – cognitive load – have proved elusive. This obstacle represents the most significant outstanding challenge for successfully embedding the theoretical and experimental work on cognitive load in empirical data from authentic learning situations. Progress to date on the theoretical and practical approaches to cognitive load are discussed along with the influences of individual differences on cognitive load in order to assess the prospects for the development and application of direct empirical measures of cognitive load especially in technology-rich contexts

Framework for Electroencephalography-based Evaluation of User Experience

Measuring brain activity with electroencephalography (EEG) is mature enough to assess mental states. Combined with existing methods, such tool can be used to strengthen the understanding of user experience. We contribute a set of methods to estimate continuously the user's mental workload, attention and recognition of interaction errors during different interaction tasks. We validate these measures on a controlled virtual environment and show how they can be used to compare different interaction techniques or devices, by comparing here a keyboard and a touch-based interface. Thanks to such a framework, EEG becomes a promising method to improve the overall usability of complex computer systems.Comment: in ACM. CHI '16 - SIGCHI Conference on Human Factors in Computing System, May 2016, San Jose, United State

Mixed Initiative Systems for Human-Swarm Interaction: Opportunities and Challenges

Human-swarm interaction (HSI) involves a number of human factors impacting human behaviour throughout the interaction. As the technologies used within HSI advance, it is more tempting to increase the level of swarm autonomy within the interaction to reduce the workload on humans. Yet, the prospective negative effects of high levels of autonomy on human situational awareness can hinder this process. Flexible autonomy aims at trading-off these effects by changing the level of autonomy within the interaction when required; with mixed-initiatives combining human preferences and automation's recommendations to select an appropriate level of autonomy at a certain point of time. However, the effective implementation of mixed-initiative systems raises fundamental questions on how to combine human preferences and automation recommendations, how to realise the selected level of autonomy, and what the future impacts on the cognitive states of a human are. We explore open challenges that hamper the process of developing effective flexible autonomy. We then highlight the potential benefits of using system modelling techniques in HSI by illustrating how they provide HSI designers with an opportunity to evaluate different strategies for assessing the state of the mission and for adapting the level of autonomy within the interaction to maximise mission success metrics.Comment: Author version, accepted at the 2018 IEEE Annual Systems Modelling Conference, Canberra, Australi

Modeling cognitive load as a self-supervised brain rate with electroencephalography and deep learning

The principal reason for measuring mental workload is to quantify the cognitive cost of performing tasks to predict human performance. Unfortunately, a method for assessing mental workload that has general applicability does not exist yet. This research presents a novel self-supervised method for mental workload modelling from EEG data employing Deep Learning and a continuous brain rate, an index of cognitive activation, without requiring human declarative knowledge. This method is a convolutional recurrent neural network trainable with spatially preserving spectral topographic head-maps from EEG data to fit the brain rate variable. Findings demonstrate the capacity of the convolutional layers to learn meaningful high-level representations from EEG data since within-subject models had a test Mean Absolute Percentage Error average of 11%. The addition of a Long-Short Term Memory layer for handling sequences of high-level representations was not significant, although it did improve their accuracy. Findings point to the existence of quasi-stable blocks of learnt high-level representations of cognitive activation because they can be induced through convolution and seem not to be dependent on each other over time, intuitively matching the non-stationary nature of brain responses. Across-subject models, induced with data from an increasing number of participants, thus containing more variability, obtained a similar accuracy to the within-subject models. This highlights the potential generalisability of the induced high-level representations across people, suggesting the existence of subject-independent cognitive activation patterns. This research contributes to the body of knowledge by providing scholars with a novel computational method for mental workload modelling that aims to be generally applicable, does not rely on ad-hoc human-crafted models supporting replicability and falsifiability.Comment: 18 pages, 12 figures, 1 tabl

Brain enhancement through cognitive training: A new insight from brain connectome

Owing to the recent advances in neurotechnology and the progress in understanding of brain cognitive functions, improvements of cognitive performance or acceleration of learning process with brain enhancement systems is not out of our reach anymore, on the contrary, it is a tangible target of contemporary research. Although a variety of approaches have been proposed, we will mainly focus on cognitive training interventions, in which learners repeatedly perform cognitive tasks to improve their cognitive abilities. In this review article, we propose that the learning process during the cognitive training can be facilitated by an assistive system monitoring cognitive workloads using electroencephalography (EEG) biomarkers, and the brain connectome approach can provide additional valuable biomarkers for facilitating leaners' learning processes. For the purpose, we will introduce studies on the cognitive training interventions, EEG biomarkers for cognitive workload, and human brain connectome. As cognitive overload and mental fatigue would reduce or even eliminate gains of cognitive training interventions, a real-time monitoring of cognitive workload can facilitate the learning process by flexibly adjusting difficulty levels of the training task. Moreover, cognitive training interventions should have effects on brain sub-networks, not on a single brain region, and graph theoretical network metrics quantifying topological architecture of the brain network can differentiate with respect to individual cognitive states as well as to different individuals' cognitive abilities, suggesting that the connectome is a valuable approach for tracking the learning progress. Although only a few studies have exploited the connectome approach for studying alterations of the brain network induced by cognitive training interventions so far, we believe that it would be a useful technique for capturing improvements of cognitive function

Évaluation de la charge mentale des pilotes en manœuvre aérienne

La charge de travail cognitive d'un pilote d'aviation, qui englobe sa capacité mentale à effectuer les manœuvres d'un avion, varie selon l’étape de pilotage et le nombre de tâches convergeant simultanément sur le pilote. Cette charge de travail peut entraîner des erreurs de pilotage aux conséquences graves. La plupart des erreurs se produisent pendant la procédure de décollage ou d'atterrissage. Cette étude vise à mesurer et prédire la charge de travail cognitive d'un pilote lors d'une procédure de décollage afin de mieux comprendre et potentiellement prévenir ces erreurs humaines. Pour y parvenir, nous avons créé une solution logicielle pour mesurer et surveiller en temps réel la fréquence cardiaque, la dilatation pupillaire et la charge de travail cognitive d'un pilote. Le logiciel est également capable de déclencher des événements de défaillance pour déclencher une modification de la charge de travail cognitive à la demande. À l'aide d'un casque électroencéphalogramme (EEG) permettant de mesurer l’activité électrique du cerveau, d'un moniteur de fréquence cardiaque, d'un eye tracker et d'un simulateur, nous avons créé une configuration d'environnement où les pilotes devaient faire décoller un avion A320 avec et sans pannes sans le savoir au préalable. Cette étude a rassemblé 136 décollages sur 13 pilotes pour plus de 9 heures de données de séries chronologiques, soit 2 millions de lignes combinées. De plus, nous avons étudié la relation entre la fréquence cardiaque, la dilatation de la pupille et la charge de travail cognitive lors d'une tâche critique. Cette étude a révélé, à l'aide d'une analyse statistique, qu'un moment critique, comme une panne de moteur, augmente la fréquence cardiaque, la dilatation de la pupille et la charge de travail cognitive d'un pilote. Ensuite, cette recherche a utilisé différents modèles d'apprentissage automatique et d'apprentissage en profondeur pour prédire la charge de travail cognitive d'un pilote pendant le décollage. Nous avons constaté qu'en utilisant un modèle d'apprentissage en profondeur long short-term memory empilé, nous étions en mesure de prédire la charge de travail cognitive 5 secondes dans le futur. Le modèle long short-term memory empilé a donné une erreur quadratique moyenne (MSE) de 44,09, une erreur racine quadratique moyenne (RMSE) de 6,64 et une erreur absolue moyenne de 5,28, prouvant qu'il est possible de prédire la charge de travail cognitive.The cognitive workload for an aviation pilot, which englobes a pilot's mental capacity to perform aircraft's maneuvers, varies according to the piloting stage and the number of tasks converging simultaneously on the pilot. This workload can lead to piloting errors with severe consequences, where most errors occur during the takeoff or landing procedure. This study aims to predict the cognitive workload of a pilot during a takeoff procedure in order to better understand and potentially prevent these human errors. To achieve this, we created a software solution to measure and monitor in real-time the heart rate, pupil dilation, and cognitive workload of a pilot. The software is also capable of triggering failure events to trigger a change in cognitive workload on demand. Using an electroencephalogram (EEG) headset which measures the electrical brain activity, a heart rate monitor, an eye tracker, and a simulator, we created an environment setup where pilots had to take off an A320 airplane with and without failures without priorly knowing it. This study gathered 136 takeoffs across 13 pilots for more than 9 hours of time-series data, or 2 million rows combined. Moreover, we investigated the relation between heart rate, pupil dilation, and cognitive workload during a critical task. This study found, using statistical F-test analysis, that a critical moment, such as an engine failure, augments the heart rate, pupil dilation, and cognitive workload of a pilot. Next, this research utilized different machine learning and deep learning models to predict the cognitive workload of a pilot during takeoff. We found that, when using a stacked long short-term memory deep learning model, we were able to predict future cognitive workload 5 seconds into the future. The stacked long short-term memory model resulted in a mean square error of 44.09, a root mean square error of 6.64, and an mean absolute error of 5.28, demonstrating that it is possible to predict future cognitive workload

TOBE: Tangible Out-of-Body Experience

We propose a toolkit for creating Tangible Out-of-Body Experiences: exposing the inner states of users using physiological signals such as heart rate or brain activity. Tobe can take the form of a tangible avatar displaying live physiological readings to reflect on ourselves and others. Such a toolkit could be used by researchers and designers to create a multitude of potential tangible applications, including (but not limited to) educational tools about Science Technologies Engineering and Mathematics (STEM) and cognitive science, medical applications or entertainment and social experiences with one or several users or Tobes involved. Through a co-design approach, we investigated how everyday people picture their physiology and we validated the acceptability of Tobe in a scientific museum. We also give a practical example where two users relax together, with insights on how Tobe helped them to synchronize their signals and share a moment

Noninvasive Physiological Measures And Workload Transitions:an Investigation Of Thresholds Using Multiple Synchronized Sensors

The purpose of this study is to determine under what conditions multiple minimally intrusive physiological sensors can be used together and validly applied for use in areas which rely on adaptive systems including adaptive automation and augmented cognition. Specifically, this dissertation investigated the physiological transitions of operator state caused by changes in the level of taskload. Three questions were evaluated including (1) Do differences exist between physiological indicators when examined between levels of difficulty? (2) Are differences of physiological indicators (which may exist) between difficulty levels affected by spatial ability? (3) Which physiological indicators (if any) account for variation in performance on a spatial task with varying difficulty levels? The Modular Cognitive State Gauge model was presented and used to determine which basic physiological sensors (EEG, ECG, EDR and eye-tracking) could validly assess changes in the utilization of two-dimensional spatial resources required to perform a spatial ability dependent task. Thirty-six volunteers (20 female, 16 male) wore minimally invasive physiological sensing devices while executing a challenging computer based puzzle task. Specifically, participants were tested with two measures of spatial ability, received training, a practice session, an experimental trial and completed a subjective workload survey. The results of this experiment confirmed that participants with low spatial ability reported higher subjective workload and performed poorer when compared to those with high spatial ability. Additionally, there were significant changes for a majority of the physiological indicators between two difficulty levels and most importantly three measures (EEG, ECG and eye-tracking) were shown to account for variability in performance on the spatial task

Assessing the Effectiveness of Workload Measures in the Nuclear Domain

An operator\u27s performance and mental workload when interacting with a complex system, such as the main control room (MCR) of a nuclear power plant (NPP), are major concerns when seeking to accomplish safe and successful operations. The impact of performance on operator workload is one of the most widely researched areas in human factors science with over five hundred workload articles published since the 1960s (Brannick, Salas, & Prince, 1997; Meshkati & Hancock, 2011). Researchers have used specific workload measures across domains to assess the effects of taskload. However, research has not sufficiently assessed the psychometric properties, such as reliability, validity, and sensitivity, which delineates and limits the roles of these measures in workload assessment (Nygren, 1991). As a result, there is no sufficiently effective measure for indicating changes in workload for distinct tasks across multiple domains (Abich, 2013). Abich (2013) was the most recent to systematically test the subjective and objective workload measures for determining the universality and sensitivity of each alone or in combination. This systematic approach assessed taskload changes within three tasks in the context of a military intelligence, surveillance, and reconnaissance (ISR) missions. The purpose for the present experiment was to determine if certain workload measures are sufficiently effective across domains by taking the findings from one domain (military) and testing whether those results hold true in a different domain, that of nuclear. Results showed that only two measures (NASA-TLX frustration and fNIR) were sufficiently effective at indicating workload changes between the three task types in the nuclear domain, but many measures were statistically significant. The results of this research effort combined with the results from Abich (2013) highlight an alarming problem. The ability of subjective and physiological measures to indicate changes in workload varies across tasks (Abich, 2013) and across domain. A single measure is not able to measure the complex construct of workload across different tasks within the same domain or across domains. This research effort highlights the importance of proper methodology. As researchers, we have to identify the appropriate workload measure for all tasks regardless of the domain by investigating the effectiveness of each measure. The findings of the present study suggest that responsible science include evaluating workload measures before use, not relying on prior research or theory. In other words, results indicate that it is only acceptable to use a measure based on prior findings if research has tested that measure on the exact task and manipulations within that specific domain

Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals

Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usa-bility qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.Comment: Published in INTERACT, Sep 2015, Bamberg, German