Assessment of mental workload across cognitive tasks using a passive brain-computer interface based on mean negative theta-band amplitudes

Brain-computer interfaces (BCI) can provide real-time and continuous assessments of mental workload in different scenarios, which can subsequently be used to optimize human-computer interaction. However, assessment of mental workload is complicated by the task-dependent nature of the underlying neural signals. Thus, classifiers trained on data from one task do not generalize well to other tasks. Previous attempts at classifying mental workload across different cognitive tasks have therefore only been partially successful. Here we introduce a novel algorithm to extract frontal theta oscillations from electroencephalographic (EEG) recordings of brain activity and show that it can be used to detect mental workload across different cognitive tasks. We use a published data set that investigated subject dependent task transfer, based on Filter Bank Common Spatial Patterns. After testing, our approach enables a binary classification of mental workload with performances of 92.00 and 92.35%, respectively for either low or high workload vs. an initial no workload condition, with significantly better results than those of the previous approach. It, nevertheless, does not perform beyond chance level when comparing high vs. low workload conditions. Also, when an independent component analysis was done first with the data (and before any additional preprocessing procedure), even though we achieved more stable classification results above chance level across all tasks, it did not perform better than the previous approach. These mixed results illustrate that while the proposed algorithm cannot replace previous general-purpose classification methods, it may outperform state-of-the-art algorithms in specific (workload) comparisons

Tracing Pilots’ Situation Assessment by Neuroadaptive Cognitive Modeling

This study presents the integration of a passive brain-computer interface (pBCI) and cognitive modeling as a method to trace pilots’ perception and processing of auditory alerts and messages during operations. Missing alerts on the flight deck can result in out-of-the-loop problems that can lead to accidents. By tracing pilots’ perception and responses to alerts, cognitive assistance can be provided based on individual needs to ensure they maintain adequate situation awareness. Data from 24 participating aircrew in a simulated flight study that included multiple alerts and air traffic control messages in single pilot setup are presented. A classifier was trained to identify pilots’ neurophysiological reactions to alerts and messages from participants’ electroencephalogram (EEG). A neuroadaptive ACT-R model using EEG data was compared to a conventional normative model regarding accuracy in representing individual pilots. Results show that passive BCI can distinguish between alerts that are processed by the pilot as task-relevant or irrelevant in the cockpit based on the recorded EEG. The neuroadaptive model’s integration of this data resulted in significantly higher performance of 87% overall accuracy in representing individual pilots’ responses to alerts and messages compared to 72% accuracy of a normative model that did not consider EEG data. We conclude that neuroadaptive technology allows for implicit measurement and tracing of pilots’ perception and processing of alerts on the flight deck. Careful handling of uncertainties inherent to passive BCI and cognitive modeling shows how the representation of pilot cognitive states can be improved iteratively for providing assistance.TU Berlin, Open-Access-Mittel – 202

Good scientific practice in MEEG Research: Progress and Perspectives

Good Scientific Practice (GSP) refers to both explicit and implicit rules, recommendations, and guidelines that help scientists to produce work that is of the highest quality at any given time, and to efficiently share that work with the community for further scrutiny or utilization.For experimental research using magneto- and electroencephalography (MEEG), GSP includes specific standards and guidelines for technical competence, which are periodically updated and adapted to new findings. However, GSP also needs to be periodically revisited in a broader light. At the LiveMEEG 2020 conference, a reflection on GSP was fostered that included explicitly documented guidelines and technical advances, but also emphasized intangible GSP: a general awareness of personal, organizational, and societal realities and how they can influence MEEG research.This article provides an extensive report on most of the LiveMEEG contributions and new literature, with the additional aim to synthesize ongoing cultural changes in GSP. It first covers GSP with respect to cognitive biases and logical fallacies, pre-registration as a tool to avoid those and other early pitfalls, and a number of resources to enable collaborative and reproducible research as a general approach to minimize misconceptions. Second, it covers GSP with respect to data acquisition, analysis, reporting, and sharing, including new tools and frameworks to support collaborative work. Finally, GSP is considered in light of ethical implications of MEEG research and the resulting responsibility that scientists have to engage with societal challenges.Considering among other things the benefits of peer review and open access at all stages, the need to coordinate larger international projects, the complexity of MEEG subject matter, and today's prioritization of fairness, privacy, and the environment, we find that current GSP tends to favor collective and cooperative work, for both scientific and for societal reasons

Neuroadaptive Technologie: Konzepte, Methoden, und Validierungen

This dissertation presents conceptual, methodological, and experimental advances in the field of neuroadaptive technology. Neuroadaptive technology refers to the category of technology that uses implicit input obtained from brain activity in order to adapt itself, e.g. to enable implicit interaction. Implicit input refers to any input obtained by a receiver that was not intended as such by the sender. Neuroadaptive technology thus detects naturally-occurring brain activity that was not intended for communication or control, and uses it to enable novel human-computer interaction paradigms. Part I of this dissertation presents different categories of neuroadaptive systems, and introduces cognitive probing, a method in which the technology deliberately elicits a brain response from the user in order to learn from it. Part II introduces two tools to help validate some core methods related to neuroadaptive technology: SEREEGA, with which electroencephalographic data can be simulated, and a classifier visualisation technique revealing which (cortical) areas a brain-computer interface classifier focused on. Finally, Part III presents two experimental studies illustrating and validating the technology described in Part I using the methods from Part II. In particular, it is demonstrated how neuroadaptive technology can be used to enable implicit cursor control using cognitive probing. Additional experimentation revealed that brain activity elicited by cursor movements can reflect internal, subjective interpretations. These experiments thus highlight both the potential benefits and the potential ethical, legal, and societal concerns of neuroadaptive technology.In dieser Dissertation werden konzeptuelle, methodologische, und experimentelle Fortschritte auf dem Gebiet der neuroadaptiven Technologie vorgestellt. Neuroadaptive Technologie bezieht sich auf die Kategorie der Technologien, die impliziten Input aus der Hirnaktivität unter Verwendung einer passiven Hirn-Computer-Schnittstelle verwenden, um sich selbst anzupassen, z.B. um implizite Interaktion zu ermöglichen. Impliziter Input bezeichnet jede Eingabe, die von einem Empfänger erhalten wird, jedoch von dem Sender nicht als solche beabsichtigt war. Die neuroadaptive Technologie erkennt also natürlich auftretende Hirnaktivität, die nicht für Kommunikation oder Kontrolle gedacht war, und nutzt sie, um neuartige Mensch-Computer-Interaktionsparadigmen zu ermöglichen. Teil I dieser Dissertation stellt verschiedene Kategorien von neuroadaptiven Systemen vor und führt cognitive probing ('kognitive Sondierung') ein: eine Methode, bei der die Technologie dem Benutzer absichtlich eine Gehirnreaktion entlockt, um von ihr zu lernen. Teil II stellt zwei Werkzeuge vor, die bei der Validierung einiger Kernmethoden der neuroadaptiven Technologie helfen sollen: SEREEGA, mit dem elektroenzephalographische Daten simuliert werden können, und eine Klassifikator-Visualisierungsmethode um zu erkennen, auf welche (kortikalen) Bereiche sich ein Klassifikator konzentriert. Schließlich werden in Teil III zwei experimentelle Studien vorgestellt, die die in Teil I beschriebene Technologien mit den Methoden aus Teil II veranschaulichen und validieren. Insbesondere wird gezeigt, wie neuroadaptive Technologie eingesetzt werden kann, um implizite Cursorsteuerung mittels cognitive probing zu ermöglichen. Zusätzliche Experimente zeigten, dass die durch die Cursorbewegungen hervorgerufene Hirnaktivität interne, subjektive Interpretationen widerspiegeln kann. Diese Experimente verdeutlichen somit sowohl den potenziellen Nutzen als auch die möglichen ethischen, rechtlichen, und gesellschaftlichen Bedenken, die in Teil I ebenfalls angesprochen wurden

Towards task-independent workload classification: Shifting from binary to continuous classification

Passive Brain-Computer-Interfaces provide a promising approach to the continuous measurement of mental workload in realistic scenarios. Typically, a BCI is calibrated to discriminate between different levels of workload induced by a specific task. However, workload in realistic scenarios is typically a result of a mixture of different tasks. Here, we present a study on investigating the possibility of a task-independent classifier, which can be applied to classify mental workload induced by various tasks (including n-back, backward span, addition, word recovery and mental rotation). Furthermore, our approach is not limited to binary classification of workload but can discriminate it on a continuous metric

Hybrid brain-computer interface with motor imagery and error-related brain activity

Objective. Brain-computer interface (BCI) systems read and interpret brain activity directly from the brain. They can provide a means of communication or locomotion for patients suffering from neurodegenerative diseases or stroke. However, non-stationarity of brain activity limits the reliable transfer of the algorithms that were trained during a calibration session to real-time BCI control. One source of non-stationarity is the user's brain response to the BCI output (feedback), for instance, whether the BCI feedback is perceived as an error by the user or not. By taking such sources of non-stationarity into account, the reliability of the BCI can be improved. Approach. In this work, we demonstrate a real-time implementation of a hybrid motor imagery BCI combining the information from the motor imagery signal and the error-related brain activity simultaneously so as to gain benefit from both sources. Main results. We show significantly improved performance in real-time BCI control across 12 participants, compared to a conventional motor imagery BCI. The significant improvement is in terms of classification accuracy, target hit rate, subjective perception of control and information-transfer rate. Moreover, our offline analyses of the recorded EEG data show that the error-related brain activity provides a more reliable source of information than the motor imagery signal. Significance. This work shows, for the first time, that the error-related brain activity classifier compared to the motor imagery classifier is more consistent when trained on calibration data and tested during online control. This likely explains why the proposed hybrid BCI allows for a more reliable means of communication or rehabilitation for patients in need

Cognitive and affective probing: a tutorial and review of active learning for neuroadaptive technology

Objective. The interpretation of neurophysiological measurements has a decades-long history, culminating in current real-time brain-computer interfacing (BCI) applications for both patient and healthy populations. Over the course of this history, one focus has been on the investigation of cortical responses to specific stimuli. Such responses can be informative with respect to the human user's mental state at the time of presentation. An ability to decode neurophysiological responses to stimuli in real time becomes particularly powerful when combined with a simultaneous ability to autonomously produce such stimuli. This allows a computer to gather stimulus-response samples and iteratively produce new stimuli based on the information gathered from previous samples, thus acquiring more, and more specific, information. This information can even be obtained without the explicit, voluntary involvement of the user. Approach. We define cognitive and affective probing, referring to an application of active learning where repeated sampling is done by eliciting implicit brain responses. In this tutorial, we provide a definition of this method that unifies different past and current implementations based on common aspects. We then discuss a number of aspects that differentiate various possible implementations of cognitive probing. Main results. We argue that a key element is the user model, which serves as both information storage and basis for subsequent probes. Cognitive probing can be used to continuously and autonomously update this model, refining the probes, and obtaining increasingly detailed or accurate information from the resulting brain activity. In contrast to a number of potential advantages of the method, cognitive probing may also pose a threat to informed consent, our privacy of thought, and our ability to assign responsibility to actions mediated by the system. Significance. This tutorial provides guidelines to both implement, and critically discuss potential ethical implications of, novel cognitive probing applications and research endeavours