Electroencephalography (EEG)-based Brain-Computer Interfaces

International audienceBrain-Computer Interfaces (BCI) are systems that can translate the brain activity patterns of a user into messages or commands for an interactive application. The brain activity which is processed by the BCI systems is usually measured using Electroencephalography (EEG). In this article, we aim at providing an accessible and up-to-date overview of EEG-based BCI, with a main focus on its engineering aspects. We notably introduce some basic neuroscience background, and explain how to design an EEG-based BCI, in particular reviewing which signal processing, machine learning, software and hardware tools to use. We present Brain Computer Interface applications, highlight some limitations of current systems and suggest some perspectives for the field

Advanced Augmentative and Alternative Communication System Based in Physiological Control

Dyskinetic Cerebral Palsy (DCP) is mainly characterized by alterations in muscle tone and involuntary movements. Therefore, these people present with difficulties in coordination and movement control, which makes walking difficult and affects their posture when seated. Additionally, their cognitive performance varies between being completely normal and severe mental retardation. People with DCP were selected as the objective of this thesis due to their multiple and complex limitations (speech problems and motor control) and because their capabilities have a great margin for improvement thanks to physiological control systems. Given their communication difficulties, some people with DCP have good motor con-trol and can communicate with written language. However, most have difficulty using Augmentative and Alternative Communication (AAC) systems. People with DCP gen-erally use concept boards to indicate the idea they want to communicate. However, most communication solutions available today are based on proprietary software that makes it difficult to customize the concept board and this type of control system. This is the motivation behind this thesis, with the aim of creating an interface with characteristics, able to be adapted to the user needs and limitations. Thus, this thesis proposes an Augmentative and Alternative Communication System for people with DCP based on physiological control. In addition, an innovative system for direct con-trol of concept boards with EMG is proposed. This control system is based on a physi-cal model that reproduces the muscular mechanical response (stiffness, inertia and viscosity). It allows for a selection of elements thanks to small pulses of EMG signal with sensors on a muscle with motor control. Its main advantage is the possibility of correcting errors during selection associated with uncontrolled muscle impulses, avoid-ing sustained muscle effort and thus reduced fatigue.La Parálisis Cerebral de tipo Discinésica (DCP) se caracteriza principalmente por las alteraciones del tono muscular y los movimientos involuntarios. Por ello, estos pacientes presentan dificultades en la coordinación y en el control de movimientos, lo cual les dificulta el caminar y afecta su postura cuando están sentados. Cabe resaltar que la capacidad cognitiva de las personas con DCP puede variar desde completamente normal, hasta un retraso mental severo. Las personas con DCP han sido seleccionadas como objetivo de esta tesis ya el margen de mejora de sus capacidades es amplio gracias a sistemas de control fisiológico, debido a sus múltiples y complejas limitaciones (problemas de habla y control motor). Debido a sus dificultades de comunicación, algunas personas con DCP se pueden comunicar con lenguaje escrito, siempre y cuando tenga un buen control motor. Sin embargo, la mayoría tienen dificultades para usar sistemas de Comunicación Aumentativos y Alternativos (AAC). De hecho, las personas con DCP utilizan generalmente tableros de conceptos para indicar la idea que quieren transmitir. Sin embargo, la mayoría las soluciones de comunicación disponibles en la actualidad están basadas en software propietario que hacen difícil la personalización del tablero de conceptos y el tipo de sistema de control. Es aquí donde surge esta tesis, con el objetivo de crear una interfaz con esas características, capaz de adaptarse a las necesidades y limitaciones del usuario. De esta forma, esta tesis propone un sistema de comunicación aumentativo y alternativo para personas con DCP basado en control fisiológico. Además, se propone un Sistema innovador de control directo sobre tableros de conceptos basado en EMG. Este Sistema de control se basa en un modelo físico que reproduce la respuesta mecánica muscular (basado en parámetros como Rigidez, Inercia y Viscosidad), permitiendo la selección de elementos gracias a pequeños pulsos de señal EMG con sensores sobre un músculo con control motor. Sus principales ventajas son la posibilidad de corregir errores durante la selección asociado a los impulsos musculares no controlados, evitar el esfuerzo muscular mantenido para alcanzar un nivel y reducir la fatiga.La Paràlisi Cerebral de tipus Discinèsica (DCP) es caracteritza principalment per les alteracions del to muscular i els moviments involuntaris. Per açò, aquests pacients presenten dificultats en la coordinació i en el control de moviments, la qual cosa els dificulta el caminar i afecta la seua postura quan estan asseguts. Cal ressaltar que la capacitat cognitiva de les persones amb DCP pot variar des de completament normal, fins a un retard mental sever. Les persones amb DCP han sigut seleccionades com a objectiu d'aquesta tesi ja el marge de millora de les seues capacitats és ampli gràcies a sistemes de control fisiològic, a causa dels seus múltiples i complexes limitacions (problemes de parla i control motor). A causa de les seues dificultats de comunicació, algunes persones amb DCP es poden comunicar amb llenguatge escrit, sempre que tinga un bon control motor. No obstant açò, la majoria tenen dificultats per a usar sistemes de Comunicació Augmentatius i Alternatius (AAC). De fet, les persones amb DCP utilitzen generalment taulers de conceptes per a indicar la idea que volen transmetre. No obstant açò, la majoria les solucions de comunicació disponibles en l'actualitat estan basades en programari propietari que fan difícil la personalització del tauler de conceptes i el tipus de sistema de control. És ací on sorgeix aquesta tesi, amb l'objectiu de crear una interfície amb aqueixes característiques, capaç d'adaptar-se a les necessitats i limitacions de l'usuari. D'aquesta forma, aquesta tesi proposa un sistema de comunicació augmentatiu i alternatiu per a persones amb DCP basat en control fisiològic. A més, es proposa un sistema innovador de control directe sobre taulers de conceptes basat en EMG. Aquest sistema de control es basa en un model físic que reprodueix la resposta mecànica muscular (basat en paràmetres com a Rigidesa, Inèrcia i Viscositat), permetent la selecció d'elements gràcies a xicotets polsos de senyal EMG amb sensors sobre un múscul amb control motor. Els seus principals avantatges són la possibilitat de corregir errors durant la selecció associat als impulsos musculars no controlats, evitar l'esforç muscular mantingut per a aconseguir un nivell i reduir la fatiga.Díaz Pineda, JA. (2017). Advanced Augmentative and Alternative Communication System Based in Physiological Control [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90418TESI

Електроенцефалографски сигнали за управљање рачунарским интерфејсом у неурорехабилитацији

Мозак-рачунар интерфејс (МоРИ) системи могу искористити карактеристичне промене мождане активности корисника као контролне сигнале уређаја (рачунара). Различити ментални задаци или спољашњи стимулуси (визуелни, аудитивни или соматосензорни) индукују промене које су кодиране у спонтаној неуралној активности. Генерисане промене се могу идентификовати мерењем можданих сигнала који представљају директну или индиректну меру електричне активности мозга...Brain Computer Interface (BCI) systems can use characteristic brain neural alterations as control signals of the device/computer. Various mental tasks or external stimulation (visual, auditory or somatosensory) induce changes which are embedded in the spontaneous neural activity. Generated changes can be extracted and identified from the brain-signal recordings that represent the (direct or indirect) measure of electrical neural activity..

EEG and ECoG features for Brain Computer Interface in Stroke Rehabilitation

The ability of non-invasive Brain-Computer Interface (BCI) to control an exoskeleton was used for motor rehabilitation in stroke patients or as an assistive device for the paralyzed. However, there is still a need to create a more reliable BCI that could be used to control several degrees of Freedom (DoFs) that could improve rehabilitation results. Decoding different movements from the same limb, high accuracy and reliability are some of the main difficulties when using conventional EEG-based BCIs and the challenges we tackled in this thesis. In this PhD thesis, we investigated that the classification of several functional hand reaching movements from the same limb using EEG is possible with acceptable accuracy. Moreover, we investigated how the recalibration could affect the classification results. For this reason, we tested the recalibration in each multi-class decoding for within session, recalibrated between-sessions, and between sessions. It was shown the great influence of recalibrating the generated classifier with data from the current session to improve stability and reliability of the decoding. Moreover, we used a multiclass extension of the Filter Bank Common Spatial Patterns (FBCSP) to improve the decoding accuracy based on features and compared it to our previous study using CSP. Sensorimotor-rhythm-based BCI systems have been used within the same frequency ranges as a way to influence brain plasticity or controlling external devices. However, neural oscillations have shown to synchronize activity according to motor and cognitive functions. For this reason, the existence of cross-frequency interactions produces oscillations with different frequencies in neural networks. In this PhD, we investigated for the first time the existence of cross-frequency coupling during rest and movement using ECoG in chronic stroke patients. We found that there is an exaggerated phase-amplitude coupling between the phase of alpha frequency and the amplitude of gamma frequency, which can be used as feature or target for neurofeedback interventions using BCIs. This coupling has been also reported in another neurological disorder affecting motor function (Parkinson and dystonia) but, to date, it has not been investigated in stroke patients. This finding might change the future design of assistive or therapeuthic BCI systems for motor restoration in stroke patients

The influence of graphical user interface on motion onset brain-computer interface performance and the effect of data augmentation on motor imagery brain-computer interface

Motor Imagery Brain Computer Interface (MI BCI) is one of the most frequently used BCI modalities, due to the versatility of its applications. However, it still has unresolved issues like time-consuming calibration, low information transfer rate, and inconsistent performance across individuals. Combining MI BCI with Motion Onset Visual Evoked Potential (mVEP) BCI in a hybrid structure may solve some of these problems. Combining MI BCI with more robust mVEP BCI, would increase the degrees of freedom thereby increasing the information transfer rate, and would also indirectly improve intrasubject consistency in performance by replacing some MI-based tasks with mVEP. Unfortunately, due to Covid -19 pandemic experimental research on hybrid BCI was not possible, therefore this thesis focuses on two BCI separately. Chapter 1 provides an overview of different BCIs modalities and the underlying neurophysiological principles, followed by the objectives of the thesis. The research contributions are also highlighted. Finally, the thesis outlines are presented at the end of this chapter. Chapter 2 presents a comprehensive state of the art to the thesis, drawing on a wide range of literature in relevant fields. Specifically, it delves into MI BCI, mVEP BCI, Deep Learning, Transfer Learning (TL), Data Augmentation (DA) and Generative Adversarial Networks (GANs). Chapter 3 investigates the effect of graphical elements, in online and offline experiments. In the offline experiment, graphical elements such as the color, size, position, and layout were explored. Replacing a default red moving bar with a green and blue bar, changing the background color from white to gray, and using smaller visual angles did not lead to statistically significant improvement in accuracy. However, the effect size of η2 (0.085) indicated a moderate effect for these changes of graphical factors. Similarly, no statistically significant difference was found for the two different layouts in online experiments. Overall, the mVEP BCI has achieved a classification accuracy of approximately 80%, and it is relatively impervious to changes in graphical interface parameters. This suggests that mVEP is a promising candidate for a hybrid BCI system combined with MI, that requires dynamic, versatile graphical design features. In Chapter 4, various DA methods are explored, including Segmentation and Recombination in Time Domain, Segmentation and Recombination in Time-Frequency Domain, and Spatial Analogy. These methods are evaluated based on three feature extraction approaches: Common Spatial Patterns, Time Domain Parameters (TDP), and Band Power. The evaluation was conducted using a validated BCI set, namely the BCI Competition IV dataset 2a, as well as a dataset obtained from our research group. The methods are effective when a small dataset of single subject are available. All three DA methods significantly affect the performance of the TDP feature extraction method. Chapter 5 explored the use of GANs for DA in combination with TL and cropped training strategies using ShallowFBCSP classifier. It also used the same validated dataset (BCI competition IV dataset 2a) as in Chapter 4. In contrast to DA method explored in Chapter 4, this DA is suitable for larger datasets and for generalizing training based on other people’s data. Applying GAN-based DA to the dataset resulted on average in a 2% improvement in average accuracy (from 68.2% to 70.7%). This study provides a novel method to enable MI GAN training with only 40 trials per participant with the rest 8 people’s data for TL, addressing the data insufficiency issue for GANs. The evaluation of generated artificial trials revealed the importance of inter-class differences in MI patterns, which can be easily identified by GANs. Overall the thesis addressed the main practical issues of both mVEP and MI BCI paving the way for their successful combination in future experiments

TOWARDS STEADY-STATE VISUALLY EVOKED POTENTIALS BRAIN-COMPUTER INTERFACES FOR VIRTUAL REALITY ENVIRONMENTS EXPLICIT AND IMPLICIT INTERACTION

In the last two decades, Brain-Computer Interfaces (BCIs) have been investigated mainly for the purpose of implementing assistive technologies able to provide new channels for communication and control for people with severe disabilities. Nevertheless, more recently, thanks to technical and scientific advances in the different research fields involved, BCIs are gaining greater attention also for their adoption by healthy users, as new interaction devices. This thesis is dedicated to to the latter goal and in particular will deal with BCIs based on the Steady State Visual Evoked Potential (SSVEP), which in previous works demonstrated to be one of the most flexible and reliable approaches. SSVEP based BCIs could find applications in different contexts, but one which is particularly interesting for healthy users, is their adoption as new interaction devices for Virtual Reality (VR) environments and Computer Games. Although being investigated since several years, BCIs still poses several limitations in terms of speed, reliability and usability with respect to ordinary interaction devices. Despite of this, they may provide additional, more direct and intuitive, explicit interaction modalities, as well as implicit interaction modalities otherwise impossible with ordinary devices. This thesis, after a comprehensive review of the different research fields being the basis of a BCI exploiting the SSVEP modality, present a state-of-the-art open source implementation using a mix of pre-existing and custom software tools. The proposed implementation, mainly aimed to the interaction with VR environments and Computer Games, has then been used to perform several experiments which are hereby described as well. Initially performed experiments aim to stress the validity of the provided implementation, as well as to show its usability with a commodity bio-signal acquisition device, orders of magnitude less expensive than commonly used ones, representing a step forward in the direction of practical BCIs for end users applications. The proposed implementation, thanks to its flexibility, is used also to perform novel experiments aimed to investigate the exploitation of stereoscopic displays to overcome a known limitation of ordinary displays in the context of SSVEP based BCIs. Eventually, novel experiments are presented investigating the use of the SSVEP modality to provide also implicit interaction. In this context, a first proof of concept Passive BCI based on the SSVEP response is presented and demonstrated to provide information exploitable for prospective applications

Deep learning in mining biological data

Recent technological advancements in data acquisition tools allowed life scientists to acquire multimodal data from different biological application domains. Categorised in three broad types (i.e., images, signals, and sequences), these data are huge in amount and complex in nature. Mining such enormous amount of data for pattern recognition is a big challenge and requires sophisticated data intensive machine learning techniques. Artificial neural network based learning systems are well known for their pattern recognition capabilities and lately their deep architectures - known as deep learning (DL) - have been successfully applied to solve many complex pattern recognition problems. To investigate how DL - especially its different architectures - has contributed and utilised in the mining of biological data pertaining to those three types, a meta analysis has been performed and the resulting resources have been critically analysed. Focusing on the use of DL to analyse patterns in data from diverse biological domains, this work investigates different DL architectures' applications to these data. This is followed by an exploration of available open access data sources pertaining to the three data types along with popular open source DL tools applicable to these data. Also, comparative investigations of these tools from qualitative, quantitative, and benchmarking perspectives are provided. Finally, some open research challenges in using DL to mine biological data are outlined and a number of possible future perspectives are put forward

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010–2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges