Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

In recent years, new research has brought the field of EEG-based Brain-Computer Interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely,“Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user-machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human-computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices

Electroencephalographic recording of the movement-related cortical potential in ecologically-valid movements:A scoping review

The movement-related cortical potential (MRCP) is a brain signal that can be recorded using surface electroencephalography (EEG) and represents the cortical processes involved in movement preparation. The MRCP has been widely researched in simple, single-joint movements, however, these movements often lack ecological validity. Ecological validity refers to the generalizability of the findings to real-world situations, such as neurological rehabilitation. This scoping review aimed to synthesize the research evidence investigating the MRCP in ecologically valid movement tasks. A search of six electronic databases identified 102 studies that investigated the MRCP during multi-joint movements; 59 of these studies investigated ecologically valid movement tasks and were included in the review. The included studies investigated 15 different movement tasks that were applicable to everyday situations, but these were largely carried out in healthy populations. The synthesized findings suggest that the recording and analysis of MRCP signals is possible in ecologically valid movements, however the characteristics of the signal appear to vary across different movement tasks (i.e., those with greater complexity, increased cognitive load, or a secondary motor task) and different populations (i.e., expert performers, people with Parkinson’s Disease, and older adults). The scarcity of research in clinical populations highlights the need for further research in people with neurological and age-related conditions to progress our understanding of the MRCPs characteristics and to determine its potential as a measure of neurological recovery and intervention efficacy. MRCP-based neuromodulatory interventions applied during ecologically valid movements were only represented in one study in this review as these have been largely delivered during simple joint movements. No studies were identified that used ecologically valid movements to control BCI-driven external devices; this may reflect the technical challenges associated with accurately classifying functional movements from MRCPs. Future research investigating MRCP-based interventions should use movement tasks that are functionally relevant to everyday situations. This will facilitate the application of this knowledge into the rehabilitation setting

The potential of error-related potentials. Analysis and decoding for control, neuro-rehabilitation and motor substitution

Las interfaces cerebro-máquina (BMIs, por sus siglas en inglés) permiten la decodificación de patrones de activación neuronal del cerebro de los usuarios para proporcionar a personas con movilidad severamente limitada, ya sea debido a un accidente o a una enfermedad neurodegenerativa, una forma de establecer una conexión directa entre su cerebro y un dispositivo. En este sentido, las BMIs basadas en técnicas no invasivas, como el electroencefalograma (EEG) han ofrecido a estos usuarios nuevas oportunidades para recuperar el control sobre las actividades de su vida diaria que de otro modo no podrían realizar, especialmente en las áreas de comunicación y control de su entorno.En los últimos años, la tecnología está avanzando a grandes pasos y con ella la complejidad de dispositivos ha incrementado significativamente, ampliando el número de posibilidades para controlar sofisticados dispositivos robóticos, prótesis con numerosos grados de libertad o incluso para la aplicación de complejos patrones de estimulación eléctrica en las propias extremidades paralizadas de un usuario, que le permitan ejecutar movimientos precisos. Sin embargo, la cantidad de información que se puede transmitir entre el cerebro y estos dispositivos sigue siendo muy limitada, tanto por el número como por la velocidad a la que se pueden decodificar los comandos neuronales. Por lo tanto, depender únicamente de las señales neuronales no garantiza un control óptimo y preciso.Para poder sacar el máximo partido de estas tecnologías, el campo de las BMIs adoptó el conocido enfoque de “control-compartido". Esta estrategia de control pretende crear un sistema de cooperación entre el usuario y un dispositivo inteligente, liberando al usuario de las tareas más pesadas requeridas para ejecutar la tarea sin llegar a perder la sensación de estar en control. De esta manera, los usuarios solo necesitan centrar su atención en los comandos de alto nivel (por ejemplo, elegir un elemento específico que agarrar, o elegir el destino final donde moverse) mientras el agente inteligente resuelve problemas de bajo nivel (como planificación de trayectorias, esquivar obstáculos, etc.) que permitan realizar la tarea designada de la manera óptima.En particular, esta tesis gira en torno a una señal neuronal cognitiva de alto nivel originada como la falta de coincidencia entre las expectativas del usuario y las acciones reales ejecutadas por los dispositivos inteligentes. Estas señales, denominadas potenciales de error (ErrPs), se consideran una forma natural de intercomunicar nuestro cerebro con máquinas y, por lo tanto, los usuarios solo requieren monitorizar las acciones de un dispositivo y evaluar mentalmente si este último se comporta correctamente o no. Esto puede verse como una forma de supervisar el comportamiento del dispositivo, en el que la decodificación de estas evaluaciones mentales se utiliza para proporcionar a estos dispositivos retroalimentación directamente relacionada con la ejecución de una tarea determinada para que puedan aprender y adaptarse a las preferencias del usuario.Dado que la respuesta neuronal de ErrP está asociada a un evento exógeno (dispositivo que comete una acción errónea), la mayoría de los trabajos desarrollados han intentado distinguir si una acción es correcta o errónea mediante la explotación de eventos discretos en escenarios bien controlados. Esta tesis presenta el primer intento de cambiar hacia configuraciones asíncronas que se centran en tareas relacionadas con el aumento de las capacidades motoras, con el objetivo de desarrollar interfaces para usuarios con movilidad limitada. En este tipo de configuraciones, dos desafíos importantes son que los eventos correctos o erróneos no están claramente definidos y los usuarios tienen que evaluar continuamente la tarea ejecutada, mientras que la clasificación de las señales EEG debe realizarse de forma asíncrona. Como resultado, los decodificadores tienen que lidiar constantemente con la actividad EEG de fondo, que típicamente conduce a una gran cantidad de errores de detección de firmas de error. Para superar estos desafíos, esta tesis aborda dos líneas principales de trabajo.Primero, explora la neurofisiología de las señales neuronales evocadas asociadas con la percepción de errores durante el uso interactivo de un BMI en escenarios continuos y más realistas.Se realizaron dos estudios para encontrar características alternativas basadas en el dominio de la frecuencia como una forma de lidiar con la alta variabilidad de las señales del EEG. Resultados, revelaron que existe un patrón estable representado como oscilaciones "theta" que mejoran la generalización durante la clasificación. Además, se utilizaron técnicas de aprendizaje automático de última generación para aplicar el aprendizaje de transferencia para discriminar asincrónicamente los errores cuando se introdujeron de forma gradual y no se conoce presumiblemente el inicio que desencadena los ErrPs. Además, los análisis de neurofisiología arrojan algo de luz sobre los mecanismos cognitivos subyacentes que provocan ErrP durante las tareas continuas, lo que sugiere la existencia de modelos neuronales en nuestro cerebro que acumulan evidencia y solo toman una decisión al alcanzar un cierto umbral. En segundo lugar, esta tesis evalúa la implementación de estos potenciales relacionados con errores en tres aplicaciones orientadas al usuario. Estos estudios no solo exploran cómo maximizar el rendimiento de decodificación de las firmas ErrP, sino que también investigan los mecanismos neuronales subyacentes y cómo los diferentes factores afectan las señales provocadas.La primera aplicación de esta tesis presenta una nueva forma de guiar a un robot móvil que se mueve en un entorno continuo utilizando solo potenciales de error como retroalimentación que podrían usarse para el control directo de dispositivos de asistencia. Con este propósito, proponemos un algoritmo basado en el emparejamiento de políticas para el aprendizaje de refuerzo inverso para inferir el objetivo del usuario a partir de señales cerebrales.La segunda aplicación presentada en esta tesis contempla los primeros pasos hacia un BCI híbrido para ejecutar distintos tipos de agarre de objetos, con el objetivo de ayudar a las personas que han perdido la funcionalidad motora de su extremidad superior. Este BMI combina la decodificación del tipo de agarre a partir de señales de EEG obtenidas del espectro de baja frecuencia con los potenciales de error provocados como resultado de la monitorización de movimientos de agarre erróneos. Los resultados muestran que, en efecto los ErrP aparecen en combinaciones de señales motoras originadas a partir de movimientos de agarre consistentes en una única repetición. Además, la evaluación de los diferentes factores involucrados en el diseño de la interfaz híbrida (como la velocidad de los estímulos, el tipo de agarre o la tarea mental) muestra cómo dichos factores afectan la morfología del subsiguiente potencial de error evocado.La tercera aplicación investiga los correlatos neuronales y los procesos cognitivos subyacentes asociados con desajustes somatosensoriales producidos por perturbaciones inesperadas durante la estimulación eléctrica neuromuscular en el brazo de un usuario. Este estudio simula los posibles errores que ocurren durante la terapia de neuro-rehabilitación, en la que la activación simultánea de la estimulación aferente mientras los sujetos se concentran en la realización de una tarea motora es crucial para una recuperación óptima. Los resultados muestran que los errores pueden aumentar la atención del sujeto en la tarea y desencadenar mecanismos de aprendizaje que al mismo tiempo podrían promover la neuroplasticidad motora.En resumen, a lo largo de esta tesis, se han diseñado varios paradigmas experimentales para mejorar la comprensión de cómo se generan los potenciales relacionados con errores durante el uso interactivo de BMI en aplicaciones orientadas al usuario. Se han propuesto diferentes métodos para pasar de la configuración bloqueada en el tiempo a la asíncrona, tanto en términos de decodificación como de percepción de los eventos erróneos; y ha explorado tres aplicaciones relacionadas con el aumento de las capacidades motoras, en las cuales los ErrPs se pueden usar para el control de dispositivos, la sustitución de motores y la neuro-rehabilitación.Brain-machine interfaces (BMIs) allow the decoding of cortical activation patterns from the users brain to provide people with severely limited mobility, due to an accident or disease, a way to establish a direct connection between their brain and a device. In this sense, BMIs based in noninvasive recordings, such as the electroencephalogram (EEG) have o↵ered these users new opportunities to regain control over activities of their daily life that they could not perform otherwise, especially in the areas of communication and control of their environment. Over the past years and with the latest technological advancements, devices have significantly grown on complexity expanding the number of possibilities to control complex robotic devices, prosthesis with numerous degrees of freedom or even to apply compound patterns of electrical stimulation on the subjects own paralyzed extremities to execute precise movements. However, the band-with of communication between brain and devices is still very limited, both in terms of the number and the speed at which neural commands can be decoded, and thus solely relying on neural signals do not guarantee accurate control them. In order to benefit of these technologies, the field of BMIs adopted the well-known approach of shared-control. This strategy intends to create a cooperation system between the user and an intelligent device, liberating the user from the burdensome parts of the task without losing the feeling of being in control. Here, users only need to focus their attention on high-level commands (e.g. choose the final destination to reach, or a specific item to grab) while the intelligent agent resolve low-level problems (e.g. trajectory planning, obstacle avoidance, etc) to perform the designated task in the optimal way. In particular, this thesis revolves around a high-level cognitive neural signal originated as the mismatch between the expectations of the user and the actual actions executed by the intelligent devices. These signals, denoted as error-related potentials (ErrPs), are thought as a natural way to intercommunicate our brain with machines and thus users only require to monitor the actions of a device and mentally assess whether the latter is behaving correctly or not. This can be seen as a way to supervise the device’s behavior, in which the decoding of these mental assessments is used to provide these devices with feedback directly related with the performance of a given task so they can learn and adapt to the user’s preferences. Since the ErrP’s neural response is associated to an exogenous event (device committing an erroneous action), most of the developed works have attempted to distinguish whether an action is correct or erroneous by exploiting discrete events under well-controlled scenarios. This thesis presents the first attempt to shift towards asynchronous settings that focus on tasks related with the augmentation of motor capabilities, with the objective of developing interfaces for users with limited mobility. In this type of setups, two important challenges are that correct or erroneous events are not clearly defined and users have to continuously evaluate the executed task, while classification of EEG signals has to be performed asynchronously. As a result, the decoders have to constantly deal with background EEG activity, which typically leads to a large number of missdetection of error signatures. To overcome these challenges, this thesis addresses two main lines of work. First, it explores the neurophysiology of the evoked neural signatures associated with the perception of errors during the interactive use of a BMI in continuous and more realistic scenarios. Two studies were performed to find alternative features based on the frequency domain as a way of dealing with the high variability of EEG signals. Results, revealed that there exists a stable pattern represented as theta oscillations that enhance generalization during classification. Also, state-of-the-art machine learning techniques were used to apply transfer learning to asynchronously discriminate errors when they were introduced in a gradual fashion and the onset that triggers the ErrPs is not presumably known. Furthermore, neurophsysiology analyses shed some light about the underlying cognitive mechanisms that elicit ErrP during continuous tasks, suggesting the existence of neural models in our brain that accumulate evidence and only take a decision upon reaching a certain threshold. Secondly, this thesis evaluates the implementation of these error-related potentials in three user-oriented applications. These studies not only explore how to maximize the decoding performance of ErrP signatures but also investigate the underlying neural mechanisms and how di↵erent factors a↵ect the elicited signals. The first application of this thesis presents a new way to guide a mobile robot moving in a continuous environment using only error potentials as feedback which could be used for the direct control of assistive devices. With this purpose, we propose an algorithm based on policy matching for inverse reinforcement learning to infer the user goal from brain signals. The second application presented in this thesis contemplates the first steps towards a hybrid BMI for grasping oriented to assist people who have lost motor functionality of their upper-limb. This BMI combines the decoding of the type of grasp from low-frequency EEG signals with error-related potentials elicited as the result of monitoring an erroneous grasping. The results show that ErrPs are elicited in combination of motor signatures from the low-frequency spectrum originated from single repetition grasping tasks and evaluates how di↵erent design factors (such as the speed of the stimuli, type of grasp or mental task) impact the morphology of the subsequent evoked ErrP. The third application investigates the neural correlates and the underlying cognitive processes associated with somatosensory mismatches produced by unexpected disturbances during neuromsucular electrical stimulation on a user’s arm. This study simulates possible errors that occur during neurorehabilitation therapy, in which the simultaneous activation of a↵erent stimulation while the subjects are concentrated in performing a motor task is crucial for optimal recovery. The results showed that errors may increase subject’s attention on the task and trigger learning mechanisms that at the same time could promote motor neuroplasticity. In summary, throughout this thesis, several experimental paradigms have been designed to improve the understanding of how error-related potentials are generated during the interactive use of BMIs in user-oriented applications. Di↵erent methods have been proposed to shift from time-locked to asynchronous settings, both in terms of decoding and perception of the erroneous events; and it has explored three applications related with the augmentation of motor capabilities, in which ErrPs can be used for control of devices, motor substitution and neurorehabilitation.<br /

Comparison of tri-polar concentric ring electrodes to disc electrodes for decoding real and imaginary finger movements, A

2019 Spring.Includes bibliographical references.The electroencephalogram (EEG) is broadly used for diagnosis of brain diseases and research of brain activities. Although the EEG provides a good temporal resolution, it suffers from poor spatial resolution due to the blurring effects of volume conduction and signal-to-noise ratio. Many efforts have been devoted to the development of novel methods that can increase the EEG spatial resolution. The surface Laplacian, which is the second derivative of the surface potential, has been applied to EEG to improve the spatial resolution. Tri-polar concentric ring electrodes (TCREs) have been shown to estimate the surface Laplacian automatically with better spatial resolution than conventional disc electrodes. The aim of this research is to study how well the TCREs can be used to acquire EEG signals to decode real and imaginary finger movements. These EEG signals will be then translated into finger movements commands. We also compare the feasibility of discriminating finger movements from one hand using EEG recorded from TCREs and conventional disc electrodes. Furthermore, we evaluated two movement-related features, temporal EEG data and spectral features, in discriminating individual finger from one hand using non-invasive EEG. To do so, movement-related potentials (MRPs) are measured and analyzed from four TCREs and conventional disc electrodes while 13 subjects performed either motor execution or motor imagery of individual finger movements. The tri-polar-EEG (tEEG) and conventional EEG (cEEG) were recorded from electrodes placed according to the 10-20 International Electrode Positioning System over the motor cortex. Our results show that the TCREs achieved higher spatial resolution than conventional disc electrodes. Moreover, the results show that signals from TCREs generated higher decoding accuracy compared to signals from conventional disc electrodes. The average decoding accuracy of five-class classification for all subjects was of 70.04 ± 7.68% when we used temporal EEG data as feature and classified it using Artificial Neural Networks (ANNs) classifier. In addition, the results show that the TCRE EEG (tEEG) provides approximately a four times enhancement in the signal-to-noise ratio (SNR) compared to disc electrode signals. We also evaluated the interdependency level between neighboring electrodes from tri-polar, disc, and disc with Hjorth's Laplacian method in time and frequency domains by calculating the mutual information (MI) and coherence. The MRP signals recorded with the TCRE system have significantly less mutual information (MI) between electrodes than the conventional disc electrode system and disc electrodes with Hjorth's Laplacian method. Also, the results show that the mean coherence between neighboring tri-polar electrodes was found to be significantly smaller than disc electrode and disc electrode with Hjorth's method, especially at higher frequencies. This lower coherence in the high frequency band between neighboring tri polar electrodes suggests that the TCREs may record a more localized neuronal activity. The successful decoding of finger movements can provide extra degrees of freedom to drive brain computer interface (BCI) applications, especially for neurorehabilitation

Co-adaptive control strategies in assistive Brain-Machine Interfaces

A large number of people with severe motor disabilities cannot access any of the available control inputs of current assistive products, which typically rely on residual motor functions. These patients are therefore unable to fully benefit from existent assistive technologies, including communication interfaces and assistive robotics. In this context, electroencephalography-based Brain-Machine Interfaces (BMIs) offer a potential non-invasive solution to exploit a non-muscular channel for communication and control of assistive robotic devices, such as a wheelchair, a telepresence robot, or a neuroprosthesis. Still, non-invasive BMIs currently suffer from limitations, such as lack of precision, robustness and comfort, which prevent their practical implementation in assistive technologies. The goal of this PhD research is to produce scientific and technical developments to advance the state of the art of assistive interfaces and service robotics based on BMI paradigms. Two main research paths to the design of effective control strategies were considered in this project. The first one is the design of hybrid systems, based on the combination of the BMI together with gaze control, which is a long-lasting motor function in many paralyzed patients. Such approach allows to increase the degrees of freedom available for the control. The second approach consists in the inclusion of adaptive techniques into the BMI design. This allows to transform robotic tools and devices into active assistants able to co-evolve with the user, and learn new rules of behavior to solve tasks, rather than passively executing external commands. Following these strategies, the contributions of this work can be categorized based on the typology of mental signal exploited for the control. These include: 1) the use of active signals for the development and implementation of hybrid eyetracking and BMI control policies, for both communication and control of robotic systems; 2) the exploitation of passive mental processes to increase the adaptability of an autonomous controller to the user\u2019s intention and psychophysiological state, in a reinforcement learning framework; 3) the integration of brain active and passive control signals, to achieve adaptation within the BMI architecture at the level of feature extraction and classification

Early Predictability of Grasping Movements by Neurofunctional Representations: A Feasibility Study

Human grasping is a relatively fast process and control signals for upper limb prosthetics cannot be generated and processed in a sufficiently timely manner. The aim of this study was to examine whether discriminating between different grasping movements at a cortical level can provide information prior to the actual grasping process, allowing for more intuitive prosthetic control. EEG datasets were captured from 13 healthy subjects who repeatedly performed 16 activities of daily living. Common classifiers were trained on features extracted from the waking-state frequency and total-frequency time domains. Different training scenarios were used to investigate whether classifiers can already be pre-trained by base networks for fine-tuning with data of a target person. A support vector machine algorithm with spatial covariance matrices as EEG signal descriptors based on Riemannian geometry showed the highest balanced accuracy (0.91 ± 0.05 SD) in discriminating five grasping categories according to the Cutkosky taxonomy in an interval from 1.0 s before to 0.5 s after the initial movement. Fine-tuning did not improve any classifier. No significant accuracy differences between the two frequency domains were apparent (p > 0.07). Neurofunctional representations enabled highly accurate discrimination of five different grasping movements. Our results indicate that, for upper limb prosthetics, it is possible to use them in a sufficiently timely manner and to predict the respective grasping task as a discrete category to kinematically prepare the prosthetic hand

Enhancement of Robot-Assisted Rehabilitation Outcomes of Post-Stroke Patients Using Movement-Related Cortical Potential

Post-stroke rehabilitation is essential for stroke survivors to help them regain independence and to improve their quality of life. Among various rehabilitation strategies, robot-assisted rehabilitation is an efficient method that is utilized more and more in clinical practice for motor recovery of post-stroke patients. However, excessive assistance from robotic devices during rehabilitation sessions can make patients perform motor training passively with minimal outcome. Towards the development of an efficient rehabilitation strategy, it is necessary to ensure the active participation of subjects during training sessions. This thesis uses the Electroencephalography (EEG) signal to extract the Movement-Related Cortical Potential (MRCP) pattern to be used as an indicator of the active engagement of stroke patients during rehabilitation training sessions. The MRCP pattern is also utilized in designing an adaptive rehabilitation training strategy that maximizes patients’ engagement. This project focuses on the hand motor recovery of post-stroke patients using the AMADEO rehabilitation device (Tyromotion GmbH, Austria). AMADEO is specifically developed for patients with fingers and hand motor deficits. The variations in brain activity are analyzed by extracting the MRCP pattern from the acquired EEG data during training sessions. Whereas, physical improvement in hand motor abilities is determined by two methods. One is clinical tests namely Fugl-Meyer Assessment (FMA) and Motor Assessment Scale (MAS) which include FMA-wrist, FMA-hand, MAS-hand movements, and MAS-advanced hand movements’ tests. The other method is the measurement of hand-kinematic parameters using the AMADEO assessment tool which contains hand strength measurements during flexion (force-flexion), and extension (force-extension), and Hand Range of Movement (HROM)

A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to do something and transforms them into a control command for controlling external devices. The feelings described by the participants when they control a self-avatar in an immersive virtual environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar (ownership illusion). It has recently been shown that inducing the ownership illusion and then manipulating the movements of one’s self-avatar can lead to compensatory motor control strategies. In order to maximize this effect, there is a need for a method that measures and monitors embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong ownership illusion. This is particularly true given that reaching a high level of both BCI performance and embodiment are inter-connected. To reach one of them, the second must be reached as well. Some limitations of many existing systems hinder their adoption for neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most systems allow the user to take single steps or to walk but do not allow both, which prevents users from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or self-paced), which prevents users from progressing from machine-dependent to machine-independent walking. Overcoming the aforementioned limitations can be done by combining different control modes and options in one single system. However, this would have a negative impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool. In this case, there will be a need to enhance BCI performance. For such purpose, many techniques have been used in the literature, such as providing modified feedback (whereby the presented feedback is not consistent with the user’s MI), sequential training (recalibrating the classifier as more data becomes available). This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility of measuring the level of embodiment of an immersive self-avatar, during the performing, observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting visual feedback that conflicts with the desired movement of embodied participants. The objective of study 2 was to develop and validate a BCI to control single steps and forward walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in cue-paced and self-paced modes. Different performance enhancement strategies were implemented to increase BCI performance. The data of these two studies were then used in study 3 to construct a generic classifier that could eliminate offline calibration for future users and shorten training time. Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD) from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with the contralateral limb or stopped walking before the participant stopped (modified feedback). In study 2, participants completed a 4-day sequential training to control the gait of an avatar in both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking (continuous control mode). The avatar moved as a response to two calibrated regularized linear discriminant analysis (RLDA) classifiers that used the μ power spectral density (PSD) over the foot area of the motor cortex as features. The classifiers were retrained after every session. During the training, and for some of the trials, positive modified feedback was presented to half of the participants, where the avatar moved correctly regardless of the participant’s real performance. In both studies, the participants’ subjective experience was analyzed using a questionnaire. Results of study 1 show that subjective levels of embodiment correlate strongly with the power differences of the event-related synchronization (ERS) within the μ frequency band, and over the motor and pre-motor cortices between the modified and regular feedback trials. Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced BCI in both modes. For the cue-paced BCI, the average offline performance (classification rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback group. For self-paced BCI, the average performance was 83% at switch control and 92% at continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models performed as well as models obtained from participant-specific offline data. The results show that there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont été utilisées pour contrôler la marche d'un égo-avatar virtuel dans le but d'être utilisées dans la réadaptation de la marche. Une ICO décode les signaux du cerveau représentant un désir de faire produire un mouvement et les transforme en une commande de contrôle pour contrôler des appareils externes. Les sentiments décrits par les participants lorsqu'ils contrôlent un égo-avatar dans un environnement virtuel immersif démontrent que les humains peuvent être incarnés dans un corps d'un avatar (illusion de propriété). Il a été récemment démontré que provoquer l’illusion de propriété puis manipuler les mouvements de l’égo-avatar peut conduire à des stratégies de contrôle moteur compensatoire. Afin de maximiser cet effet, il existe un besoin d'une méthode qui mesure et surveille les niveaux d’incarnation des participants immergés dans la réalité virtuelle (RV) pour induire et maintenir une forte illusion de propriété. D'autre part, atteindre un niveau élevé de performances (taux de classification) ICO et d’incarnation est interconnecté. Pour atteindre l'un d'eux, le second doit également être atteint. Certaines limitations de plusieurs de ces systèmes entravent leur adoption pour la neuroréhabilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systèmes permettent à l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas à un utilisateur de passer des pas à la marche; 3- la plupart fonctionnent en un seul mode d’ICO, rythmé (cue-paced) ou auto-rythmé (self-paced). Surmonter les limitations susmentionnées peut être fait en combinant différents modes et options de commande dans un seul système. Cependant, cela aurait un impact négatif sur les performances de l’ICO, diminuant ainsi son utilité en tant qu'outil potentiel de réhabilitation. Dans ce cas, il sera nécessaire d'améliorer les performances des ICO. À cette fin, de nombreuses techniques ont été utilisées dans la littérature, telles que la rétroaction modifiée, le recalibrage du classificateur et l'utilisation d'un classificateur générique. Le projet de cette thèse a été réalisé en 3 études, avec objectif d'étudier dans l'étude 1, la possibilité de mesurer le niveau d'incarnation d'un égo-avatar immersif, lors de l'exécution, de l'observation et de l'imagination de la marche, à l'aide des techniques encéphalogramme (EEG), en présentant une rétroaction visuelle qui entre en conflit avec la commande du contrôle moteur des sujets incarnés. L'objectif de l'étude 2 était de développer un BCI pour contrôler les pas et la marche vers l’avant d'un égo-avatar dans la réalité virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmés et auto-rythmés. Différentes stratégies d'amélioration des performances ont été mises en œuvre pour augmenter la performance (taux de classification) de l’ICO. Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation. Vingt participants sains différents ont participé aux études 1 et 2. Dans l'étude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affiché dans un casque de RV du point de vue de la première personne (1PP). Ils ont été invités à performer, à regarder ou à imaginer un seul pas en avant ou la marche vers l’avant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatéral ou a arrêté de marcher avant que le participant ne s'arrête (rétroaction modifiée). Dans l'étude 2, les participants ont participé à un entrainement séquentiel de 4 jours pour contrôler la marche d'un avatar dans les deux modes de l’ICO. En mode rythmé, ils ont imaginé un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers l’avant . En mode auto-rythmé, il leur a été demandé d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrôle intermittent) ou en maintenir l'IM de marche vers l’avant (mode de contrôle continu). L'avatar s'est déplacé en réponse à deux classificateurs ‘Regularized Linear Discriminant Analysis’ (RLDA) calibrés qui utilisaient comme caractéristiques la densité spectrale de puissance (Power Spectral Density; PSD) des bandes de fréquences µ (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont été recalibrés après chaque session. Au cours de l’entrainement et pour certains des essais, une rétroaction modifiée positive a été présentée à la moitié des participants, où l'avatar s'est déplacé correctement quelle que soit la performance réelle du participant. Dans les deux études, l'expérience subjective des participants a été analysée à l'aide d'un questionnaire. Les résultats de l'étude 1 montrent que les niveaux subjectifs d’incarnation sont fortement corrélés à la différence de la puissance de la synchronisation liée à l’événement (Event-Related Synchronization; ERS) sur la bande de fréquence μ et sur le cortex moteur et prémoteur entre les essais de rétroaction modifiés et réguliers. L'étude 2 a montré que tous les participants étaient capables d’utiliser le BCI rythmé et auto-rythmé dans les deux modes. Pour le BCI rythmé, la performance hors ligne moyenne au jour 1 était de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amélioré la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne était de 85,9±8,4% pour le groupe de rétroaction modifié (77-97%) contre 75% pour le groupe de rétroaction non modifié. Pour le BCI auto-rythmé, la performance moyenne était de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de l’ICO ont été améliorées par la rétroaction modifiée (p = 0,001). Enfin, les résultats de l'étude 3 montrent que pour la classification des initialisations des pas et de la marche, il a été possible de construire des modèles génériques à partir de données hors ligne spécifiques aux participants. Les résultats montrent la possibilité de concevoir une ICO ne nécessitant aucun entraînement spécifique au participant

BCI applications based on artificial intelligence oriented to deep learning techniques

A Brain-Computer Interface, BCI, can decode the brain signals corresponding to the intentions of individuals who have lost neuromuscular connection, to reestablish communication to control external devices. To this aim, BCI acquires brain signals as Electroencephalography (EEG) or Electrocorticography (ECoG), uses signal processing techniques and extracts features to train classifiers for providing proper control instructions. BCI development has increased in the last decades, improving its performance through the use of different signal processing techniques for feature extraction and artificial intelligence approaches for classification, such as deep learning-oriented classifiers. All of these can assure more accurate assistive systems but also can enable an analysis of the learning process of signal characteristics for the classification task. Initially, this work proposes the use of a priori knowledge and a correlation measure to select the most discriminative ECoG signal electrodes. Then, signals are processed using spatial filtering and three different types of temporal filtering, followed by a classifier made of stacked autoencoders and a softmax layer to discriminate between ECoG signals from two types of visual stimuli. Results show that the average accuracy obtained is 97% (+/- 0.02%), which is similar to state-of-the-art techniques, nevertheless, this method uses minimal prior physiological and an automated statistical technique to select some electrodes to train the classifier. Also, this work presents classifier analysis, figuring out which are the most relevant signal features useful for visual stimuli classification. The features and physiological information such as the brain areas involved are compared. Finally, this research uses Convolutional Neural Networks (CNN) or Convnets to classify 5 categories of motor tasks EEG signals. Movement-related cortical potentials (MRCPs) are used as a priori information to improve the processing of time-frequency representation of EEG signals. Results show an increase of more than 25% in average accuracy compared to a state-of-the-art method that uses the same database. In addition, an analysis of CNN or ConvNets filters and feature maps is done to and the most relevant signal characteristics that can help classify the five types of motor tasks.DoctoradoDoctor en Ingeniería Eléctrica y Electrónic