Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

In recent years, new research has brought the field of EEG-based Brain-Computer Interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely,“Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user-machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human-computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices

Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person’s life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics

Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person’s life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics

Defining brain–machine interface applications by matching interface performance with device requirements

Interaction with machines is mediated by human-machine interfaces (HMIs). Brain-machine interfaces (BMIs) are a particular class of HMIs and have so far been studied as a communication means for people who have little or no voluntary control of muscle activity. In this context, low-performing interfaces can be considered as prosthetic applications. On the other hand, for able-bodied users, a BMI would only be practical if conceived as an augmenting interface. In this paper, a method is introduced for pointing out effective combinations of interfaces and devices for creating real-world applications. First, devices for domotics, rehabilitation and assistive robotics, and their requirements, in terms of throughput and latency, are described. Second, HMIs are classified and their performance described, still in terms of throughput and latency. Then device requirements are matched with performance of available interfaces. Simple rehabilitation and domotics devices can be easily controlled by means of BMI technology. Prosthetic hands and wheelchairs are suitable applications but do not attain optimal interactivity. Regarding humanoid robotics, the head and the trunk can be controlled by means of BMIs, while other parts require too much throughput. Robotic arms, which have been controlled by means of cortical invasive interfaces in animal studies, could be the next frontier for non-invasive BMIs. Combining smart controllers with BMIs could improve interactivity and boost BMI applications. © 2007 Elsevier B.V. All rights reserved

Physiological Responses During Hybrid BNCI Control of an Upper-Limb Exoskeleton

When combined with assistive robotic devices, such as wearable robotics, brain/neural-computer interfaces (BNCI) have the potential to restore the capabilities of handicapped people to carry out activities of daily living. To improve applicability of such systems, workload and stress should be reduced to a minimal level. Here, we investigated the user’s physiological reactions during the exhaustive use of the interfaces of a hybrid control interface. Eleven BNCI-naive healthy volunteers participated in the experiments. All participants sat in a comfortable chair in front of a desk and wore a whole-arm exoskeleton as well as wearable devices for monitoring physiological, electroencephalographic (EEG) and electrooculographic (EoG) signals. The experimental protocol consisted of three phases: (i) Set-up, calibration and BNCI training; (ii) Familiarization phase ; and (iii) Experimental phase during which each subject had to perform EEG and EoG tasks. After completing each task, the NASA-TLX questionnaire and self-assessment manikin (SAM) were completed by the user. We found significant differences (p-value < 0.05) in heart rate variability (HRV) and skin conductance level (SCL) between participants during the use of the two different biosignal modalities (EEG, EoG) of the BNCI. This indicates that EEG control is associated with a higher level of stress (associated with a decrease in HRV) and mental work load (associated with a higher level of SCL) when compared to EoG control. In addition, HRV and SCL modulations correlated with the subject’s workload perception and emotional responses assessed through NASA-TLX questionnaires and SAM

The role of electrical stimulation for rehabilitation and regeneration after spinal cord injury

Electrical stimulation is used to elicit muscle contraction and can be utilized for neurorehabilitation following spinal cord injury when paired with voluntary motor training. This technology is now an important therapeutic intervention that results in improvement in motor function in patients with spinal cord injuries. The purpose of this review is to summarize the various forms of electrical stimulation technology that exist and their applications. Furthermore, this paper addresses the potential future of the technology

A MECHANISTIC APPROACH TO POSTURAL DEVELOPMENT IN CHILDREN

Upright standing is intrinsically unstable and requires active control. The central nervous system's feedback process is the active control that integrates multi-sensory information to generate appropriate motor commands to control the plant (the body with its musculotendon actuators). Maintaining standing balance is not trivial for a developing child because the feedback and the plant are both developing and the sensory inputs used for feedback are continually changing. Knowledge gaps exist in characterizing the critical ability of adaptive multi-sensory reweighting for standing balance control in children. Furthermore, the separate contributions of the plant and feedback and their relationship are poorly understood in children, especially when considering that the body is multi-jointed and feedback is multi-sensory. The purposes of this dissertation are to use a mechanistic approach to study multi-sensory abilities of typically developing (TD) children and children with Developmental Coordination Disorder (DCD). The specific aims are: 1) to characterize postural control under different multi-sensory conditions in TD children and children with DCD; 2) to characterize the development of adaptive multi-sensory reweighting in TD children and children with DCD; and, 3) to identify the plant and feedback for postural control in TD children and how they change in response to visual reweighting. In the first experiment (Aim 1), TD children, adults, and 7-year-old children with DCD are tested under four sensory conditions (no touch/no vision, with touch/no vision, no touch/with vision, and with touch/with vision). We found that touch robustly attenuated standing sway in all age groups. Children with DCD used touch less effectively than their TD peers and they also benefited from using vision to reduce sway. In the second experiment (Aim 2), TD children (4- to 10-year-old) and children with DCD (6- to 11-year-old) were presented with simultaneous small-amplitude touch bar and visual scene movement at 0.28 and 0.2 Hz, respectively, within five conditions that independently varied the amplitude of the stimuli. We found that TD children can reweight to both touch and vision from 4 years on and the amount of reweighting increased with age. However, multi-sensory fusion (i.e., inter-modal reweighting) was only observed in the older children. Children with DCD reweight to both touch and vision at a later age (10.8 years) than their TD peers. Even older children with DCD do not show advanced multisensory fusion. Two signature deficits of multisensory reweighting are a weak vision reweighting and a general phase lag to both sensory modalities. The final aim involves closed-loop system identification of the plant and feedback using electromyography (EMG) and kinematic responses to a high- or low-amplitude visual perturbation and two mechanical perturbations in children ages six and ten years and adults. We found that the plant is different between children and adults. Children demonstrate a smaller phase difference between trunk and leg than adults at higher frequencies. Feedback in children is qualitatively similar to adults. Quantitatively, children show less phase advance at the peak of the feedback curve which may be due to a longer time delay. Under the high and low visual amplitude conditions, children show less gain change (interpreted as reweighting) than adults in the kinematic and EMG responses. The observed kinematic and EMG reweighting are mainly due to the different use of visual information by the central nervous system as measured by the open-loop mapping from visual scene angle to EMG activity. The plant and the feedback do not contribute to reweighting

Development of EEG-based technologies for the characterization and treatment of neurological diseases affecting the motor function

This thesis presents a set of studies applying signal processing and data mining techniques in real-time working systems to register, characterize and condition the movement-related cortical activity of healthy subjects and of patients with neurological disorders affecting the motor function. Patients with two of the most widespread neurological affections impairing the motor function are considered here: patients with essential tremor and patients who have suffered a cerebro-vascular accident. The different chapters in the presented thesis show results regarding the normal cortical activity associated with the planning and execution of motor actions with the upper-limb, and the pathological activity related to the patients' motor dysfunction (measurable with muscle electrodes or movement sensors). The initial chapters of the book present i) a revision of the basic concepts regarding the role of the cerebral cortex in the motor control and the way in which the electroencephalographic activity allows its analysis and conditioning, ii) a study on the cortico-muscular interaction at the tremor frequency in patients with essential tremor under the effects of a drug reducing their tremor, and finally iii) a study based on evolutionary algorithms that aims to identify cortical patterns related to the planning of a number of motor tasks performed with a single arm. In the second half of the thesis book, two brain-computer interface systems to be used in rehabilitation scenarios with essential tremor patients and with patients with a stroke are proposed. In the first system, the electroencephalographic activity is used to anticipate voluntary movement actions, and this information is integrated in a multimodal platform estimating and suppressing the pathological tremors. In the second case, a conditioning paradigm for stroke patients based on the identification of the motor intention with temporal precision is presented and tested with a cohort of four patients along a month during which the patients undergo eight intervention sessions. The presented thesis has yielded advances from both the technological and the scientific points of view in all studies proposed. The main contributions from the technological point of view are: ¿ The design of an integrated upper-limb platform working in real-time. The platform was designed to acquire information from different types of noninvasive sensors (EEG, EMG and gyroscopic sensors) characterizing the planning and execution of voluntary movements. The platform was also capable of processing online the acquired data and generating an electrical feedback. ¿ The development of signal processing and classifying techniques adapted to the kind of signal recorded in the two kinds of patients considered in this thesis (patients with essential tremor and patients with a stroke) and to the requirements of online processing and real-time single-trial function desired for BCI applications. Especially in this regard, an original methodology to detect onsets of voluntary movements using slow cortical potentials and cortical rhythms has been presented. ¿ The design and validation in real-time of asynchronous BCI systems using motor planning EEG segments to anticipate or detect when patients begin a voluntary movement with the upper-limb. ¿ The proof of concept of the advantages of an EEG system integrated in a multimodal human-robot interface architecture that constitutes the first multimodal interface using the combined acquisition of EEG, EMG and gyroscopic data, which allows the concurrent characterization of different parts of the body associated with the execution of a movement. The main scientific contributions of this thesis are: ¿ The study of the EEG-based anticipation of voluntary movements presented in Chapter 5 of the thesis was the first demonstration (to the author's knowledge) of the capacity of the EEG signal to provide reliable movement predictions based on single-trial classification of online data of healthy subjects and ET patients. This study also provides, for the first time, the results of a BCI system tested in ET patients and it represents an original approach to BCI applications for this group of patients. ¿ It has been presented the first neurophysiological study using EEG and EMG data to analyze the effects of a drug on cortical activity and tremors of patients with ET. In addition, the obtained results have shown for the first time that a significant correlation exists between the dynamics of specific cortical oscillations and pathological tremor manifestation as a consequence of the drug effects. ¿ It has been proposed for the first time an experiment to inspect whether the EEG signal carries enough information to classify up to seven different tasks performed with a single limb. Both the methodology applied and the validation procedure are also innovative in this sort of studies. ¿ It has been demonstrated for the first time the relevance of combining different cortical sources of information (such as BP and ERD) to estimate the initiation of voluntary movements with the upper-limb. In this line, special relevance may be given to the positive results achieved with stroke patients, improving the results presented by similar previous EEG-based studies by other research groups. It has also been proposed for the first time an upper-limb intervention protocol for stroke patients using BP and ERD patterns to provide proprioceptive feedback tightly associated with the patients' expectations of movement. The effects of the proposed intervention have been studied with a small group of patients

A neuroprothesis for tremor management

Tremor is the most common movement disorder, affecting ∼ 15 % of people over 50 years old according to some estimates. It appears due to a number of syndromes, being essential tremor and Parkinson's disease the most prevalent among them. None of these conditions is fully understood. Tremor is currently treated through drugs or neurosurgery, but unfortunately, it is not managed effectively in ∼25 % of the patients. Therefore, it constitutes a major cause of loss of independence and quality of life. Various alternative approaches for tremor management are reported in the literature. Among them, those devices that rely on the application of forces to the tremulous segments show a considerable potential. A number of prototypes that exploit this principle are available, spanning fixed devices and orthoses. However, none of them has fulfilled user's expectation for continuous use during daily living. This thesis presents the development and validation of a neuroprosthesis for tremor management. A neuroprosthesis is a system that restores or compensates for a neurological function that is lost. In this case, the neuroprosthesis aims at compensating the functional disability caused by the tremor. To this end, it applies forces to the tremulous limb through the control of muscle contraction, which is modulated according to the characteristics of the tremor. The concept design envisions the device as a textile that is worn on the affected limb, thus meeting the usability requirements defined by the patients. The development of the neuroprosthesis comprised the following tasks: 1. The development of a concept design of the neuroprosthesis, which incorporates state of the art knowledge on tremor, and user's needs. 2. The design and validation of a cognitive interface that parameterizes the tremor in functional contexts. This interface provides the information that the neuroprosthesis uses for tremor suppression. Two versions are developed: a multimodal interface that integrates the recordings of the whole neuromusculoskeletal system, and an interface incorporating only wearable movement sensors. The latter is intended for the functional validation of the neuroprosthesis, while the former is a proof of concept of an optimal interface for this type of applications. 3. The development of a novel approach for tremor suppression through transcutaneous neurostimulation. The approach relies on the modulation of muscle cocontraction as a means of attenuating the tremor without the need of conventional actuators. The experimental validation here provided demonstrates the feasibility and interest of the approach. In parallel with the validation of the neuroprosthesis, I performed a detailed study on the physiology of motoneurons in tremor, given the lack of a complete description of its behavior. The outcome of this study contributes to the interpretation of the results obtained with the neuroprosthesis, and opens new research lines, both related to alternative interventions and basic neuroscience. In summary, the results here presented demonstrate that tremor may be accurately parameterized while the patient performs functional activities, and that this information may be exploited to drive a neuroprosthesis for tremor management. Furthermore, the novel approach for tremor suppression presented in this dissertation constitutes a potential approach for treating upper limb tremor, either alone, or as a complement to pharmacotherapy. These results encourage the validation of the neuroprosthesis in a large cohort of patients, in order to enable its translation to the market. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------El temblor es el trastorno del movimiento más común, afectando, según algunas estimaciones, al ∼15 % de la población de más de 50 años. Existen diversos "síndromes" que causan temblor, siendo el temblor esencial y la enfermedad de Parkinson los que presentan mayor prevalencia. Además, cabe resaltar que no existe una descripción completa de ninguno de ellos. En la actualidad el temblor se trata mediante una serie de fármacos o neurocirugía. A pesar de ello, el ∼ 25 % de los pacientes sufren problemas funcionales debido a su condición. Por tanto, es evidente que el temblor constituye una de las principales causas de dependencia y pérdida de calidad de vida. Realizando una revisión de las publicaciones científicas sobre el temblor, se observa que se ha propuesto un considerable número de tratamientos alternativos. Entre ellos destacan los dispositivos que se fundamentan en la aplicación de fuerzas sobre los segmentos afectados por el temblor, de los que ya se ha evaluado una serie de prototipos. Estos abarcan desde dispositivos fijados a otras estructuras hasta ortesis. Sin embargo, ninguno de ellos satisface las expectativas de los usuarios para su uso durante el día a día. Esta tesis presenta el diseño y validación de una neruoprótesis para el tratamiento del temblor. Una neuroprótesis es un sistema que reemplaza o compensa una función neurológica perdida. En este caso, la neuroprótesis tiene como objetivo compensar la discapacidad motora causada por el temblor. Para ello aplica fuerzas al miembro afectado a través del control del nivel de contracción muscular, que se modula según las características del temblor. El diseño conceptual contempla al dispositivo como un textil que se viste en el brazo afectado, satisfaciendo los requisitos de usabilidad definidos por los pacientes. El desarrollo de la neuroprótesis abarcó las siguientes tareas: 1. El desarrollo del diseño conceptual de la neuroprótesis, que incorpora el conocimiento actual sobre el temblor, y las necesidades de los usuarios. 2. El diseño y validación de una interfaz cognitiva que parametriza el temblor durante tareas funcionales. La información obtenida con esta interfaz es usada por la neuroprótesis para modular la corriente aplicada mediante técnicas de neuroestimulación. Se desarrollan dos versiones de la interfaz cognitiva: una interfaz multimodal que integra información de todo el sistema neuromusculoesquelético, y una interfaz que implementa únicamente sensores vestibles de movimiento. La segunda interfaz fue la que se usó durante la validación funcional de la neuroprótesis, mientras que la primera es una prueba de concepto de una interfaz óptima para este tipo de aplicaciones. 3. El desarrollo de una nueva aproximación para la supresión del temblor mediante neuroestimulación transcutánea. Dicha aproximación se fundamenta en la modulación del grado de co-contracción de los músculos afectados como forma de atenuar el temblor, sin necesidad de usar actuadores convencionales. La evaluación experimental sirvió para demostrar la viabilidad e interés de la intervención. En paralelo a la validación de la neuroprótesis, llevé a cabo un estudio detallado de la fisiología de las motoneuronas en el caso del temblor, dado que no existe una descripción del funcionamiento de las mismas en el caso de este trastorno. Este estudio sirve para ayudar a la interpretación de los resultados de la neuroprótesis, y para abrir una serie de líneas futuras de investigación, tanto sobre nuevas intervenciones para el temblor, como sobre neurociencia básica. En resumen, los resultados que se presentan en esta tesis demuestran que es posible parametrizar de una forma precisa el temblor durante la realización de tareas funcionales, y que esta información sirve para controlar una neuroprótesis para el tratamiento del temblor. Además, la nueva aproximación para la compensación del temblor que se presenta tiene el potencial de convertirse en un tratamiento alternativo para el temblor de miembro superior, ya sea de forma independiente o como complemento a los fármacos. Estos resultados alientan la validación de la neuroprótesis en una cohorte grande de pacientes, con el objetivo de facilitar su transferencia al mercado