Quadcopter Flight Control Using a Non-invasive Multi-Modal Brain Computer Interface

Brain-Computer Interfaces (BCIs) translate neuronal information into commands to control external software or hardware, which can improve the quality of life for both healthy and disabled individuals. Here, a multi-modal BCI which combines motor imagery (MI) and steady-state visual evoked potential (SSVEP) is proposed to achieve stable control of a quadcopter in three-dimensional physical space. The complete information common spatial pattern (CICSP) method is used to extract two MI features to control the quadcopter to fly left-forward and right-forward, and canonical correlation analysis (CCA) is employed to perform the SSVEP classification for rise and fall. Eye blinking is designed to switch these two modes while hovering. Real-time feedback is provided to subjects by a global camera. Two flight tasks were conducted in physical space in order to certify the reliability of the BCI system. Subjects were asked to control the quadcopter to fly forward along the zig-zag pattern to pass through a gate in the relatively simple task. For the other complex task, the quadcopter was controlled to pass through two gates successively according to an S-shaped route. The performance of the BCI system is quantified using suitable metrics and subjects are able to acquire 86.5% accuracy for the complicated flight task. It is demonstrated that the multi-modal BCI has the ability to increase the accuracy rate, reduce the task burden, and improve the performance of the BCI system in the real world

A brain-computer interface integrated with virtual reality and robotic exoskeletons for enhanced visual and kinaesthetic stimuli

Brain-computer interfaces (BCI) allow the direct control of robotic devices for neurorehabilitation and measure brain activity patterns following the user’s intent. In the past two decades, the use of non-invasive techniques such as electroencephalography and motor imagery in BCI has gained traction. However, many of the mechanisms that drive the proficiency of humans in eliciting discernible signals for BCI remains unestablished. The main objective of this thesis is to explore and assess what improvements can be made for an integrated BCI-robotic system for hand rehabilitation. Chapter 2 presents a systematic review of BCI-hand robot systems developed from 2010 to late 2019 in terms of their technical and clinical reports. Around 30 studies were identified as eligible for review and among these, 19 were still in their prototype or pre-clinical stages of development. A degree of inferiority was observed from these systems in providing the necessary visual and kinaesthetic stimuli during motor imagery BCI training. Chapter 3 discusses the theoretical background to arrive at a hypothesis that an enhanced visual and kinaesthetic stimulus, through a virtual reality (VR) game environment and a robotic hand exoskeleton, will improve motor imagery BCI performance in terms of online classification accuracy, class prediction probabilities, and electroencephalography signals. Chapters 4 and 5 focus on designing, developing, integrating, and testing a BCI-VR-robot prototype to address the research aims. Chapter 6 tests the hypothesis by performing a motor imagery BCI paradigm self-experiment with an enhanced visual and kinaesthetic stimulus against a control. A significant increase (p = 0.0422) in classification accuracies is reported among groups with enhanced visual stimulus through VR versus those without. Six out of eight sessions among the VR groups have a median of class probability values exceeding a pre-set threshold value of 0.6. Finally, the thesis concludes in Chapter 7 with a general discussion on how these findings could suggest the role of new and emerging technologies such as VR and robotics in advancing BCI-robotic systems and how the contributions of this work may help improve the usability and accessibility of such systems, not only in rehabilitation but also in skills learning and education

Physical Diagnosis and Rehabilitation Technologies

The book focuses on the diagnosis, evaluation, and assistance of gait disorders; all the papers have been contributed by research groups related to assistive robotics, instrumentations, and augmentative devices

Design and Optimization of a BCI-Driven Telepresence Robot Through Programming by Demonstration

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8788527Improving the life quality of people with severe motor paralysis has a significant impact on restoring their functional independence to perform activities of daily living (ADL). Telepresence is a subfield of the robotic-assisted route, where human plays the role of an operator, sending high-level instructions to an as sistive robot while receiving sensory feedback. However, for severely motor-impaired people, conventional interaction modalities may not be suitable due to their complete paralysis. Thus, designing alternative ways of interaction such as Brain-Computer Interfaces (BCI) is essential for a telepresence capability. We propose a novel framework that integrates a BCI system and a humanoid robot to develop a brain-controlled telepresence system with multimodal control features. In particular, the low-level control is executed by Programming by Demonstration (PbD) models, and the higher-level cognitive commands are produced by a BCI system to perform vital ADLs. The presented system is based on real-time decoding of attention-modulated neural responses elicited in the brain electroencephalographic signals and generating multiple control commands. As a result, the system allows a user to interact with a humanoid robot while receiving auditory and visual feedback from the robot's sensors. We validated our system across ten subjects in a realistic scenario. The experimental results show the feasibility of the approach in the design of a telepresence robot with high BCI decoding performances

Enhancing brain/neural-machine interfaces for upper limb motor restoration in chronic stroke and cervical spinal cord injury

Operation of assistive exoskeletons based on voluntary control of sensorimotor rhythms (SMR, 8-12 Hz) enables intuitive control of finger or arm movements in severe paralysis after chronic stroke or cervical spinal cord injury (SCI). To improve reliability of such systems outside the laboratory, in particular when brain activity is recorded non-invasively with scalp electroencephalography (EEG), a hybrid EEG/electrooculography (EOG) brain/neural-machine interface (B/NMI) was recently introduced. Besides providing assistance, recent studies indicate that repeated use of such systems can trigger neural recovery. However, important prerequisites have to achieved before broader use in clinical settings or everyday life environments is feasible. Current B/NMI systems predominantly restore hand function, but do not allow simultaneous control of more proximal joints for whole-arm motor coordination as required for most stroke survivors suffering from paralysis in the entire upper limb. Besides paralysis, cognitive impairments including post-stroke fatigue due to the brain lesion reduce the capacity to maintain effortful B/NMI control over a longer period of time. This impedes the applicability in daily life assistance and might even limits the efficacy of neurorehabilitation training. In contrast to stroke survivors, tetraplegics due to cervical SCI lack motor function in both hands. Given that most activities of daily living (ADL) involve bimanual manipulation, e.g., to open the lid of a bottle, bilateral exoskeleton control is required but was not shown yet in tetraplegics. To further enhance B/NMI systems, we first investigated whether B/NMI whole-arm exoskeleton control in hemiplegia after chronic stroke is feasible and safe. In contrast to simple grasping, control of more complex tasks involving the entire upper limb was not feasible with established B/NMIs because high- dimensionality of such multiple joint systems exceeds the bandwidth of these interfaces. Thus, we blended B/NMI control with vision-guidance to receive a semiautonomous whole-arm exoskeleton control. Such setup allowed to divide ADL tasks into a sequence of EEG/EOG-triggered sub-tasks reducing complexity for the user. While, for instance, a drinking task was resolved into EOG-induced reaching, lifting and placing back the cup, grasping and releasing movements were based on intuitive SMR control. Feasibility of such shared vision-guided B/NMI control was assumed when executions were initialized within 3 s (fluent control) and a minimum of 75 % of subtasks were executed within that time (reliable control). We showed feasibility in healthy subjects as well as stroke survivors without report of any side effects documenting safe use. Similarly, feasibility and safety of bilateral B/NMI control after cervical SCI was evaluated. To enable bilateral B/NMI control, established EEG-based grasping and EOG-based releasing or stop commands were complemented with a novel EOG command allowing to switch laterality by performing prolonged horizontal eye movements (>1 s) to the left or to the right. Study results with healthy subjects and tetraplegics document fluent initialization of grasping motions below 3 s as well as safe use as unintended grasping could be stopped before a full motion was conducted. Superiority of novel bilateral control was documented by a higher accuracy of up to 22 % in tetraplegics compared to a bilateral control without prolonged EOG command. Lastly, as reliable B/NMI control is cognitively demanding, e.g., by imagining or attempting the desired movements, we investigated whether heart rate variability (HRV) can be used as biomarker to predict declining control performance, which is often reported in stroke survivors due to their cognitive impairments. Referring to the close brain-heart connection, we showed in healthy subjects that a decline in HRV is specific as well as predictive to a decline in B/NMI control performance within a single training session. The predictive link was revealed by a Granger-causality analysis. In conclusion, we could demonstrate important enhancements in B/NMI control paradigms including complex whole-arm exoskeleton control as well as individual performance monitoring within a training session based on HRV. Both achievements contribute to broaden the use as a standard therapy in stroke neurorehabilitation. Especially the predictive characteristic of HRV paves the way for adaptive B/NMI control paradigms to account for individual differences among impaired stroke survivors. Moreover, we also showed feasibility and safety of a novel implementation for bilateral B/NMI control, which is necessary for reliable operation of two hand-exoskeletons for bimanual ADLs after SCI

On Tackling Fundamental Constraints in Brain-Computer Interface Decoding via Deep Neural Networks

A Brain-Computer Interface (BCI) is a system that provides a communication and control medium between human cortical signals and external devices, with the primary aim to assist or to be used by patients who suffer from a neuromuscular disease. Despite significant recent progress in the area of BCI, there are numerous shortcomings associated with decoding Electroencephalography-based BCI signals in real-world environments. These include, but are not limited to, the cumbersome nature of the equipment, complications in collecting large quantities of real-world data, the rigid experimentation protocol and the challenges of accurate signal decoding, especially in making a system work in real-time. Hence, the core purpose of this work is to investigate improving the applicability and usability of BCI systems, whilst preserving signal decoding accuracy. Recent advances in Deep Neural Networks (DNN) provide the possibility for signal processing to automatically learn the best representation of a signal, contributing to improved performance even with a noisy input signal. Subsequently, this thesis focuses on the use of novel DNN-based approaches for tackling some of the key underlying constraints within the area of BCI. For example, recent technological improvements in acquisition hardware have made it possible to eliminate the pre-existing rigid experimentation procedure, albeit resulting in noisier signal capture. However, through the use of a DNN-based model, it is possible to preserve the accuracy of the predictions from the decoded signals. Moreover, this research demonstrates that by leveraging DNN-based image and signal understanding, it is feasible to facilitate real-time BCI applications in a natural environment. Additionally, the capability of DNN to generate realistic synthetic data is shown to be a potential solution in reducing the requirement for costly data collection. Work is also performed in addressing the well-known issues regarding subject bias in BCI models by generating data with reduced subject-specific features. The overall contribution of this thesis is to address the key fundamental limitations of BCI systems. This includes the unyielding traditional experimentation procedure, the mandatory extended calibration stage and sustaining accurate signal decoding in real-time. These limitations lead to a fragile BCI system that is demanding to use and only suited for deployment in a controlled laboratory. Overall contributions of this research aim to improve the robustness of BCI systems and enable new applications for use in the real-world

Desarrollo y análisis de estrategias avanzadas de interacción en sistemas robóticos complejos de rehabilitación y asistencia

Los últimos informes indican que la incidencia de los accidentes cerebro vasculares van en aumento. Los supervivientes de un accidente cerebro vascular generalmente experimentan hemiparesia, lo que provoca un deterioro de las extremidades que implica un notable deterioro de la calidad de vida. La escasez de recursos junto con la necesidad de rehabilitación y asistencia que presentan estas personas, hacen que el cuidado y los ejercicios de rehabilitación que proporcionan las plataformas robóticas de rehabilitación cobren aún más importancia en la actualidad y en los próximos años. La presente Tesis Doctoral se centra en el desarrollo de una arquitectura multimodal capaz de implementar sistemas robóticos complejos de rehabilitación y asistencia. Con esta arquitectura, se plantea implementar y evaluar los siguientes sistemas. Primero, llevar a cabo la implementación de un sistema complejo de robótica asistencial. Después, desarrollar un sistema para la realización de terapias competitivas. Seguidamente, implementar un sistema para realizar terapias cooperativas. Y finalmente, desarrollar un sistema capaz de implementar terapias de tele-rehabilitación mediante una estrategia de teleoperación maestro-esclavo. Los principales resultados de esta tesis se han publicado en dos artículos en revistas incluidas en el Journal Citation Reports (JCR). La publicación “Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living” se ha evaluado la implementación de un sistema complejo de robótica asistencial basado en la arquitectura multimodal desarrollada en esta tesis, en el que se ha integrado con éxito multitud de sensores junto con diferentes interfaces de control y dispositivos robóticos para la creación de un sistema autónomo capaz de ayudar a un usuario a realizar actividades de la vida diaria. Por otro lado, en la publicación “Differences in Physiological Reactions Due to a Competitive Rehabilitation Game Modality”, se han analizado los cambios que se producen en el estado afectivo del paciente en una terapia de neurorrehabilitación asistida por robots debidos a una modalidad de juego multijugador de tipo competitivo. Finalmente, cabe destacar que uno de los resultados de la presente tesis ha dado lugar a la patente ES1234596U: Dispositivo robótico interconectable para rehabilitación de extremidades