Volitional Control of Lower-limb Prosthesis with Vision-assisted Environmental Awareness

Early and reliable prediction of user’s intention to change locomotion mode or speed is critical for a smooth and natural lower limb prosthesis. Meanwhile, incorporation of explicit environmental feedback can facilitate context aware intelligent prosthesis which allows seamless operation in a variety of gait demands. This dissertation introduces environmental awareness through computer vision and enables early and accurate prediction of intention to start, stop or change speeds while walking. Electromyography (EMG), Electroencephalography (EEG), Inertial Measurement Unit (IMU), and Ground Reaction Force (GRF) sensors were used to predict intention to start, stop or increase walking speed. Furthermore, it was investigated whether external emotional music stimuli could enhance the predictive capability of intention prediction methodologies. Application of advanced machine learning and signal processing techniques on pre-movement EEG resulted in an intention prediction system with low latency, high sensitivity and low false positive detection. Affective analysis of EEG suggested that happy music stimuli significantly (

Enhancement of Robot-Assisted Rehabilitation Outcomes of Post-Stroke Patients Using Movement-Related Cortical Potential

Post-stroke rehabilitation is essential for stroke survivors to help them regain independence and to improve their quality of life. Among various rehabilitation strategies, robot-assisted rehabilitation is an efficient method that is utilized more and more in clinical practice for motor recovery of post-stroke patients. However, excessive assistance from robotic devices during rehabilitation sessions can make patients perform motor training passively with minimal outcome. Towards the development of an efficient rehabilitation strategy, it is necessary to ensure the active participation of subjects during training sessions. This thesis uses the Electroencephalography (EEG) signal to extract the Movement-Related Cortical Potential (MRCP) pattern to be used as an indicator of the active engagement of stroke patients during rehabilitation training sessions. The MRCP pattern is also utilized in designing an adaptive rehabilitation training strategy that maximizes patients’ engagement. This project focuses on the hand motor recovery of post-stroke patients using the AMADEO rehabilitation device (Tyromotion GmbH, Austria). AMADEO is specifically developed for patients with fingers and hand motor deficits. The variations in brain activity are analyzed by extracting the MRCP pattern from the acquired EEG data during training sessions. Whereas, physical improvement in hand motor abilities is determined by two methods. One is clinical tests namely Fugl-Meyer Assessment (FMA) and Motor Assessment Scale (MAS) which include FMA-wrist, FMA-hand, MAS-hand movements, and MAS-advanced hand movements’ tests. The other method is the measurement of hand-kinematic parameters using the AMADEO assessment tool which contains hand strength measurements during flexion (force-flexion), and extension (force-extension), and Hand Range of Movement (HROM)

Non-Linear Adapted Spatio-Temporal Filter for Single-Trial Identification of Movement-Related Cortical Potential

The execution or imagination of a movement is reflected by a cortical potential that can be recorded by electroencephalography (EEG) as Movement-Related Cortical Potentials (MRCPs). The identification of MRCP from a single trial is a challenging possibility to get a natural control of a Brain–Computer Interface (BCI). We propose a novel method for MRCP detection based on optimal non-linear filters, processing different channels of EEG including delayed samples (getting a spatio-temporal filter). Different outputs can be obtained by changing the order of the temporal filter and of the non-linear processing of the input data. The classification performances of these filters are assessed by cross-validation on a training set, selecting the best ones (adapted to the user) and performing a majority voting from the best three to get an output using test data. The method is compared to another state-of-the-art filter recently introduced by our group when applied to EEG data recorded from 16 healthy subjects either executing or imagining 50 self-paced upper-limb palmar grasps. The new approach has a median accuracy on the overall dataset of 80%, which is significantly better than that of the previous filter (i.e., 63%). It is feasible for online BCI system design with asynchronous, self-paced applications

Control of an ambulatory exoskeleton with a brain-machine interface for spinal cord injury gait rehabilitation

The closed-loop control of rehabilitative technologies by neural commands has shown a great potential to improve motor recovery in patients suffering from paralysis. Brain-machine interfaces (BMI) can be used as a natural control method for such technologies. BMI provides a continuous association between the brain activity and peripheral stimulation, with the potential to induce plastic changes in the nervous system. Paraplegic patients, and especially the ones with incomplete injuries, constitute a potential target population to be rehabilitated with brain-controlled robotic systems, as they may improve their gait function after the reinforcement of their spared intact neural pathways. This paper proposes a closed-loop BMI system to control an ambulatory exoskeleton-without any weight or balance support-for gait rehabilitation of incomplete spinal cord injury (SCI) patients. The integrated system was validated with three healthy subjects, and its viability in a clinical scenario was tested with four SCI patients. Using a cue-guided paradigm, the electroencephalographic signals of the subjects were used to decode their gait intention and to trigger the movements of the exoskeleton. We designed a protocol with a special emphasis on safety, as patients with poor balance were required to stand and walk. We continuously monitored their fatigue and exertion level, and conducted usability and user-satisfaction tests after the experiments. The results show that, for the three healthy subjects, 84.44 ± 14.56% of the trials were correctly decoded. Three out of four patients performed at least one successful BMI session, with an average performance of 77.6 1 ± 14.72%. The shared control strategy implemented (i.e., the exoskeleton could only move during specific periods of time) was effective in preventing unexpected movements during periods in which patients were asked to relax. On average, 55.22 ± 16.69% and 40.45 ± 16.98% of the trials (for healthy subjects and patients, respectively) would have suffered from unexpected activations (i.e., false positives) without the proposed control strategy. All the patients showed low exertion and fatigue levels during the performance of the experiments. This paper constitutes a proof-of-concept study to validate the feasibility of a BMI to control an ambulatory exoskeleton by patients with incomplete paraplegia (i.e., patients with good prognosis for gait rehabilitation)

Sensor Approach for Brain Pathophysiology of Freezing of Gait in Parkinson\u27s Disease Patients

Parkinson\u27s Disease (PD) affects over 1% of the population over 60 years of age and is expected to reach 1 million in the USA by the year 2020, growing by 60 thousand each year. It is well understood that PD is characterized by dopaminergic loss, leading to decreased executive function causing motor symptoms such as tremors, bradykinesia, dyskinesia, and freezing of gait (FoG) as well as non-motor symptoms such as loss of smell, depression, and sleep abnormalities. A PD diagnosis is difficult to make since there is no worldwide approved test and difficult to manage since its manifestations are widely heterogeneous among subjects. Thus, understanding the patient subsets and the neural biomarkers that set them apart will lead to improved personalized care. To explore the physiological alternations caused by PD on neurological pathways and their effect on motor control, it is necessary to detect the neural activity and its dissociation with healthy physiological function. To this effect, this study presents a custom ultra-wearable sensor solution, consisting of electroencephalograph, electromyograph, ground reaction force, and symptom measurement sensors for the exploration of neural biomarkers during active gait paradigms. Additionally, this study employed novel de-noising techniques for dealing with the motion artifacts associated with active gait EEG recordings and compared time-frequency features between a group of PD with FoG and a group of age-matched controls and found significant differences between several EEG frequency bands during start and end of normal walking (with a p\u3c0.05)

Neurological Tremor: Sensors, Signal Processing and Emerging Applications

Neurological tremor is the most common movement disorder, affecting more than 4% of elderly people. Tremor is a non linear and non stationary phenomenon, which is increasingly recognized. The issue of selection of sensors is central in the characterization of tremor. This paper reviews the state-of-the-art instrumentation and methods of signal processing for tremor occurring in humans. We describe the advantages and disadvantages of the most commonly used sensors, as well as the emerging wearable sensors being developed to assess tremor instantaneously. We discuss the current limitations and the future applications such as the integration of tremor sensors in BCIs (brain-computer interfaces) and the need for sensor fusion approaches for wearable solutions

Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

BCI applications based on artificial intelligence oriented to deep learning techniques

A Brain-Computer Interface, BCI, can decode the brain signals corresponding to the intentions of individuals who have lost neuromuscular connection, to reestablish communication to control external devices. To this aim, BCI acquires brain signals as Electroencephalography (EEG) or Electrocorticography (ECoG), uses signal processing techniques and extracts features to train classifiers for providing proper control instructions. BCI development has increased in the last decades, improving its performance through the use of different signal processing techniques for feature extraction and artificial intelligence approaches for classification, such as deep learning-oriented classifiers. All of these can assure more accurate assistive systems but also can enable an analysis of the learning process of signal characteristics for the classification task. Initially, this work proposes the use of a priori knowledge and a correlation measure to select the most discriminative ECoG signal electrodes. Then, signals are processed using spatial filtering and three different types of temporal filtering, followed by a classifier made of stacked autoencoders and a softmax layer to discriminate between ECoG signals from two types of visual stimuli. Results show that the average accuracy obtained is 97% (+/- 0.02%), which is similar to state-of-the-art techniques, nevertheless, this method uses minimal prior physiological and an automated statistical technique to select some electrodes to train the classifier. Also, this work presents classifier analysis, figuring out which are the most relevant signal features useful for visual stimuli classification. The features and physiological information such as the brain areas involved are compared. Finally, this research uses Convolutional Neural Networks (CNN) or Convnets to classify 5 categories of motor tasks EEG signals. Movement-related cortical potentials (MRCPs) are used as a priori information to improve the processing of time-frequency representation of EEG signals. Results show an increase of more than 25% in average accuracy compared to a state-of-the-art method that uses the same database. In addition, an analysis of CNN or ConvNets filters and feature maps is done to and the most relevant signal characteristics that can help classify the five types of motor tasks.DoctoradoDoctor en Ingeniería Eléctrica y Electrónic

Fusion of virtual reality and brain-machine interfaces for the assessment and rehabilitation of patients with spinal cord injury

La presente tesis está centrada en la utilización de nuevas tecnologías (Interfaces Cerebro-Máquina y Realidad Virtual). En la primera parte de la tesis se describe la definición y la aplicación de un conjunto de métricas para evaluar el estado funcional de los pacientes con lesión medular en el contexto de un sistema de realidad virtual para la rehabilitación de los miembros superiores. El objetivo de este primer estudio es demostrar que la realidad virtual puede utilizarse, en combinación con sensores inerciales para rehabilitar y evaluar simultáneamente. 15 pacientes con lesión medular llevaron a cabo 3 sesiones con el sistema de realidad virtual Toyra y se aplicó el conjunto definido de métricas a las grabaciones obtenidas con los sensores inerciales. Se encontraron correlaciones entre algunas de las métricas definidas y algunas de las escalas clínicas utilizadas con frecuencia en el contexto de la rehabilitación. En la segunda parte de la tesis se ha combinado una retroalimentación virtual con un estimulador eléctrico funcional (en adelante FES, por sus siglas en inglés Functional Electrical Stimulator), ambos controlados por un Interfaz Cerebro-Máquina (BMI por sus siglas en inglés Brain-Machine Interface), para desarrollar un nuevo tipo de enfoque terapéutico para los pacientes. El sistema ha sido utilizado por 4 pacientes con lesión medular que intentaron mover sus manos. Esta intención desencadenó simultáneamente el FES y la retroalimentación virtual, cerrando la mano de los pacientes y mostrándoles una fuente adicional de retroalimentación para complementar la terapia. Este trabajo es, de acuerdo al estado del arte revisado, el primero que integra BMI, FES y realidad virtual como terapia para pacientes con lesión medular. Se han obtenido resultados clínicos prometedores por 4 pacientes con lesión medular después de realizar 5 sesiones de terapia con el sistema, mostrando buenos niveles de precisión en las diferentes sesiones (79,13% en promedio). En la tercera parte de la tesis se ha definido una nueva métrica para estudiar los cambios de conectividad cerebral en los pacientes con lesión medular, que incluye información de las interacciones neuronales entre diferentes áreas. El objetivo de este estudio ha sido extraer información clínicamente relevante de la actividad del EEG cuando se realizan terapias basadas en BMI