Volitional Control of Lower-limb Prosthesis with Vision-assisted Environmental Awareness

Early and reliable prediction of user’s intention to change locomotion mode or speed is critical for a smooth and natural lower limb prosthesis. Meanwhile, incorporation of explicit environmental feedback can facilitate context aware intelligent prosthesis which allows seamless operation in a variety of gait demands. This dissertation introduces environmental awareness through computer vision and enables early and accurate prediction of intention to start, stop or change speeds while walking. Electromyography (EMG), Electroencephalography (EEG), Inertial Measurement Unit (IMU), and Ground Reaction Force (GRF) sensors were used to predict intention to start, stop or increase walking speed. Furthermore, it was investigated whether external emotional music stimuli could enhance the predictive capability of intention prediction methodologies. Application of advanced machine learning and signal processing techniques on pre-movement EEG resulted in an intention prediction system with low latency, high sensitivity and low false positive detection. Affective analysis of EEG suggested that happy music stimuli significantly (

Electroencephalographic recording of the movement-related cortical potential in ecologically-valid movements:A scoping review

The movement-related cortical potential (MRCP) is a brain signal that can be recorded using surface electroencephalography (EEG) and represents the cortical processes involved in movement preparation. The MRCP has been widely researched in simple, single-joint movements, however, these movements often lack ecological validity. Ecological validity refers to the generalizability of the findings to real-world situations, such as neurological rehabilitation. This scoping review aimed to synthesize the research evidence investigating the MRCP in ecologically valid movement tasks. A search of six electronic databases identified 102 studies that investigated the MRCP during multi-joint movements; 59 of these studies investigated ecologically valid movement tasks and were included in the review. The included studies investigated 15 different movement tasks that were applicable to everyday situations, but these were largely carried out in healthy populations. The synthesized findings suggest that the recording and analysis of MRCP signals is possible in ecologically valid movements, however the characteristics of the signal appear to vary across different movement tasks (i.e., those with greater complexity, increased cognitive load, or a secondary motor task) and different populations (i.e., expert performers, people with Parkinson’s Disease, and older adults). The scarcity of research in clinical populations highlights the need for further research in people with neurological and age-related conditions to progress our understanding of the MRCPs characteristics and to determine its potential as a measure of neurological recovery and intervention efficacy. MRCP-based neuromodulatory interventions applied during ecologically valid movements were only represented in one study in this review as these have been largely delivered during simple joint movements. No studies were identified that used ecologically valid movements to control BCI-driven external devices; this may reflect the technical challenges associated with accurately classifying functional movements from MRCPs. Future research investigating MRCP-based interventions should use movement tasks that are functionally relevant to everyday situations. This will facilitate the application of this knowledge into the rehabilitation setting

Development of an EEG-based recurrent neural network for online gait decoding

Recent neuroscientific literature has shown that the use of brain-controlled robotic exoskeletons in walking rehabilitation induces neuroplasticity modi- fications, possibly leading to a higher likelihood of recovery and maintenance of lost motor functions due to a neural lesion, with respect to traditional re- habilitation. However, the gait decoding from brain signals remains an open challenge. The aim of this work is to implement and validate a deep learning model for online gait decoding that exploits Electroencephalography (EEG) infor- mation to predict the intention of initiating a step, which could be used to trigger the assistance of a lower-limb exoskeleton. In particular, the model exploits a Gated Recurrent Units (GRU) deep neural network to handle the time-dependent features which were identified by analysing the neural cor- relates preceding the step onset (i.e., Movement-Related Cortical Potentials (MRCP)). The network was evaluated on a pre-recorded dataset of 11 healthy subjects walking on a treadmill. The network’s architecture (e.g., number of GRU units) was optimized through grid search. In addition, to deal with the data scarcity problem of neurophysiological applications, I proposed a data augmentation procedure to increase the dataset available to train the model of each subject. With the proposed approach, the model achieved an average accuracy in detecting the step onset of 89.7 ± 7.7% with just the 15% of the dataset for each subject (∼70 steps), and up to 97.8 ± 1.3% with the whole dataset (∼440 steps). This thesis support the use of a memory-based deep learning model to de- code walking activity from non-invasive brain recordings. In future works, this model will be exploited in real time as a more effective input for devices restoring locomotion in impaired people, such as robotic exoskeletons.Recent neuroscientific literature has shown that the use of brain-controlled robotic exoskeletons in walking rehabilitation induces neuroplasticity modi- fications, possibly leading to a higher likelihood of recovery and maintenance of lost motor functions due to a neural lesion, with respect to traditional re- habilitation. However, the gait decoding from brain signals remains an open challenge. The aim of this work is to implement and validate a deep learning model for online gait decoding that exploits Electroencephalography (EEG) infor- mation to predict the intention of initiating a step, which could be used to trigger the assistance of a lower-limb exoskeleton. In particular, the model exploits a Gated Recurrent Units (GRU) deep neural network to handle the time-dependent features which were identified by analysing the neural cor- relates preceding the step onset (i.e., Movement-Related Cortical Potentials (MRCP)). The network was evaluated on a pre-recorded dataset of 11 healthy subjects walking on a treadmill. The network’s architecture (e.g., number of GRU units) was optimized through grid search. In addition, to deal with the data scarcity problem of neurophysiological applications, I proposed a data augmentation procedure to increase the dataset available to train the model of each subject. With the proposed approach, the model achieved an average accuracy in detecting the step onset of 89.7 ± 7.7% with just the 15% of the dataset for each subject (∼70 steps), and up to 97.8 ± 1.3% with the whole dataset (∼440 steps). This thesis support the use of a memory-based deep learning model to de- code walking activity from non-invasive brain recordings. In future works, this model will be exploited in real time as a more effective input for devices restoring locomotion in impaired people, such as robotic exoskeletons

A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to do something and transforms them into a control command for controlling external devices. The feelings described by the participants when they control a self-avatar in an immersive virtual environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar (ownership illusion). It has recently been shown that inducing the ownership illusion and then manipulating the movements of one’s self-avatar can lead to compensatory motor control strategies. In order to maximize this effect, there is a need for a method that measures and monitors embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong ownership illusion. This is particularly true given that reaching a high level of both BCI performance and embodiment are inter-connected. To reach one of them, the second must be reached as well. Some limitations of many existing systems hinder their adoption for neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most systems allow the user to take single steps or to walk but do not allow both, which prevents users from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or self-paced), which prevents users from progressing from machine-dependent to machine-independent walking. Overcoming the aforementioned limitations can be done by combining different control modes and options in one single system. However, this would have a negative impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool. In this case, there will be a need to enhance BCI performance. For such purpose, many techniques have been used in the literature, such as providing modified feedback (whereby the presented feedback is not consistent with the user’s MI), sequential training (recalibrating the classifier as more data becomes available). This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility of measuring the level of embodiment of an immersive self-avatar, during the performing, observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting visual feedback that conflicts with the desired movement of embodied participants. The objective of study 2 was to develop and validate a BCI to control single steps and forward walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in cue-paced and self-paced modes. Different performance enhancement strategies were implemented to increase BCI performance. The data of these two studies were then used in study 3 to construct a generic classifier that could eliminate offline calibration for future users and shorten training time. Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD) from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with the contralateral limb or stopped walking before the participant stopped (modified feedback). In study 2, participants completed a 4-day sequential training to control the gait of an avatar in both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking (continuous control mode). The avatar moved as a response to two calibrated regularized linear discriminant analysis (RLDA) classifiers that used the μ power spectral density (PSD) over the foot area of the motor cortex as features. The classifiers were retrained after every session. During the training, and for some of the trials, positive modified feedback was presented to half of the participants, where the avatar moved correctly regardless of the participant’s real performance. In both studies, the participants’ subjective experience was analyzed using a questionnaire. Results of study 1 show that subjective levels of embodiment correlate strongly with the power differences of the event-related synchronization (ERS) within the μ frequency band, and over the motor and pre-motor cortices between the modified and regular feedback trials. Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced BCI in both modes. For the cue-paced BCI, the average offline performance (classification rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback group. For self-paced BCI, the average performance was 83% at switch control and 92% at continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models performed as well as models obtained from participant-specific offline data. The results show that there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont été utilisées pour contrôler la marche d'un égo-avatar virtuel dans le but d'être utilisées dans la réadaptation de la marche. Une ICO décode les signaux du cerveau représentant un désir de faire produire un mouvement et les transforme en une commande de contrôle pour contrôler des appareils externes. Les sentiments décrits par les participants lorsqu'ils contrôlent un égo-avatar dans un environnement virtuel immersif démontrent que les humains peuvent être incarnés dans un corps d'un avatar (illusion de propriété). Il a été récemment démontré que provoquer l’illusion de propriété puis manipuler les mouvements de l’égo-avatar peut conduire à des stratégies de contrôle moteur compensatoire. Afin de maximiser cet effet, il existe un besoin d'une méthode qui mesure et surveille les niveaux d’incarnation des participants immergés dans la réalité virtuelle (RV) pour induire et maintenir une forte illusion de propriété. D'autre part, atteindre un niveau élevé de performances (taux de classification) ICO et d’incarnation est interconnecté. Pour atteindre l'un d'eux, le second doit également être atteint. Certaines limitations de plusieurs de ces systèmes entravent leur adoption pour la neuroréhabilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systèmes permettent à l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas à un utilisateur de passer des pas à la marche; 3- la plupart fonctionnent en un seul mode d’ICO, rythmé (cue-paced) ou auto-rythmé (self-paced). Surmonter les limitations susmentionnées peut être fait en combinant différents modes et options de commande dans un seul système. Cependant, cela aurait un impact négatif sur les performances de l’ICO, diminuant ainsi son utilité en tant qu'outil potentiel de réhabilitation. Dans ce cas, il sera nécessaire d'améliorer les performances des ICO. À cette fin, de nombreuses techniques ont été utilisées dans la littérature, telles que la rétroaction modifiée, le recalibrage du classificateur et l'utilisation d'un classificateur générique. Le projet de cette thèse a été réalisé en 3 études, avec objectif d'étudier dans l'étude 1, la possibilité de mesurer le niveau d'incarnation d'un égo-avatar immersif, lors de l'exécution, de l'observation et de l'imagination de la marche, à l'aide des techniques encéphalogramme (EEG), en présentant une rétroaction visuelle qui entre en conflit avec la commande du contrôle moteur des sujets incarnés. L'objectif de l'étude 2 était de développer un BCI pour contrôler les pas et la marche vers l’avant d'un égo-avatar dans la réalité virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmés et auto-rythmés. Différentes stratégies d'amélioration des performances ont été mises en œuvre pour augmenter la performance (taux de classification) de l’ICO. Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation. Vingt participants sains différents ont participé aux études 1 et 2. Dans l'étude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affiché dans un casque de RV du point de vue de la première personne (1PP). Ils ont été invités à performer, à regarder ou à imaginer un seul pas en avant ou la marche vers l’avant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatéral ou a arrêté de marcher avant que le participant ne s'arrête (rétroaction modifiée). Dans l'étude 2, les participants ont participé à un entrainement séquentiel de 4 jours pour contrôler la marche d'un avatar dans les deux modes de l’ICO. En mode rythmé, ils ont imaginé un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers l’avant . En mode auto-rythmé, il leur a été demandé d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrôle intermittent) ou en maintenir l'IM de marche vers l’avant (mode de contrôle continu). L'avatar s'est déplacé en réponse à deux classificateurs ‘Regularized Linear Discriminant Analysis’ (RLDA) calibrés qui utilisaient comme caractéristiques la densité spectrale de puissance (Power Spectral Density; PSD) des bandes de fréquences µ (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont été recalibrés après chaque session. Au cours de l’entrainement et pour certains des essais, une rétroaction modifiée positive a été présentée à la moitié des participants, où l'avatar s'est déplacé correctement quelle que soit la performance réelle du participant. Dans les deux études, l'expérience subjective des participants a été analysée à l'aide d'un questionnaire. Les résultats de l'étude 1 montrent que les niveaux subjectifs d’incarnation sont fortement corrélés à la différence de la puissance de la synchronisation liée à l’événement (Event-Related Synchronization; ERS) sur la bande de fréquence μ et sur le cortex moteur et prémoteur entre les essais de rétroaction modifiés et réguliers. L'étude 2 a montré que tous les participants étaient capables d’utiliser le BCI rythmé et auto-rythmé dans les deux modes. Pour le BCI rythmé, la performance hors ligne moyenne au jour 1 était de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amélioré la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne était de 85,9±8,4% pour le groupe de rétroaction modifié (77-97%) contre 75% pour le groupe de rétroaction non modifié. Pour le BCI auto-rythmé, la performance moyenne était de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de l’ICO ont été améliorées par la rétroaction modifiée (p = 0,001). Enfin, les résultats de l'étude 3 montrent que pour la classification des initialisations des pas et de la marche, il a été possible de construire des modèles génériques à partir de données hors ligne spécifiques aux participants. Les résultats montrent la possibilité de concevoir une ICO ne nécessitant aucun entraînement spécifique au participant

Physical Diagnosis and Rehabilitation Technologies

The book focuses on the diagnosis, evaluation, and assistance of gait disorders; all the papers have been contributed by research groups related to assistive robotics, instrumentations, and augmentative devices

Enhancement of Robot-Assisted Rehabilitation Outcomes of Post-Stroke Patients Using Movement-Related Cortical Potential

Post-stroke rehabilitation is essential for stroke survivors to help them regain independence and to improve their quality of life. Among various rehabilitation strategies, robot-assisted rehabilitation is an efficient method that is utilized more and more in clinical practice for motor recovery of post-stroke patients. However, excessive assistance from robotic devices during rehabilitation sessions can make patients perform motor training passively with minimal outcome. Towards the development of an efficient rehabilitation strategy, it is necessary to ensure the active participation of subjects during training sessions. This thesis uses the Electroencephalography (EEG) signal to extract the Movement-Related Cortical Potential (MRCP) pattern to be used as an indicator of the active engagement of stroke patients during rehabilitation training sessions. The MRCP pattern is also utilized in designing an adaptive rehabilitation training strategy that maximizes patients’ engagement. This project focuses on the hand motor recovery of post-stroke patients using the AMADEO rehabilitation device (Tyromotion GmbH, Austria). AMADEO is specifically developed for patients with fingers and hand motor deficits. The variations in brain activity are analyzed by extracting the MRCP pattern from the acquired EEG data during training sessions. Whereas, physical improvement in hand motor abilities is determined by two methods. One is clinical tests namely Fugl-Meyer Assessment (FMA) and Motor Assessment Scale (MAS) which include FMA-wrist, FMA-hand, MAS-hand movements, and MAS-advanced hand movements’ tests. The other method is the measurement of hand-kinematic parameters using the AMADEO assessment tool which contains hand strength measurements during flexion (force-flexion), and extension (force-extension), and Hand Range of Movement (HROM)

Sensor Approach for Brain Pathophysiology of Freezing of Gait in Parkinson\u27s Disease Patients

Parkinson\u27s Disease (PD) affects over 1% of the population over 60 years of age and is expected to reach 1 million in the USA by the year 2020, growing by 60 thousand each year. It is well understood that PD is characterized by dopaminergic loss, leading to decreased executive function causing motor symptoms such as tremors, bradykinesia, dyskinesia, and freezing of gait (FoG) as well as non-motor symptoms such as loss of smell, depression, and sleep abnormalities. A PD diagnosis is difficult to make since there is no worldwide approved test and difficult to manage since its manifestations are widely heterogeneous among subjects. Thus, understanding the patient subsets and the neural biomarkers that set them apart will lead to improved personalized care. To explore the physiological alternations caused by PD on neurological pathways and their effect on motor control, it is necessary to detect the neural activity and its dissociation with healthy physiological function. To this effect, this study presents a custom ultra-wearable sensor solution, consisting of electroencephalograph, electromyograph, ground reaction force, and symptom measurement sensors for the exploration of neural biomarkers during active gait paradigms. Additionally, this study employed novel de-noising techniques for dealing with the motion artifacts associated with active gait EEG recordings and compared time-frequency features between a group of PD with FoG and a group of age-matched controls and found significant differences between several EEG frequency bands during start and end of normal walking (with a p\u3c0.05)

Anticipatory detection of turning in humans for intuitive control of robotic mobility assistance

Many wearable lower-limb robots for walking assistance have been developed in recent years. However, it remains unclear how they can be commanded in an intuitive and efficient way by their user. In particular, providing robotic assistance to neurologically impaired individuals in turning remains a significant challenge. The control should be safe to the users and their environment, yet yield sufficient performance and enable natural human-machine interaction. Here, we propose using the head and trunk anticipatory behaviour in order to detect the intention to turn in a natural, non-intrusive way, and use it for triggering turning movement in a robot for walking assistance. We therefore study head and trunk orientation during locomotion of healthy adults, and investigate upper body anticipatory behaviour during turning. The collected walking and turning kinematics data are clustered using the k-means algorithm and cross-validation tests and k-nearest neighbours method are used to evaluate the performance of turning detection during locomotion. Tests with seven subjects exhibited accurate turning detection. Head anticipated turning by more than 400–500 ms in average across all subjects. Overall, the proposed method detected turning 300 ms after its initiation and 1230 ms before the turning movement was completed. Using head anticipatory behaviour enabled to detect turning faster by about 100 ms, compared to turning detection using only pelvis orientation measurements. Finally, it was demonstrated that the proposed turning detection can improve the quality of human–robot interaction by improving the control accuracy and transparency

Brain-Computer interface control of stepping from invasive electrocorticography upper-limb motor imagery in a patient with quadriplegia

Introduction: Most spinal cord injuries (SCI) result in lower extremities paralysis, thus diminishing ambulation. Using brain-computer interfaces (BCI), patients may regain leg control using neural signals that actuate assistive devices. Here, we present a case of a subject with cervical SCI with an implanted electrocorticography (ECoG) device and determined whether the system is capable of motor-imagery-initiated walking in an assistive ambulator.Methods: A 24-year-old male subject with cervical SCI (C5 ASIA A) was implanted before the study with an ECoG sensing device over the sensorimotor hand region of the brain. The subject used motor-imagery (MI) to train decoders to classify sensorimotor rhythms. Fifteen sessions of closed-loop trials followed in which the subject ambulated for one hour on a robotic-assisted weight-supported treadmill one to three times per week. We evaluated the stability of the best-performing decoder over time to initiate walking on the treadmill by decoding upper-limb (UL) MI.Results: An online bagged trees classifier performed best with an accuracy of 84.15% averaged across 9 weeks. Decoder accuracy remained stable following throughout closed-loop data collection.Discussion: These results demonstrate that decoding UL MI is a feasible control signal for use in lower-limb motor control. Invasive BCI systems designed for upper-extremity motor control can be extended for controlling systems beyond upper extremity control alone. Importantly, the decoders used were able to use the invasive signal over several weeks to accurately classify MI from the invasive signal. More work is needed to determine the long-term consequence between UL MI and the resulting lower-limb control