A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to do something and transforms them into a control command for controlling external devices. The feelings described by the participants when they control a self-avatar in an immersive virtual environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar (ownership illusion). It has recently been shown that inducing the ownership illusion and then manipulating the movements of one’s self-avatar can lead to compensatory motor control strategies. In order to maximize this effect, there is a need for a method that measures and monitors embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong ownership illusion. This is particularly true given that reaching a high level of both BCI performance and embodiment are inter-connected. To reach one of them, the second must be reached as well. Some limitations of many existing systems hinder their adoption for neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most systems allow the user to take single steps or to walk but do not allow both, which prevents users from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or self-paced), which prevents users from progressing from machine-dependent to machine-independent walking. Overcoming the aforementioned limitations can be done by combining different control modes and options in one single system. However, this would have a negative impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool. In this case, there will be a need to enhance BCI performance. For such purpose, many techniques have been used in the literature, such as providing modified feedback (whereby the presented feedback is not consistent with the user’s MI), sequential training (recalibrating the classifier as more data becomes available). This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility of measuring the level of embodiment of an immersive self-avatar, during the performing, observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting visual feedback that conflicts with the desired movement of embodied participants. The objective of study 2 was to develop and validate a BCI to control single steps and forward walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in cue-paced and self-paced modes. Different performance enhancement strategies were implemented to increase BCI performance. The data of these two studies were then used in study 3 to construct a generic classifier that could eliminate offline calibration for future users and shorten training time. Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD) from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with the contralateral limb or stopped walking before the participant stopped (modified feedback). In study 2, participants completed a 4-day sequential training to control the gait of an avatar in both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking (continuous control mode). The avatar moved as a response to two calibrated regularized linear discriminant analysis (RLDA) classifiers that used the μ power spectral density (PSD) over the foot area of the motor cortex as features. The classifiers were retrained after every session. During the training, and for some of the trials, positive modified feedback was presented to half of the participants, where the avatar moved correctly regardless of the participant’s real performance. In both studies, the participants’ subjective experience was analyzed using a questionnaire. Results of study 1 show that subjective levels of embodiment correlate strongly with the power differences of the event-related synchronization (ERS) within the μ frequency band, and over the motor and pre-motor cortices between the modified and regular feedback trials. Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced BCI in both modes. For the cue-paced BCI, the average offline performance (classification rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback group. For self-paced BCI, the average performance was 83% at switch control and 92% at continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models performed as well as models obtained from participant-specific offline data. The results show that there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont été utilisées pour contrôler la marche d'un égo-avatar virtuel dans le but d'être utilisées dans la réadaptation de la marche. Une ICO décode les signaux du cerveau représentant un désir de faire produire un mouvement et les transforme en une commande de contrôle pour contrôler des appareils externes. Les sentiments décrits par les participants lorsqu'ils contrôlent un égo-avatar dans un environnement virtuel immersif démontrent que les humains peuvent être incarnés dans un corps d'un avatar (illusion de propriété). Il a été récemment démontré que provoquer l’illusion de propriété puis manipuler les mouvements de l’égo-avatar peut conduire à des stratégies de contrôle moteur compensatoire. Afin de maximiser cet effet, il existe un besoin d'une méthode qui mesure et surveille les niveaux d’incarnation des participants immergés dans la réalité virtuelle (RV) pour induire et maintenir une forte illusion de propriété. D'autre part, atteindre un niveau élevé de performances (taux de classification) ICO et d’incarnation est interconnecté. Pour atteindre l'un d'eux, le second doit également être atteint. Certaines limitations de plusieurs de ces systèmes entravent leur adoption pour la neuroréhabilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systèmes permettent à l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas à un utilisateur de passer des pas à la marche; 3- la plupart fonctionnent en un seul mode d’ICO, rythmé (cue-paced) ou auto-rythmé (self-paced). Surmonter les limitations susmentionnées peut être fait en combinant différents modes et options de commande dans un seul système. Cependant, cela aurait un impact négatif sur les performances de l’ICO, diminuant ainsi son utilité en tant qu'outil potentiel de réhabilitation. Dans ce cas, il sera nécessaire d'améliorer les performances des ICO. À cette fin, de nombreuses techniques ont été utilisées dans la littérature, telles que la rétroaction modifiée, le recalibrage du classificateur et l'utilisation d'un classificateur générique. Le projet de cette thèse a été réalisé en 3 études, avec objectif d'étudier dans l'étude 1, la possibilité de mesurer le niveau d'incarnation d'un égo-avatar immersif, lors de l'exécution, de l'observation et de l'imagination de la marche, à l'aide des techniques encéphalogramme (EEG), en présentant une rétroaction visuelle qui entre en conflit avec la commande du contrôle moteur des sujets incarnés. L'objectif de l'étude 2 était de développer un BCI pour contrôler les pas et la marche vers l’avant d'un égo-avatar dans la réalité virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmés et auto-rythmés. Différentes stratégies d'amélioration des performances ont été mises en œuvre pour augmenter la performance (taux de classification) de l’ICO. Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation. Vingt participants sains différents ont participé aux études 1 et 2. Dans l'étude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affiché dans un casque de RV du point de vue de la première personne (1PP). Ils ont été invités à performer, à regarder ou à imaginer un seul pas en avant ou la marche vers l’avant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatéral ou a arrêté de marcher avant que le participant ne s'arrête (rétroaction modifiée). Dans l'étude 2, les participants ont participé à un entrainement séquentiel de 4 jours pour contrôler la marche d'un avatar dans les deux modes de l’ICO. En mode rythmé, ils ont imaginé un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers l’avant . En mode auto-rythmé, il leur a été demandé d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrôle intermittent) ou en maintenir l'IM de marche vers l’avant (mode de contrôle continu). L'avatar s'est déplacé en réponse à deux classificateurs ‘Regularized Linear Discriminant Analysis’ (RLDA) calibrés qui utilisaient comme caractéristiques la densité spectrale de puissance (Power Spectral Density; PSD) des bandes de fréquences µ (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont été recalibrés après chaque session. Au cours de l’entrainement et pour certains des essais, une rétroaction modifiée positive a été présentée à la moitié des participants, où l'avatar s'est déplacé correctement quelle que soit la performance réelle du participant. Dans les deux études, l'expérience subjective des participants a été analysée à l'aide d'un questionnaire. Les résultats de l'étude 1 montrent que les niveaux subjectifs d’incarnation sont fortement corrélés à la différence de la puissance de la synchronisation liée à l’événement (Event-Related Synchronization; ERS) sur la bande de fréquence μ et sur le cortex moteur et prémoteur entre les essais de rétroaction modifiés et réguliers. L'étude 2 a montré que tous les participants étaient capables d’utiliser le BCI rythmé et auto-rythmé dans les deux modes. Pour le BCI rythmé, la performance hors ligne moyenne au jour 1 était de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amélioré la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne était de 85,9±8,4% pour le groupe de rétroaction modifié (77-97%) contre 75% pour le groupe de rétroaction non modifié. Pour le BCI auto-rythmé, la performance moyenne était de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de l’ICO ont été améliorées par la rétroaction modifiée (p = 0,001). Enfin, les résultats de l'étude 3 montrent que pour la classification des initialisations des pas et de la marche, il a été possible de construire des modèles génériques à partir de données hors ligne spécifiques aux participants. Les résultats montrent la possibilité de concevoir une ICO ne nécessitant aucun entraînement spécifique au participant

Choreographic and Somatic Approaches for the Development of Expressive Robotic Systems

As robotic systems are moved out of factory work cells into human-facing environments questions of choreography become central to their design, placement, and application. With a human viewer or counterpart present, a system will automatically be interpreted within context, style of movement, and form factor by human beings as animate elements of their environment. The interpretation by this human counterpart is critical to the success of the system's integration: knobs on the system need to make sense to a human counterpart; an artificial agent should have a way of notifying a human counterpart of a change in system state, possibly through motion profiles; and the motion of a human counterpart may have important contextual clues for task completion. Thus, professional choreographers, dance practitioners, and movement analysts are critical to research in robotics. They have design methods for movement that align with human audience perception, can identify simplified features of movement for human-robot interaction goals, and have detailed knowledge of the capacity of human movement. This article provides approaches employed by one research lab, specific impacts on technical and artistic projects within, and principles that may guide future such work. The background section reports on choreography, somatic perspectives, improvisation, the Laban/Bartenieff Movement System, and robotics. From this context methods including embodied exercises, writing prompts, and community building activities have been developed to facilitate interdisciplinary research. The results of this work is presented as an overview of a smattering of projects in areas like high-level motion planning, software development for rapid prototyping of movement, artistic output, and user studies that help understand how people interpret movement. Finally, guiding principles for other groups to adopt are posited.Comment: Under review at MDPI Arts Special Issue "The Machine as Artist (for the 21st Century)" http://www.mdpi.com/journal/arts/special_issues/Machine_Artis

Robotic Platforms for Assistance to People with Disabilities

People with congenital and/or acquired disabilities constitute a great number of dependents today. Robotic platforms to help people with disabilities are being developed with the aim of providing both rehabilitation treatment and assistance to improve their quality of life. A high demand for robotic platforms that provide assistance during rehabilitation is expected because of the health status of the world due to the COVID-19 pandemic. The pandemic has resulted in countries facing major challenges to ensure the health and autonomy of their disabled population. Robotic platforms are necessary to ensure assistance and rehabilitation for disabled people in the current global situation. The capacity of robotic platforms in this area must be continuously improved to benefit the healthcare sector in terms of chronic disease prevention, assistance, and autonomy. For this reason, research about human–robot interaction in these robotic assistance environments must grow and advance because this topic demands sensitive and intelligent robotic platforms that are equipped with complex sensory systems, high handling functionalities, safe control strategies, and intelligent computer vision algorithms. This Special Issue has published eight papers covering recent advances in the field of robotic platforms to assist disabled people in daily or clinical environments. The papers address innovative solutions in this field, including affordable assistive robotics devices, new techniques in computer vision for intelligent and safe human–robot interaction, and advances in mobile manipulators for assistive tasks

System Identification of Bipedal Locomotion in Robots and Humans

The ability to perform a healthy walking gait can be altered in numerous cases due to gait disorder related pathologies. The latter could lead to partial or complete mobility loss, which affects the patients’ quality of life. Wearable exoskeletons and active prosthetics have been considered as a key component to remedy this mobility loss. The control of such devices knows numerous challenges that are yet to be addressed. As opposed to fixed trajectories control, real-time adaptive reference generation control is likely to provide the wearer with more intent control over the powered device. We propose a novel gait pattern generator for the control of such devices, taking advantage of the inter-joint coordination in the human gait. Our proposed method puts the user in the control loop as it maps the motion of healthy limbs to that of the affected one. To design such control strategy, it is critical to understand the dynamics behind bipedal walking. We begin by studying the simple compass gait walker. We examine the well-known Virtual Constraints method of controlling bipedal robots in the image of the compass gait. In addition, we provide both the mechanical and control design of an affordable research platform for bipedal dynamic walking. We then extend the concept of virtual constraints to human locomotion, where we investigate the accuracy of predicting lower limb joints angular position and velocity from the motion of the other limbs. Data from nine healthy subjects performing specific locomotion tasks were collected and are made available online. A successful prediction of the hip, knee, and ankle joints was achieved in different scenarios. It was also found that the motion of the cane alone has sufficient information to help predict good trajectories for the lower limb in stairs ascent. Better estimates were obtained using additional information from arm joints. We also explored the prediction of knee and ankle trajectories from the motion of the hip joints

Trunk Rehabilitation Using Cable-Driven Robotic Systems

Upper body control is required to complete many daily tasks. One needs to stabilize the head and trunk over the pelvis, as one shifts the center of mass to interact with the world. While healthy individuals can perform activities that require leaning, reaching, and grasping readily, those with neurological and musculoskeletal disorders present with control deficits. These deficits can lead to difficulty in shifting the body center of mass away from the stable midline, leading to functional limitations and a decline in the quality of activity. Often these patient groups use canes, walkers, and wheelchairs for support, leading to occasional strapping or joint locking of the body for trunk stabilization. Current rehabilitation strategies focus on isolated components of stability. This includes strengthening, isometric exercises, hand-eye coordination tasks, isolated movement, and proprioceptive training. Although all these components are evidence based and directly correlate to better stability, motor learning theories such as those by Nikolai Bernstein, suggest that task and context specific training can lead to better outcomes. In specific, based on our experimentation, we believe functional postural exploration, while encompassing aspects of strengthening, hand-eye coordination, and proprioceptive feedback can provide better results. In this work, we present two novel cable robotic platforms for seated and standing posture training. The Trunk Support Trainer (TruST) is a platform for seated posture rehabilitation that provides controlled external wrench on the human trunk in any direction in real-time. The Stand Trainer is a platform for standing posture rehabilitation that can control the trunk, pelvis, and knees, simultaneously. The system works through the use of novel force-field algorithms that are modular and user-specific. The control uses an assist-as-needed strategy to apply forces on the user during regions of postural instability. The device also allows perturbations for postural reactive training. We have conducted several studies using healthy adult populations and pilot studies on patient groups including cerebral palsy, cerebellar ataxia, and spinal cord injury. We propose new training methods that incorporate motor learning theory and objective interventions for improving posture control. We identify novel methods to characterize posture in form of the “8-point star test”. This is to assess the postural workspace. We also demonstrate novel methods for functional training of posture and balance. Our results show that training with our robotic platforms can change the trunk kinematics. Specifically, healthy adults are able to translate the trunk further and rotate the trunk more anteriorly in the seated position. In the standing position, they can alter their reach strategy to maintain the upper trunk more vertically while reaching. Similarly, Cerebral Palsy patients improve their trunk translations, reaching workspace, and maintain a more vertical posture after training, in the seated position. Our results also showed that an Ataxia patient was able to improve their reaching workspace and trunk translations in the standing position. Finally, our results show that the robotic platforms can successfully reduce trunk and pelvis sway in spinal cord injury patients. The results of the pilot studies suggest that training with our robotic platforms and methods is beneficial in improving trunk control

A 3-DoF Robotic Platform for the Rehabilitation and Assessment of Reaction Time and Balance Skills of MS Patients

The central nervous system (CNS) exploits anticipatory (APAs) and compensatory (CPAs) postural adjustments to maintain the balance.The postural adjustments comprising stability of the center of mass (CoM) and the pressure distribution of the body influence each other if there is a lack of performance in either of them.Any predictable or sudden perturbation may pave the way for the divergence of CoM from equilibrium and in homogeneous pressure distribution of the body.Such a situation is often observed in daily livings of Multiple Sclerosis (MS) patients owing to their poor APAs and CPAs, and induces their falls.The way of minimizing risk of falls in neurological patients is utilizing perturbation-based rehabilitation, as it is efficient in the recovery of the balance disorder.In the light of the findings, we present the design, implementation, and experimental evaluation of a novel 3 DoF parallel manipulator to treat the balance disorder of MS.The robotic platform allows angular motion of the ankle based on its anthropomorphic freedom.The end-effector endowed with upper and lower platforms is designed to evaluate both the pressure distribution of each foot and the CoM of the body, respectively.Data gathered from the platforms are utilized to both evaluate performance of the patients and used in high-level control of the robotic platform to regulate the difficulty level of tasks.In this study, kinematic and dynamic analyses of the robot are derived and validated in the simulation environment. Low-level control of the prototype is also successfully implemented through PID controller.The capacity of each platform is evaluated with a set of experiments considering assessment of pressure distribution and CoM of the foot-like-objects on the end-effector. Experimental results indicate that such a system well-address the need for balance skill training and assessment through the APAs and CPAs.Comment: 12 figures, 29 pages, PLOS ON

Dispositivo de realidade virtual para melhoria da marcha em pacientes com a doença de Parkinson

Dissertação de mestrado em Computer ScienceIn recent years there have been many improvements to medical procedures, involving the use of augmented reality technology to provide new innovative approaches to difficult tasks that are often required of the patients, requiring less physical exertion from the to achieve the same results or simply looking at the problem in a new perspective. Virtual reality technology has the capability of creating an interactive, motivating environment in which practice intensity and feedback can be manipulated to create individualised treatments to retrain movement. Currently there is a very large amount of people suffering from minor to severe functional limitations, impairments such as loss of range of motion, decreased reacting times, disordered movement organisation, and impaired force generation create deficits in motor control that effect the personss capacity for independent living and economic self-sufficiency. The use of augmented reality is starting to be used in more medical scenario’s and in the treatment of many diseases generally co-related with motor difficulties or recovery treatments. One of the diseases that has been looked more prominently for augmented reality development is the Parkinson’s disease which causes its patients to suffer severe gait constriction and whose generalised gait treatments didn’t produce a significant improvement in the patients gait without the use of heavy medication. One other important detail to take notice is that the Parkinsons disease causes the patient to abruptly enter a freezing state without any kind of warning which can lead the patient to fall and severally harm itself depending on the situation at hand. The objective of this thesis is to explore the possibilities of the use of augmented reality in an attempt to improve gait in patients suffering from Parkinson’s disease. For this purpose many augmented reality glasses were analysed selecting the best one in terms of affordability, comfort and utility. The application developed has the objective of improving the patients gait by displaying an augmented reality supper- imposed path for the patient to follow matching auditory cues with each of the patients steps and also helping the patient of he suddenly finds himself affected by a ”freezing” episode.Recentemente tem sido feitos vários melhoramentos nos procedimentos médicos, recorrendo ao uso de tecnologias como realidade aumentada para fornecer uma nova abordagem a tarefas complicadas que são frequentemente requeridas aos pacientes, requerendo um menor esforço físico e feedback imediato ou simplesmente para obter uma nova perspetiva sobre o problema em questão. O uso de realidade aumentada tem vindo a ser cada vez maior, sendo usado em cada vez mais procedimentos e para tratamento de variadas condições principalmente focadas em dificuldades motoras e fisioterapia. Uma das doenças que despertou maior interesse no uso de realidade aumentada no seu tratamento é a doença de Parkinson, conhecida por causa deterioramento nas capacidades motoras dos afetados causando problemas na marcha da pessoa que, afetam varias tarefas do seu dia a dia. Outro detalhe importante da doença de Parkinson é que os afetados também tem o que são chamados de episódios de ”congelamento” que acontecem quando o paciente de repente e sem nenhum aviso previ-o fica paralisado durante uns instantes, o que pode provocar a queda da pessoa. Estes episódios não são constantes podendo variar bastante na ocorrência e na intensidade de pessoa a pessoa. O objetivo desta dissertação é a exploração das possibilidades do uso de realidade aumentada numa tentativa de melhorar a marcha das pessoas afetadas com a doença de Parkinson. Para este propósito muitas ferramentas de realidade virtual foram examinadas escolhendo uma que seja o menos intrusiva possível para facilitar o uso pelo paciente e que tenha as especificações necessárias para o bem funcionar da aplicação. A aplicação de realidade virtual terá então o objetivo de melhorar a marcha do paciente através do seu uso mostrando ”pégadas” que irão servir para o paciente se orientar e ajudar o paciente quando ele estiver sobre o efeito de congelamento para evitar que cause danos graves a si próprio caso ocorra numa situação complicada

A 3-DoF robotic platform for the rehabilitation and assessment of reaction time and balance skills of MS patients

The central nervous system (CNS) exploits anticipatory (APAs) and compensatory (CPAs) postural adjustments to maintain the balance. The postural adjustments comprising stability of the center of mass (CoM) and the pressure distribution of the body influence each other if there is a lack of performance in either of them. Any predictable or sudden perturbation may pave the way for the divergence of CoM from equilibrium and inhomogeneous pressure distribution of the body. Such a situation is often observed in the daily lives of Multiple Sclerosis (MS) patients due to their poor APAs and CPAs and induces their falls. The way of minimizing the risk of falls in neurological patients is by utilizing perturbation-based rehabilitation, as it is efficient in the recovery of the balance disorder. In light of the findings, we present the design, implementation, and experimental evaluation of a novel 3 DoF parallel manipulator to treat the balance disorder of MS. The robotic platform allows angular motion of the ankle based on its anthropomorphic freedom. Moreover, the end-effector endowed with upper and lower platforms is designed to evaluate both the pressure distribution of each foot and the CoM of the body, respectively. Data gathered from the platforms are utilized to both evaluate the performance of the patients and used in high-level control of the robotic platform to regulate the difficulty level of tasks. In this study, kinematic and dynamic analyses of the robot are derived and validated in the simulation environment. Low-level control of the first prototype is also successfully implemented through the PID controller. The capacity of each platform is evaluated with a set of experiments considering the assessment of pressure distribution and CoM of the foot-like objects on the end-effector. The experimental results indicate that such a system well-address the need for balance skill training and assessment through the APAs and CPAs