Upper Limb Posture Estimation in Robotic and Virtual Reality-based Rehabilitation.

New motor rehabilitation therapies include virtual reality (VR) and robotic technologies. In limb rehabilitation, limb posture is required to (1) provide a limb realistic representation in VR games and (2) assess the patient improvement. When exoskeleton devices are used in the therapy, the measurements of their joint angles cannot be directly used to represent the posture of the patient limb, since the human and exoskeleton kinematic models differ. In response to this shortcoming, we propose a method to estimate the posture of the human limb attached to the exoskeleton. We use the exoskeleton joint angles measurements and the constraints of the exoskeleton on the limb to estimate the human limb joints angles. This paper presents (a) the mathematical formulation and solution to the problem, (b) the implementation of the proposed solution on a commercial exoskeleton system for the upper limb rehabilitation, (c) its integration into a rehabilitation VR game platform, and (d) the quantitative assessment of the method during elbow and wrist analytic training. Results show that this method properly estimates the limb posture to (i) animate avatars that represent the patient in VR games and (ii) obtain kinematic data for the patient assessment during elbow and wrist analytic rehabilitation

Inverse kinematics for upper limb compound movement estimation in exoskeleton-assisted rehabilitation

Robot-Assisted Rehabilitation (RAR) is relevant for treating patients affected by nervous system injuries (e.g., stroke and spinal cord injury) -- The accurate estimation of the joint angles of the patient limbs in RAR is critical to assess the patient improvement -- The economical prevalent method to estimate the patient posture in Exoskeleton-based RAR is to approximate the limb joint angles with the ones of the Exoskeleton -- This approximation is rough since their kinematic structures differ -- Motion capture systems (MOCAPs) can improve the estimations, at the expenses of a considerable overload of the therapy setup -- Alternatively, the Extended Inverse Kinematics Posture Estimation (EIKPE) computational method models the limb and Exoskeleton as differing parallel kinematic chains -- EIKPE has been tested with single DOFmovements of the wrist and elbow joints -- This paper presents the assessment of EIKPEwith elbow-shoulder compoundmovements (i.e., object prehension) -- Ground-truth for estimation assessment is obtained from an optical MOCAP (not intended for the treatment stage) -- The assessment shows EIKPE rendering a good numerical approximation of the actual posture during the compoundmovement execution, especially for the shoulder joint angles -- This work opens the horizon for clinical studies with patient groups, Exoskeleton models, and movements types -

Development of an intelligent robotic system for rehabilitation of upper limbs using a collaborative robot

Mestrado de dupla diplomação com a UTFPR - Universidade Tecnológica Federal do ParanáRehabilitation is a relevant process for the recovery from dysfunctions and improves the realisation of patient’s Activities ofDaily Living (ADLs). Therefore, the development of technologies for this field has significant importance because the improvement of the rehabilitation can affect many people. This work proposes a robotic system for the rehabilitation of the upper limbs using a collaborative robot and an intelligent control algorithm that makes the solution robust and adaptable to each patient. The UR3 from Universal Robots© was used to implement two Reinforcement Learning algorithms, the SARSA and Q-learning, applied to this rehabilitation problem. The goal of this system provides a common training force applying resistance on the movement performed by the patient. This thesis is divided into twomain parts. The first one was the development of a simulation composed by the UR3 and a human model in V-REP platformthat could be controlled through a dedicated interface or externally through the MATLAB using the self-control algorithms. This simulation was created with a graphical interface for visualisation, and a human-machine interface, to control the robotic system with RL algorithm, built onMATLAB. The results obtained with the simulation presented the expected system behaviour. The second part was the experiment of the real system with a healthy subject. The experiment was divided in two phases the first considering the training only in one axis and second in the three Cartesian axes. The used algorithms were the same as the simulation, but in this case, they were implemented in Python language. The experiment considering one axis presents satisfactory results, while for the three axes the results were not so good. The obtained results with the real system experiment for one and three axis were compared with the human armmodel proposed in other studies to validate the applied methodology. This work represents an important contribution for the field because presents a new feature to help therapists and patients to get better results in the rehabilitation process.A reabilitação é um processo relevante para a recuperação de disfunções e para uma melhor realização das Atividades de Vida Diária (AVDs) do paciente. Portanto, o desenvolvimento de tecnologias para este campo tem uma importância significativa, pois o aprimoramento da reabilitação pode afetar muitas pessoas. Este trabalho propõe um sistema robótico para a reabilitação dos membros superiores utilizando umrobô colaborativo e umalgoritmo de controle inteligente, o que torna a solução robusta e adaptável para cada paciente. O robô UR3 da Universal Robots© foi usado como base para a implementação de dois algoritmos de Reinforcement Learning (RL), o SARSA e o Q-learning, aplicados a esse problema de reabilitação. O objetivo deste sistema é fornecer umtreinamento de força comum, aplicando uma resistência ao movimento realizado pelo paciente. Basicamente, esta tese está dividida em duas partes principais. A primeira foi o desenvolvimento de uma simulação composta pelo UR3 e um modelo do corpo humano na plataforma V-REP, que pudesse ser controlado através de uma interface dedicada ou, externamente, através do MATLAB usando os algoritmos de Reinforcement Learning. Essa simulação foi criada com uma interface gráfica para visualização e uma interface homem-máquina, para controlar o sistema robótico com o algoritmo RL, construído no MATLAB. Os resultados obtidos com a simulação apresentaram o comportamento esperado do sistema. A segunda parte foi o experimento do sistema real com umindivíduo saudável. O experimento foi dividido em duas fases: a primeira considerando o treinamento apenas em um eixo e a segunda nos três eixos cartesianos. Os algoritmos utilizados foram os mesmos da simulação,mas, neste caso, foramimplementados na linguagem Python. Os resultados são apresentados quer em simulação quer com o robô real e validam a metodologia desenvolvida e aplicada. Os resultados obtidos com o experimento real do sistema para a apenas umeixo foram comparados com a simulação do modelo do braço humano proposto em outros trabalhos para validar ametodologia aplicada. Este trabalho representa uma contribuição importante para o campo da reabilitação, pois apresenta umnovo recurso para ajudar terapeutas e pacientes a obter melhores resultados no processo de reabilitação

Une méthode de mesure du mouvement humain pour la programmation par démonstration

Programming by demonstration (PbD) is an intuitive approach to impart a task to a robot from one or several demonstrations by the human teacher. The acquisition of the demonstrations involves the solution of the correspondence problem when the teacher and the learner differ in sensing and actuation. Kinesthetic guidance is widely used to perform demonstrations. With such a method, the robot is manipulated by the teacher and the demonstrations are recorded by the robot's encoders. In this way, the correspondence problem is trivial but the teacher dexterity is afflicted which may impact the PbD process. Methods that are more practical for the teacher usually require the identification of some mappings to solve the correspondence problem. The demonstration acquisition method is based on a compromise between the difficulty of identifying these mappings, the level of accuracy of the recorded elements and the user-friendliness and convenience for the teacher. This thesis proposes an inertial human motion tracking method based on inertial measurement units (IMUs) for PbD for pick-and-place tasks. Compared to kinesthetic guidance, IMUs are convenient and easy to use but can present a limited accuracy. Their potential for PbD applications is investigated. To estimate the trajectory of the teacher's hand, 3 IMUs are placed on her/his arm segments (arm, forearm and hand) to estimate their orientations. A specific method is proposed to partially compensate the well-known drift of the sensor orientation estimation around the gravity direction by exploiting the particular configuration of the demonstration. This method, called heading reset, is based on the assumption that the sensor passes through its original heading with stationary phases several times during the demonstration. The heading reset is implemented in an integration and vector observation algorithm. Several experiments illustrate the advantages of this heading reset. A comprehensive inertial human hand motion tracking (IHMT) method for PbD is then developed. It includes an initialization procedure to estimate the orientation of each sensor with respect to the human arm segment and the initial orientation of the sensor with respect to the teacher attached frame. The procedure involves a rotation and a static position of the extended arm. The measurement system is thus robust with respect to the positioning of the sensors on the segments. A procedure for estimating the position of the human teacher relative to the robot and a calibration procedure for the parameters of the method are also proposed. At the end, the error of the human hand trajectory is measured experimentally and is found in an interval between $28.5$ mm and $61.8$ mm. The mappings to solve the correspondence problem are identified. Unfortunately, the observed level of accuracy of this IHMT method is not sufficient for a PbD process. In order to reach the necessary level of accuracy, a method is proposed to correct the hand trajectory obtained by IHMT using vision data. A vision system presents a certain complementarity with inertial sensors. For the sake of simplicity and robustness, the vision system only tracks the objects but not the teacher. The correction is based on so-called Positions Of Interest (POIs) and involves 3 steps: the identification of the POIs in the inertial and vision data, the pairing of the hand POIs to objects POIs that correspond to the same action in the task, and finally, the correction of the hand trajectory based on the pairs of POIs. The complete method for demonstration acquisition is experimentally evaluated in a full PbD process. This experiment reveals the advantages of the proposed method over kinesthesy in the context of this work.La programmation par démonstration est une approche intuitive permettant de transmettre une tâche à un robot à partir d'une ou plusieurs démonstrations faites par un enseignant humain. L'acquisition des démonstrations nécessite cependant la résolution d'un problème de correspondance quand les systèmes sensitifs et moteurs de l'enseignant et de l'apprenant diffèrent. De nombreux travaux utilisent des démonstrations faites par kinesthésie, i.e., l'enseignant manipule directement le robot pour lui faire faire la tâche. Ce dernier enregistre ses mouvements grâce à ses propres encodeurs. De cette façon, le problème de correspondance est trivial. Lors de telles démonstrations, la dextérité de l'enseignant peut être altérée et impacter tout le processus de programmation par démonstration. Les méthodes d'acquisition de démonstration moins invalidantes pour l'enseignant nécessitent souvent des procédures spécifiques pour résoudre le problème de correspondance. Ainsi l'acquisition des démonstrations se base sur un compromis entre complexité de ces procédures, le niveau de précision des éléments enregistrés et la commodité pour l'enseignant. Cette thèse propose ainsi une méthode de mesure du mouvement humain par capteurs inertiels pour la programmation par démonstration de tâches de ``pick-and-place''. Les capteurs inertiels sont en effet pratiques et faciles à utiliser, mais sont d'une précision limitée. Nous étudions leur potentiel pour la programmation par démonstration. Pour estimer la trajectoire de la main de l'enseignant, des capteurs inertiels sont placés sur son bras, son avant-bras et sa main afin d'estimer leurs orientations. Une méthode est proposée afin de compenser partiellement la dérive de l'estimation de l'orientation des capteurs autour de la direction de la gravité. Cette méthode, appelée ``heading reset'', est basée sur l'hypothèse que le capteur passe plusieurs fois par son azimut initial avec des phases stationnaires lors d'une démonstration. Cette méthode est implémentée dans un algorithme d'intégration et d'observation de vecteur. Des expériences illustrent les avantages du ``heading reset''. Cette thèse développe ensuite une méthode complète de mesure des mouvements de la main humaine par capteurs inertiels (IHMT). Elle comprend une première procédure d'initialisation pour estimer l'orientation des capteurs par rapport aux segments du bras humain ainsi que l'orientation initiale des capteurs par rapport au repère de référence de l'humain. Cette procédure, consistant en une rotation et une position statique du bras tendu, est robuste au positionnement des capteurs. Une seconde procédure est proposée pour estimer la position de l'humain par rapport au robot et pour calibrer les paramètres de la méthode. Finalement, l'erreur moyenne sur la trajectoire de la main humaine est mesurée expérimentalement entre 28.5 mm et 61.8 mm, ce qui n'est cependant pas suffisant pour la programmation par démonstration. Afin d'atteindre le niveau de précision nécessaire, une nouvelle méthode est développée afin de corriger la trajectoire de la main par IHMT à partir de données issues d'un système de vision, complémentaire des capteurs inertiels. Pour maintenir une certaine simplicité et robustesse, le système de vision ne suit que les objets et pas l'enseignant. La méthode de correction, basée sur des ``Positions Of Interest (POIs)'', est constituée de 3 étapes: l'identification des POIs dans les données issues des capteurs inertiels et du système de vision, puis l'association de POIs liées à la main et de POIs liées aux objets correspondant à la même action, et enfin, la correction de la trajectoire de la main à partir des paires de POIs. Finalement, la méthode IHMT corrigée est expérimentalement évaluée dans un processus complet de programmation par démonstration. Cette expérience montre l'avantage de la méthode proposée sur la kinesthésie dans le contexte de ce travail