Hand Gestures Replicating Robot Arm based on MediaPipe

A robotic arm is any variety of programmable mechanical devices designed to operate items like a human arm and is one of the most beneficial innovations of the 20th century, quickly becoming a cornerstone of many industries. It can perform a variety of tasks and duties that may be time-consuming, difficult, or dangerous to humans. The gesture-based control interface offers many opportunities for more natural, configurable, and easy human-machine interaction. It can expand the capabilities of the GUI and command line interfaces that we use today with the mouse and keyboard. This work proposed changing the concept of remote controls for operating a hand-operated robotic arm to get rid of buttons and joysticks by replacing them with a more intuitive approach to controlling a robotic arm via the hand gestures of the user. The proposed system performs vision-based hand gesture recognition and a robot arm that can replicate the user's hand gestures using image processing. The system detects and recognizes hand gestures using Python and sends a command to the microcontroller which is the Arduino board connected to the robot arm to replicate the recognized gesture. Five servo motors are connected to the Arduino Nano to control the fingers of the robot arm; These servos are related to the robot arm prototype. It is worth noting that this system was able to repeat the user's hand gestures with an accuracy of up to 96%

Robotic exoskeletons: A perspective for the rehabilitation of arm coordination in stroke patients

Upper-limb impairment after stroke is caused by weakness, loss of individual joint control, spasticity, and abnormal synergies. Upper-limb movement frequently involves abnormal, stereotyped, and fixed synergies, likely related to the increased use of sub-cortical networks following the stroke. The flexible coordination of the shoulder and elbow joints is also disrupted. New methods for motor learning, based on the stimulation of activity- dependent neural plasticity have been developed. These include robots that can adaptively assist active movements and generate many movement repetitions. However, most of these robots only control the movement of the hand in space. The aim of the present text is to analyze the potential of robotic exoskeletons to specifically rehabilitate joint motion and particularly inter-joint coordination. First, a review of studies on upper-limb coordination in stroke patients is presented and the potential for recovery of coordination is examined. Second, issues relating to the mechanical design of exoskeletons and the transmission of constraints between the robotic and human limbs are discussed. The third section considers the development of different methods to control exoskeletons: existing rehabilitation devices and approaches to the control and rehabilitation of joint coordinations are then reviewed, along with preliminary clinical results available. Finally, perspectives and future strategies for the design of control mechanisms for rehabilitation exoskeletons are discussed

Human to robot hand motion mapping methods: review and classification

In this article, the variety of approaches proposed in literature to address the problem of mapping human to robot hand motions are summarized and discussed. We particularly attempt to organize under macro-categories the great quantity of presented methods, that are often difficult to be seen from a general point of view due to different fields of application, specific use of algorithms, terminology and declared goals of the mappings. Firstly, a brief historical overview is reported, in order to provide a look on the emergence of the human to robot hand mapping problem as a both conceptual and analytical challenge that is still open nowadays. Thereafter, the survey mainly focuses on a classification of modern mapping methods under six categories: direct joint, direct Cartesian, taskoriented, dimensionality reduction based, pose recognition based and hybrid mappings. For each of these categories, the general view that associates the related reported studies is provided, and representative references are highlighted. Finally, a concluding discussion along with the authors’ point of view regarding future desirable trends are reported.This work was supported in part by the European Commission’s Horizon 2020 Framework Programme with the project REMODEL under Grant 870133 and in part by the Spanish Government under Grant PID2020-114819GB-I00.Peer ReviewedPostprint (published version

Effects of overground walking with a robotic exoskeleton on lower limb muscle synergies

Les exosquelettes robotisés de marche (ERM) représentent une intervention prometteuse dans le domaine de la réadaptation locomotrice. Sur le plan clinique, les ERM facilitent la mise en application de principes de neuroplasticité. Jusqu'à présent, la majorité des études analysant les effets de l’ERM a été menée avec des ERM fournissant une assistance robotique complète le long d’une trajectoire de mouvements prédéfinie des membres inférieurs (MI) de façon à reproduire la marche de façon quasi parfaite à très basse vitesse. La nouvelle génération d’ERM, maintenant disponible sur le marché, propose de nouveaux modes de contrôles qui permettent, entre autres, une liberté de mouvement accrue aux MIs (c.-à-d. trajectoire non imposée) et une possibilité d’offrir une assistance ou résistance aux mouvements de différentes intensités surtout pendant la phase d’oscillation du cycle de marche. Cependant, les effets de ces modes de contrôles sur la coordination musculaire des MI pendant la marche au sol avec l’ERM, caractérisé via l’extraction de synergies musculaires (SM), restent méconnus. Cette thèse mesure et compare les caractéristiques des SM (c.-à-d. nombre, profils d’activation, composition musculaire et contribution relative des muscles) pendant la la marche au sol sans ou avec un ERM paramétré avec six différents modes de contrôle chez des individus en bonne santé (articles #1 et #2) et d’autres ayant une lésion médullaire incomplète (LMI) (article #3). Les signaux électromyographiques (EMG) des différents muscles clés des MI, enregistrés lors de la marche, ont été utilisés afin d’extraire les SM avec un algorithme de factorisation matricielle non négative. La similarité des cosinus et les coefficients de corrélation ont caractérisé les similitudes entre les caractéristiques des SM. Les résultats montrent que: 1) les profils d'activation temporelle et le nombre de SM sont modifiés en fonction de la vitesse de marche avec, entre autres une augmentation de la vitesse de marche entrainant une fusion de SM, chez les individus en bonne santé marchant sans ERM ; 2) lorsque ces derniers marchent avec un ERM, les différents modes de contrôle testés ne dupliquent pas adéquatement les SM retrouvées lors de la marche sans ERM. En fait, uniquement le mode de contrôle libérant la contrainte de trajectoire de mouvements des MIs dans le plan sagittal lors de la phase d’oscillation reproduit les principales caractéristiques des SM retrouvées pendant la marche sans ERM ; 3) le nombre et la composition musculaire des SM sont modifiés pendant la marche sans ERM chez les personnes ayant une LMI. Cependant, parmi tous les modes de contrôle étudiés, seul le mode de contrôle libérant le contrôle de la trajectoire de mouvements des MI et assistant l’oscillation du MIs (c.-à-d. HASSIST) permets l’extraction de SM similaire à celles observées chez des individus en santé lors d'une marche sans ERM. Dans l’ensemble, cette thèse a mis en évidence le fait que différentes demandes biomécaniques liées à la marche (c.-à-d. vitesse de marche, modes de contrôle de l’ERM) modifient le nombre et les caractéristiques de SM chez les personnes en santé. Cette thèse a également confirmé que la coordination musculaire, mise en évidence via l’analyse de SM, est altérée chez les personnes ayant une LMI et a tendance à se normaliser lors de la marche avec l’ERM paramétré dans le mode de HASSIST. Les nouvelles preuves appuieront les professionnels de la réadaptation dans le processus de prise de décision concernant la sélection du mode de contrôle des MIs lors de l’entrainement locomoteur utilisant avec un ERM.Wearable robotic exoskeletons (WRE) represent a promising rehabilitation intervention for locomotor rehabilitation training that aligns with activity-based neuroplasticity principles in terms of optimal sensory input, massed repetition, and proper kinematics. Thus far, most studies that investigated the effects of WRE have used WRE that provide full robotic assistance and fixed trajectory guidance to the lower extremity (L/E) to generate close-to-normal walking kinematics, usually at very slow speeds. Based on clinicians’ feedback, current commercially-available WRE have additional control options to be able to integrate these devices into the recovery process of individuals who have maintained some ability to walk after an injury to the central nervous system. In this context, WRE now offer additional degrees of movements for the L/E to move freely and different strategies to assist or resist movement, particularly during the gait cycle’s swing phase. However, the extent that these additional WRE control options affect L/E neuromuscular control during walking, typically characterized using muscle synergies (MSs), remains unknown. This thesis measures and compares MSs characteristics (i.e., number, temporal activation profile, and muscles contributing to a specific synergy [weightings]) during typical overground walking, with and without a WRE, in six different control modes, in abled-bodied individuals (Articles #1 and #2) and individuals with incomplete spinal cord injury (iSCI; Article #3). Surface EMG of key L/E muscles were recorded while walking and used to extract MSs using a non-negative matrix factorization algorithm. Cosine similarity and correlation coefficients characterized, grouped, and indicated similarities between MS characteristics. Results demonstrated that: 1) the number of MSs and MS temporal activation profiles in able-bodied individuals walking without WRE are modified by walking speed and that, as speed increased, specific MSs were fused or merged compared to MSs at slow speeds; 2) In able-bodied individuals walking with WRE, few WRE control modes maintained the typical MSs characteristics that were found during overground walking without WRE. Moreover, freeing the L/E swing trajectory imposed by the WRE best reproduced those MSs characteristics during overground walking without the WRE; and 3) After an iSCI, alterations to the number and the composition of MSs were observed during walking without WRE. However, of all WRE control modes that were investigated, only HASSIST (i.e., freeing WRE control over L/E swing trajectory while assisting the user’s self-selected trajectory) reproduced the number and composition of MSs found in abled-bodied individuals during overground walking without WRE. Altogether, the results of this thesis demonstrated that different walking-related biomechanical demands (i.e., walking speed) and most of the WRE control modes can alter some MSs, and their characteristics, in able-bodied individuals. This research also confirmed that impaired muscle coordination, assessed via MSs, can adapt when walking with a WRE set with specific control options (e.g., HASSIST). These MS adaptations mimicked typical MS characteristics extracted during overground walking. The evidence generated by this thesis will support the decision-making process when selecting specific L/E control options during WRE walking, allowing rehabilitation professionals to refine WRE locomotor training protocols

On neuromechanical approaches for the study of biological and robotic grasp and manipulation

abstract: Biological and robotic grasp and manipulation are undeniably similar at the level of mechanical task performance. However, their underlying fundamental biological vs. engineering mechanisms are, by definition, dramatically different and can even be antithetical. Even our approach to each is diametrically opposite: inductive science for the study of biological systems vs. engineering synthesis for the design and construction of robotic systems. The past 20 years have seen several conceptual advances in both fields and the quest to unify them. Chief among them is the reluctant recognition that their underlying fundamental mechanisms may actually share limited common ground, while exhibiting many fundamental differences. This recognition is particularly liberating because it allows us to resolve and move beyond multiple paradoxes and contradictions that arose from the initial reasonable assumption of a large common ground. Here, we begin by introducing the perspective of neuromechanics, which emphasizes that real-world behavior emerges from the intimate interactions among the physical structure of the system, the mechanical requirements of a task, the feasible neural control actions to produce it, and the ability of the neuromuscular system to adapt through interactions with the environment. This allows us to articulate a succinct overview of a few salient conceptual paradoxes and contradictions regarding under-determined vs. over-determined mechanics, under- vs. over-actuated control, prescribed vs. emergent function, learning vs. implementation vs. adaptation, prescriptive vs. descriptive synergies, and optimal vs. habitual performance. We conclude by presenting open questions and suggesting directions for future research. We hope this frank and open-minded assessment of the state-of-the-art will encourage and guide these communities to continue to interact and make progress in these important areas at the interface of neuromechanics, neuroscience, rehabilitation and robotics.The electronic version of this article is the complete one and can be found online at: https://jneuroengrehab.biomedcentral.com/articles/10.1186/s12984-017-0305-

Concept of an exoskeleton for industrial applications with modulated impedance based on Electromyographic signal recorded from the operator

The introduction of an active exoskeleton that enhances the operator power in the manufacturing field was demonstrated in literature to lead to beneficial effects in terms of reducing fatiguing and the occurrence of musculo-skeletal diseases. However, a large number of manufacturing operations would not benefit from power increases because it rather requires the modulation of the operator stiffness. However, in literature, considerably less attention was given to those robotic devices that regulate their stiffness based on the operator stiffness, even if their introduction in the line would aid the operator during different manipulations respect with the exoskeletons with variable power. In this thesis the description of the command logic of an exoskeleton for manufacturing applications, whose stiffness is modulated based on the operator stiffness, is described. Since the operator stiffness cannot be mechanically measured without deflecting the limb, an estimation based on the superficial Electromyographic signal is required. A model composed of 1 joint and 2 antagonist muscles was developed to approximate the elbow and the wrist joints. Each muscle was approximated as the Hill model and the analysis of the joint stiffness, at different joint angle and muscle activations, was performed. The same Hill muscle model was then implemented in a 2 joint and 6 muscles (2J6M) model which approximated the elbow-shoulder system. Since the estimation of the exerted stiffness with a 2J6M model would be quite onerous in terms of processing time, the estimation of the operator end-point stiffness in realtime would therefore be questionable. Then, a linear relation between the end-point stiffness and the component of muscle activation that does not generate any end-point force, is proposed. Once the stiffness the operator exerts was estimated, three command logics that identifies the stiffness the exoskeleton is required to exert are proposed. These proposed command logics are: Proportional, Integral 1 s, and Integral 2 s. The stiffening exerted by a device in which a Proportional logic is implemented is proportional, sample by sample, to the estimated stiffness exerted by the operator. The stiffening exerted by the exoskeleton in which an Integral logic is implemented is proportional to the stiffness exerted by the operator, averaged along the previous 1 second (Integral 1 s) or 2 seconds (Integral 2 s). The most effective command logic, among the proposed ones, was identified with empirical tests conducted on subjects using a wrist haptic device (the Hi5, developed by the Bioengineering group of the Imperial College of London). The experimental protocol consisted in a wrist flexion/extension tracking task with an external perturbation, alternated with isometric force exertion for the estimation of the occurrence of the fatigue. The fatigue perceived by the subject, the tracking error, defined as the RMS of the difference between wrist and target angles, and the energy consumption, defined as the sum of the squared signals recorded from two antagonist muscles, indicated the Integral 1 s logic to be the most effective for controlling the exoskeleton. A logistic relation between the stiffness exerted by the subject and the stiffness exerted by the robotic devices was selected, because it assured a smooth transition between the maximum and the minimum stiffness the device is required to exert. However, the logistic relation parameters are subject-specific, therefore an experimental estimation is required. An example was provided. Finally, the literature about variable stiffness actuators was analyzed to identify the most suitable device for exoskeleton stiffness modulation. This actuator is intended to be integrated on an existing exoskeleton that already enhances the operator power based on the operator Electromyographic signal. The identified variable stiffness actuator is the DLR FSJ, which controls its stiffness modulating the preload of a single spring