Human to robot hand motion mapping methods: review and classification

In this article, the variety of approaches proposed in literature to address the problem of mapping human to robot hand motions are summarized and discussed. We particularly attempt to organize under macro-categories the great quantity of presented methods, that are often difficult to be seen from a general point of view due to different fields of application, specific use of algorithms, terminology and declared goals of the mappings. Firstly, a brief historical overview is reported, in order to provide a look on the emergence of the human to robot hand mapping problem as a both conceptual and analytical challenge that is still open nowadays. Thereafter, the survey mainly focuses on a classification of modern mapping methods under six categories: direct joint, direct Cartesian, taskoriented, dimensionality reduction based, pose recognition based and hybrid mappings. For each of these categories, the general view that associates the related reported studies is provided, and representative references are highlighted. Finally, a concluding discussion along with the authors’ point of view regarding future desirable trends are reported.This work was supported in part by the European Commission’s Horizon 2020 Framework Programme with the project REMODEL under Grant 870133 and in part by the Spanish Government under Grant PID2020-114819GB-I00.Peer ReviewedPostprint (published version

Biomimetic Manipulator Control Design for Bimanual Tasks in the Natural Environment

As robots become more prolific in the human environment, it is important that safe operational procedures are introduced at the same time; typical robot control methods are often very stiff to maintain good positional tracking, but this makes contact (purposeful or accidental) with the robot dangerous. In addition, if robots are to work cooperatively with humans, natural interaction between agents will make tasks easier to perform with less effort and learning time. Stability of the robot is particularly important in this situation, especially as outside forces are likely to affect the manipulator when in a close working environment; for example, a user leaning on the arm, or task-related disturbance at the end-effector. Recent research has discovered the mechanisms of how humans adapt the applied force and impedance during tasks. Studies have been performed to apply this adaptation to robots, with promising results showing an improvement in tracking and effort reduction over other adaptive methods. The basic algorithm is straightforward to implement, and allows the robot to be compliant most of the time and only stiff when required by the task. This allows the robot to work in an environment close to humans, but also suggests that it could create a natural work interaction with a human. In addition, no force sensor is needed, which means the algorithm can be implemented on almost any robot. This work develops a stable control method for bimanual robot tasks, which could also be applied to robot-human interactive tasks. A dynamic model of the Baxter robot is created and verified, which is then used for controller simulations. The biomimetic control algorithm forms the basis of the controller, which is developed into a hybrid control system to improve both task-space and joint-space control when the manipulator is disturbed in the natural environment. Fuzzy systems are implemented to remove the need for repetitive and time consuming parameter tuning, and also allows the controller to actively improve performance during the task. Experimental simulations are performed, and demonstrate how the hybrid task/joint-space controller performs better than either of the component parts under the same conditions. The fuzzy tuning method is then applied to the hybrid controller, which is shown to slightly improve performance as well as automating the gain tuning process. In summary, a novel biomimetic hybrid controller is presented, with a fuzzy mechanism to avoid the gain tuning process, finalised with a demonstration of task-suitability in a bimanual-type situation.EPSR

Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

Advances in Robot Kinematics : Proceedings of the 15th international conference on Advances in Robot Kinematics

International audienceThe motion of mechanisms, kinematics, is one of the most fundamental aspect of robot design, analysis and control but is also relevant to other scientific domains such as biome- chanics, molecular biology, . . . . The series of books on Advances in Robot Kinematics (ARK) report the latest achievement in this field. ARK has a long history as the first book was published in 1991 and since then new issues have been published every 2 years. Each book is the follow-up of a single-track symposium in which the participants exchange their results and opinions in a meeting that bring together the best of world’s researchers and scientists together with young students. Since 1992 the ARK symposia have come under the patronage of the International Federation for the Promotion of Machine Science-IFToMM.This book is the 13th in the series and is the result of peer-review process intended to select the newest and most original achievements in this field. For the first time the articles of this symposium will be published in a green open-access archive to favor free dissemination of the results. However the book will also be o↵ered as a on-demand printed book.The papers proposed in this book show that robot kinematics is an exciting domain with an immense number of research challenges that go well beyond the field of robotics.The last symposium related with this book was organized by the French National Re- search Institute in Computer Science and Control Theory (INRIA) in Grasse, France

Progettazione e Controllo di Mani Robotiche

The application of dexterous robotic hands out of research laboratories has been limited by the intrinsic complexity that these devices present. This is directly reflected as an economically unreasonable cost and a low overall reliability. Within the research reported in this thesis it is shown how the problem of complexity in the design of robotic hands can be tackled, taking advantage of modern technologies (i.e. rapid prototyping), leading to innovative concepts for the design of the mechanical structure, the actuation and sensory systems. The solutions adopted drastically reduce the prototyping and production costs and increase the reliability, reducing the number of parts required and averaging their single reliability factors. In order to get guidelines for the design process, the problem of robotic grasp and manipulation by a dual arm/hand system has been reviewed. In this way, the requirements that should be fulfilled at hardware level to guarantee successful execution of the task has been highlighted. The contribution of this research from the manipulation planning side focuses on the redundancy resolution that arise in the execution of the task in a dexterous arm/hand system. In literature the problem of coordination of arm and hand during manipulation of an object has been widely analyzed in theory but often experimentally demonstrated in simplified robotic setup. Our aim is to cover the lack in the study of this topic and experimentally evaluate it in a complex system as a anthropomorphic arm hand system

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Hybrid optical and magnetic manipulation of microrobots

Microrobotic systems have the potential to provide precise manipulation on cellular level for diagnostics, drug delivery and surgical interventions. These systems vary from tethered to untethered microrobots with sizes below a micrometer to a few microns. However, their main disadvantage is that they do not have the same capabilities in terms of degrees-of-freedom, sensing and control as macroscale robotic systems. In particular, their lack of on-board sensing for pose or force feedback, their control methods and interface for automated or manual user control are limited as well as their geometry has few degrees-of-freedom making three-dimensional manipulation more challenging. This PhD project is on the development of a micromanipulation framework that can be used for single cell analysis using the Optical Tweezers as well as a combination of optical trapping and magnetic actuation for recon gurable microassembly. The focus is on untethered microrobots with sizes up to a few tens of microns that can be used in enclosed environments for ex vivo and in vitro medical applications. The work presented investigates the following aspects of microrobots for single cell analysis: i) The microfabrication procedure and design considerations that are taken into account in order to fabricate components for three-dimensional micromanipulation and microassembly, ii) vision-based methods to provide 6-degree-offreedom position and orientation feedback which is essential for closed-loop control, iii) manual and shared control manipulation methodologies that take into account the user input for multiple microrobot or three-dimensional microstructure manipulation and iv) a methodology for recon gurable microassembly combining the Optical Tweezers with magnetic actuation into a hybrid method of actuation for microassembly.Open Acces

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Anthropomorphic surgical system for soft tissue robot-assisted surgery

Over the past century, abdominal surgery has seen a rapid transition from open procedures to less invasive methods such as laparoscopy and robot-assisted minimally invasive surgery (R-A MIS). These procedures have significantly decreased blood loss, postoperative morbidity and length of hospital stay in comparison with open surgery. R-A MIS has offered refined accuracy and more ergonomic instruments for surgeons, further minimising trauma to the patient.This thesis aims to investigate, design and prototype a novel system for R-A MIS that will provide more natural and intuitive manipulation of soft tissues and, at the same time, increase the surgeon's dexterity. The thesis reviews related work on surgical systems and discusses the requirements for designing surgical instrumentation. From the background research conducted in this thesis, it is clear that training surgeons in MIS procedures is becoming increasingly long and arduous. Furthermore, most available systems adopt a design similar to conventional laparoscopic instruments or focus on different techniques with debatable benefits. The system proposed in this thesis not only aims to reduce the training time for surgeons but also to improve the ergonomics of the procedure.In order to achieve this, a survey was conducted among surgeons, regarding their opinions on surgical training, surgical systems, how satisfied they are with them and how easy they are to use. A concept for MIS robotic instrumentation was then developed and a series of focus group meetings with surgeons were run to discuss it. The proposed system, named microAngelo, is an anthropomorphic master-slave system that comprises a three-digit miniature hand that can be controlled using the master, a three-digit sensory exoskeleton. While multi-fingered robotic hands have been developed for decades, none have been used for surgical operations. As the system has a human centred design, its relation to the human hand is discussed. Prototypes of both the master and the slave have been developed and their design and mechanisms is demonstrated. The accuracy and repeatability of the master as well as the accuracy and force capabilities of the slave are tested and discussed

Vers des environnements virtuels plus écologiques : étude des modifications du comportement moteur en réalité virtuelle lors de l'ajout d'informations haptiques par un mécanisme parallèle entraîné par câbles

Introduction : Les nouvelles technologies qui permettent de capter et d’analyser les mouvements des utilisateurs ne cessent de se développer et représentent un potentiel intéressant dans le domaine de la santé. Grâce à l’essor de ces nouvelles technologies, des systèmes de réalité virtuelle (RV) clefs en main intègrent les services de réadaptation, et les études démontrent leur capacité à optimiser la rééducation motrice et l’évaluation des clients présentant des troubles du contrôle moteur. Le marché de la RV est ainsi en pleine expansion, et l’ajout d’informations haptiques permettant de modéliser les caractéristiques physiques des entités virtuelles représente un intérêt considérable pour améliorer l’écologie des environnements virtuels (EVs) et le transfert des apprentissages aux activités quotidiennes. Toutefois, l’effet de l’ajout de ces informations sur le comportement moteur des sujets demeure très peu connu. L’objectif principal de cette thèse était ainsi d’évaluer l’impact de l’ajout d’informations haptiques, par un mécanisme parallèle entrainé par câbles (robot à câbles), sur le contrôle moteur de sujets sains, lors de la réalisation de tâches complexes et fonctionnelles dans des EVs. Les deux hypothèses principales étaient que cet ajout améliore le contrôle du mouvement lors de tâche de manutention d’objet ayant des contraintes environnementales statiques, et modifie les stratégies locomotrices proactives en présence de contraintes dynamiques. Méthode : Le comportement moteur de participants sains a été analysé lors de la réalisation de deux tâches. En premier lieu, une tâche de manutention de caisse nécessitant la préhension et le déplacement d’une caisse à partir d’une posture debout a été étudiée. Celle-ci a été réalisée dans un environnement réel et dans des EVs, en absence et en présence d’informations haptiques, relatives aux contraintes physiques de l’étagère et de la caisse manipulée, fournies grâce à un robot à câbles (Chapitre 3, N=12). En second lieu, une tâche nécessitant l’évitement d’avatars au cours de la marche sur un tapis roulant a été réalisée en présence et en absence de risque de contact physique avec les avatars, délivré par un robot à câbles (Chapitre 4, N=10). Les EVs étaient vus au travers d’un visiocasque. Résultats : La première étude a démontré une amélioration des paramètres spatiaux du mouvement réalisé dans l’EV en présente d’informations haptiques, au cours des différentes phases de la tâche de manutention (préhension, montée et descente de la caisse). L’organisation spatiale du mouvement était ainsi plus similaire à ce qui était observé dans un environnement réel, avec un meilleur respect des contraintes environnementales (éloignement plus important de la caisse avec l’étagère, trajectoire plus longue). De plus, le contrôle du mouvement était influencé par la demande de précision requise pour ne pas toucher les étagères en présence d’informations haptiques uniquement. La deuxième étude a démontré la mise en place de stratégies motrices plus précautionneuses pour éviter les avatars lors de l’ajout d’informations haptiques. Les participants tendaient à anticiper plus précocement l’évitement des avatars. Ils maintenaient une distance minimale plus importante avec les avatars et conservaient un espace péripersonnel plus large, indépendamment de l’angle d’approche de l’avatar. Conclusion : L’ajout d’informations haptiques dans les EVs impacte les stratégies motrices proactives des participants sains aussi bien lors de la tâche de manutention de caisse que de locomotion avec évitement d’avatars. Les résultats suggèrent que l’ajout d’informations haptiques favorise la prise en compte des entités virtuelles lors de la planification mouvement. Ces informations haptiques imposent en effet des restrictions plus réalistes dans les possibilités d’actions fournies par les EVs, et modifient probablement l’évaluation des conséquences que représente le contact avec les entités virtuelles. Il serait pertinent de poursuivre l’étude de l’influence de ces informations afin de proposer à des clients ayant des déficiences motrices des environnements encore plus écologiques, qui favorisent l’évaluation et la prise en compte des risques implicites que représentent les entités environnementales.Introduction: New technologies that capture and analyze user movement are constantly developing and represent a great potential in healthcare. Thanks to the recent technological advances, turnkey virtual reality (VR) systems are progressively integrated into the rehabilitation setting, and studies have demonstrated their ability to optimise sensorimotor rehabilitation and clinical assessment of people with motor control disorders. The market for VR is growing and adding haptic feedback that provides physical characteristics to virtual entities represents a great potential to improve the ecological validity of virtual environments (VE) and to the transfer of learning to daily tasks. However, the impact that adding haptic feedback has on motor behavior remains poorly understood. The main objective of this thesis was to assess the impact of adding haptic feedback, using a novel cable-driven parallel robot, on the motor control of healthy participants during complex, functional tasks in VEs. The two mains hypotheses were that haptic feedback improves motor control during a handling task with static environmental constraints and modifies proactive locomotor strategies in the presence of dynamic constraints. Method: The motor behavior of healthy participants was analysed during two tasks. First, a manual handling task was studied during which participants grasped and moved a crate while standing. This task was realised in a real environment and in VEs with the absence and the presence of haptic information. The latter simulated the physical constraints of the shelf and the crate to be manipulated using a cable-driven robot (Chapter 3, N=12). Second, avatar avoidance tasks were realised when participants walked on a self-paced treadmill in the absence and then in the presence of a risk of physical contact with avatars. Contact was simulated by a cable-driven robot (Chapter 4, N=10). VEs were viewed through a head mounted display for all tasks. Results: The first study showed that adding haptic feedback to the VE improved spatial parameters of movement realised in a VE during all phases of movement (reaching, ascent and descent phases). The spatial organisation of movement was closer to those observed in a physical environment, and better respected environmental constraints (higher clearances from the shelf and longer trajectories). Moreover, movement control was influenced by task precision required to avoid any contact with the shelf in the presence of haptic feedback only. The second study demonstrated that when avoiding avatars in VR, more cautious behavior was measured in the presence of potential physical contact. Participants tended to start their avoidance strategy earlier and increased minimum clearance along with a larger personal space regardless of the avatar’s approach angle. Conclusion: Adding haptic feedback in VEs impacts the proactive motor strategies of healthy participants during a manual handling task as well as a locomotor task involving the avoidance of avatars. These results suggest that adding haptic feedback enhances one’s consideration of virtual entities during movement planning. Haptic information imposes more realistic restrictions on the actions afforded by EVs, and likely modifies the perceived consequences of potential contact with virtual entities. It will be important to continue to study the impact of haptic feedback within VEs to provide even more ecological environments to people with motor deficits in order to improve assessment and the consideration of implicit risks posed by the environment