Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

Multivariable Loop-Shaping in Bilateral Telemanipulation

Abstract This paper presents an architecture and control methodology for obtaining transparency and stability robustness in a multivariable bilateral teleoperator system. The work presented here extends a previously published single-input, single-output approach to accommodate multivariable systems. The extension entails the use of impedance control techniques, which are introduced to render linear the otherwise nonlinear dynamics of the master and slave manipulators, in addition to a diagonalization multivariable loop shaping technique, used to render tractable the multivariable compensator design. A multivariable measure of transparency is proposed based on the relative singular values of the environment and transmitted impedance matrices. The approach is experimentally demonstrated on a three degree-of-freedom scaled telemanipulator pair with a highly coupled environment. Using direct measurement of the power delivered to the operator to assess the system's stability robustness, along with the proposed measure of multivariable transparency, the loop-shaping compensation is shown to improve the stability robustness by a factor of two and the transparency by more than a factor of five. Fite and Goldfarb Multivariable Loop Shaping … 2 Introduction Bilateral teleoperation systems provide for human interaction with an environment while alleviating the necessity of direct contact between the two. Using a pair of robot manipulators, such a system enables dexterous human manipulation in remote, hazardous, or otherwise inaccessible environments. Bilateral telemanipulators can additionally incorporate power attenuation or amplification between the human operator and environment, allowing for human manipulation of microscopic objects (in the case of macro-micro bilateral telemanipulation) or large-scale objects (in the case of man-amplifiers). The teleoperative performance can be characterized by the transparency, which is a measure of the extent to which the telemanipulation system presents the undistorted dynamics of the environment to the human operator. A common goal in the control of bilateral telemanipulation is to provide transparent teleoperation while ensuring the robust stability of the human-telemanipulator-environment loop. Prior Work Several researchers have investigated aspects of transparency and stability in telemanipulation, primarily through the use of two-port network modeling techniques. Doyle [7], to assess the stability of a macro-micro bilateral telemanipulator interacting with a passive human operator and environment. Though the telemanipulator itself was a singledegree-of-freedom system, the human-teleoperator-environment interaction was formulated in a manner that required multivariable tools in order to assess stability robustness. Colgate did not explicitly treat transparency, but instead utilized impedance shaping to intentionally alter the dynamics as perceived by the human operator through the telemanipulator. Itoh et al. experimentally implemented a six degree-of-freedom telemanipulator using passivity theory to address stability robustness, but instead of providing transparency, the telemanipulator was controlled to exhibit a task-oriented dynamic behavior specified in order to facilitate a particular telemanipulation task. Hashstrudi-Zaad and Salcudean theoretically assessed the performance and stability robustness of a three degree-of-freedom telemanipulator by incorporating a parallel force/position control to linearize and decouple the manipulators, and by assuming the human operator and environment to be decoupled, in which case the analysis reduces to that required for three decoupled single-degree-of-freedom systems. In contrast to the combined hybrid parameter/passivity based approach, the architecture proposed by Fite et al. [8] formulates the teleoperation system as a single feedback loop to which the tools of classical control theory can then be applied to address the performance and stability robustness. In so doing, the stability robustness of the system is addressed in a non-conservative manner, and the transparency is addressed only in the bandwidth of interest. This loop shaping approach was developed in a single input, single output context; since telemanipulation Fite and Goldfarb Multivariable Loop Shaping … 4 applications generally involve systems with coupled multiple degrees of freedom, however, such a method is of limited utility without extension to the multivariable case. As such, the work presented in this paper extends this previously published approach to the multivariable case of telemanipulation. Specifically, the extension entails the use of impedance control techniques to render linear the otherwise nonlinear dynamics of the master and slave manipulators, and employs a diagonalization multivariable loop shaping technique used to render tractable the multivariable loop shaping compensator design. A multivariable measure of transparency is additionally proposed based on the relative singular values of the environment and transmitted impedance matrices. 3 Multivariable Telemanipulation Architecture Fite and Goldfarb Multivariable Loop Shaping … 6 Given the master/human and slave/environment dynamics as previously described, the loop shaping telemanipulation architecture is obtained by combining the master/human and slave/environment subsystems with the position and force scaling matrices, C 1 and C 2 , respectively, as shown in The transparency of the multivariable teleoperation loop is determined by the relative distortion between the transmitted impedance (i.e., the impedance felt by the human operator) and the actual environment impedance. The impedance transmitted to the human operator by the telemanipulation system is given by: For perfect transparency, the transmitted impedance transfer function matrix of Eq. (3) should equal the actual environment impedance, Z e . In practice, these matrices need only be similar within some frequency band of interest. Thus, within this band of interest, perfect transparency requires the singular values of the transmitted impedance transfer function matrix to equal those of the actual environment impedance transfer function matrix. As such, a measure of the desired multivariable performance can be given by the ratio of the respective singular values of the impedance transmitted to the human operator to those of the environment impedance: where n rank = ) ( e t Z , Z and i δ represents distortion in the teleoperative system. A desired bandwidth of transparency can be prescribed by ensuring that the distortion i δ in each singular Fite and Goldfarb Multivariable Loop Shaping … 7 value is less than some allowable amount of distortion ∆ for a desired bandwidth of operation. For ∆= 3 dB, a prescription for good teleoperative performance can be written as: where t Ω is a desired bandwidth of teleoperative transparency. The overall objective of the control architecture is to achieve the desired performance specified by Eq. (5) while ensuring the robust stability of the closed-loop system. With the introduction of a loop shaping compensator in the motion communication channel, th

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

Position referenced force augmentation in teleoperated hydraulic manipulators operating under delayed and lossy networks: a pilot study.

Position error between motions of the master and slave end-effectors is inevitable as it originates from hard-to-avoid imperfections in controller design and model uncertainty. Moreover, when a slave manipulator is controlled through a delayed and lossy communication channel, the error between the desired motion originating from the master device and the actual movement of the slave manipulator end-effector is further exacerbated. This paper introduces a force feedback scheme to alleviate this problem by simply guiding the operator to slow down the haptic device motion and, in turn, allows the slave manipulator to follow the desired trajectory closely. Using this scheme, the master haptic device generates a force, which is proportional to the position error at the slave end-effector, and opposite to the operator's intended motion at the master site. Indeed, this force is a signal or cue to the operator for reducing the hand speed when position error, due to delayed and lossy network, appears at the slave site. Effectiveness of the proposed scheme is validated by performing experiments on a hydraulic telemanipulator setup developed for performing live-line maintenance. Experiments are conducted when the system operates under both dedicated and wireless networks. Results show that the scheme performs well in reducing the position error between the haptic device and the slave end-effector. Specifically, by utilizing the proposed force, the mean position error, for the case presented here, reduces by at least 92% as compared to the condition without the proposed force augmentation scheme. The scheme is easy to implement, as the only required on-line measurement is the angular displacement of the slave manipulator joints

Human-robot interaction for telemanipulation by small unmanned aerial systems

This dissertation investigated the human-robot interaction (HRI) for the Mission Specialist role in a telemanipulating unmanned aerial system (UAS). The emergence of commercial unmanned aerial vehicle (UAV) platforms transformed the civil and environmental engineering industries through applications such as surveying, remote infrastructure inspection, and construction monitoring, which normally use UAVs for visual inspection only. Recent developments, however, suggest that performing physical interactions in dynamic environments will be important tasks for future UAS, particularly in applications such as environmental sampling and infrastructure testing. In all domains, the availability of a Mission Specialist to monitor the interaction and intervene when necessary is essential for successful deployments. Additionally, manual operation is the default mode for safety reasons; therefore, understanding Mission Specialist HRI is important for all small telemanipulating UAS in civil engineering, regardless of system autonomy and application. A 5 subject exploratory study and a 36 subject experimental study were conducted to evaluate variations of a dedicated, mobile Mission Specialist interface for aerial telemanipulation from a small UAV. The Shared Roles Model was used to model the UAS human-robot team, and the Mission Specialist and Pilot roles were informed by the current state of practice for manipulating UAVs. Three interface camera view designs were tested using a within-subjects design, which included an egocentric view (perspective from the manipulator), exocentric view (perspective from the UAV), and mixed egocentric-exocentric view. The experimental trials required Mission Specialist participants to complete a series of tasks with physical, visual, and verbal requirements. Results from these studies found that subjects who preferred the exocentric condition performed tasks 50% faster when using their preferred interface; however, interface preferences did not affect performance for participants who preferred the mixed condition. This result led to a second finding that participants who preferred the exocentric condition were distracted by the egocentric view during the mixed condition, likely caused by cognitive tunneling, and the data suggest tradeoffs between performance improvements and attentional costs when adding information in the form of multiple views to the Mission Specialist interface. Additionally, based on this empirical evaluation of multiple camera views, the exocentric view was recommended for use in a dedicated Mission Specialist telemanipulation interface. Contributions of this thesis include: i) conducting the first focused HRI study of aerial telemanipulation, ii) development of an evaluative model for telemanipulation performance, iii) creation of new recommendations for aerial telemanipulation interfacing, and iv) contribution of code, hardware designs, and system architectures to the open-source UAV community. The evaluative model provides a detailed framework, a complement to the abstraction of the Shared Roles Model, that can be used to measure the effects of changes in the system, environment, operators, and interfacing factors on performance. The practical contributions of this work will expedite the use of manipulating UAV technologies by scientists, researchers, and stakeholders, particularly those in civil engineering, who will directly benefit from improved manipulating UAV performance

Bilateral Control - Operational enhancements

A succinct definition of the word bilateral is having two sides [1]. In robotics the term bilateral control is used to define the specific interaction of two systems by means of position and/or force. Bilateral systems are composed of two sides named master and slave side. The aim of such an arrangement is such that position command dictated by master side is followed by a slave side, and at the same time the force sensation of the remote environment experienced by slave is transferred to the mater - human operator. This way bilateral system may be perceived as an “impendanceless” extension of the human operator providing the touch information of the remote (or inaccessible) environment. In a sense bilateral systems are a mechatronics extension of the teleoperated systems. There are many applications of this structure which requires critical manipulations like nuclear material handling, robotic surgery, and micro material handling and assembly. In all these applications a human operator is required to have as close to real as possible contact with object that should be manipulated or in other word the telepresence of the operator is required. In this thesis work various important aspects of bilateral control systems are discussed. These aspects include problems of (i) acquisition of information on master and slave side, (ii) analysis and selection of the proper structure of the control systems to ensure fidelity of the system behavior. The work has been done to enhance the performance of the bilateral control system by: (i) Enhancing position and velocity measurements obtained from incremental encoder having limited number of pulses per revolution. A few algorithms are investigated and their improvements are proposed; (ii) Increasing system robustness by using acceleration controller based on disturbance observer. The robust system design based on disturbance observer is known but its application requires very fast sampling and high bandwidth of the observer. In this work the discrete time realization of the observer is presented in details and selection of the necessary filters and the sampling so to achieve a good trade-off for observer realization is discussed and experimentally confirmed; (iii) Increasing the bandwidth of force sensation by using reaction force observer. For transparent operation of a bilateral system the bandwidth of force sensation is of the major interest. All force sensors do have relatively slow dynamics and observer based structures seems providing better behavior of the overall system. In this work the observer of the interaction force is examined and design procedure is established. In order to verify all of the proposed ideas a versatile bilateral system is designed and built and experimental verification is carried out on this system

Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

On the Value of Estimating Human Arm Stiffness during Virtual Teleoperation with Robotic Manipulators

Teleoperated robotic systems are widely spreading in multiple different fields, from hazardous environments exploration to surgery. In teleoperation, users directly manipulate a master device to achieve task execution at the slave robot side; this interaction is fundamental to guarantee both system stability and task execution performance. In this work, we propose a non-disruptive method to study the arm endpoint stiffness. We evaluate how users exploit the kinetic redundancy of the arm to achieve stability and precision during the execution of different tasks with different master devices. Four users were asked to perform two planar trajectories following virtual tasks using both a serial and a parallel link master device. Users' arm kinematics and muscular activation were acquired and combined with a user-specific musculoskeletal model to estimate the joint stiffness. Using the arm kinematic Jacobian, the arm end-point stiffness was derived. The proposed non-disruptive method is capable of estimating the arm endpoint stiffness during the execution of virtual teleoperated tasks. The obtained results are in accordance with the existing literature in human motor control and show, throughout the tested trajectory, a modulation of the arm endpoint stiffness that is affected by task characteristics and hand speed and acceleration