Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

Neural network enhanced robot tool identification and calibration for bilateral teleoperation

© 2013 IEEE. In teleoperated surgery, the transmission of force feedback from the remote environment to the surgeon at the local site requires the availability of reliable force information in the system. In general, a force sensor is mounted between the slave end-effector and the tool for measuring the interaction forces generated at the remote sites. Such as the acquired force value includes not only the interaction force but also the tool gravity. This paper presents a neural network (NN) enhanced robot tool identification and calibration for bilateral teleoperation. The goal of this experimental study is to implement and validate two different techniques for tool gravity identification using Curve Fitting (CF) and Artificial Neural Networks (ANNs), separately. After tool identification, calibration of multi-axis force sensor based on Singular Value Decomposition (SVD) approach is introduced for alignment of the forces acquired from the force sensor and acquired from the robot. Finally, a bilateral teleoperation experiment is demonstrated using a serial robot (LWR4+, KUKA, Germany) and a haptic manipulator (SIGMA 7, Force Dimension, Switzerland). Results demonstrated that the calibration of the force sensor after identifying tool gravity component by using ANN shows promising performance than using CF. Additionally, the transparency of the system was demonstrated using the force and position tracking between the master and slave manipulators

Improved human-robot collaborative control of redundant robot for teleoperated minimally invasive surgery

© 2016 IEEE. An improved human-robot collaborative control scheme is proposed in a teleoperated minimally invasive surgery scenario, based on a hierarchical operational space formulation of a seven-degree-of-freedom redundant robot. Redundancy is exploited to guarantee a remote center of motion (RCM) constraint and to provide a compliant behavior for the medical staff. Based on the implemented hierarchical control framework, an RCM constraint and a safe constraint are applied to the null-space motion to achieve the surgical tasks with human-robot interaction. Due to the physical interactions, safety and accuracy of the surgery may be affected. The control framework integrates an adaptive compensator to enhance the accuracy of the surgical tip and to maintain the RCM constraint in a decoupled way avoiding any physical interactions. The system performance is verified on a patient phantom. Compared with the methods proposed in the literature, results show that the accuracy of both the RCM constraint and the surgical tip is improved. The compliant swivel motion of the robot arm is also constrained in a defined area, and the interaction force on the abdominal wall becomes smaller

Manipulability in trajectory tracking for constrained redundant manipulators via sequential quadratic programming

Trajectory tracking methods for constrained redundant manipulators are presented in this thesis, where the end-effector of a redundant serial manipulator has to track a desired trajectory while some points on its kinematic chain satisfy one or more constraints. In addition, two manipulability indexes are taken into account in order to optimize the trajectory. The first index is defined in terms of the geometric Jacobian of the manipulator in the constrained configuration. The second index is based on the constrained Jacobian, which maps velocities from joint space to task space, taking into account the holonomic constraints. Three methods for solving the trajectory tracking problem are discussed. The first two, kinematic control (KC) and quadratic programming (QP), are widely discussed in literature. The third, sequential quadratic programming (SQP), is a new approach, unlike KC or QP, has as advantages (despite some shortcomings) not explicitly depend on pseudoinverse Jacobian, derivative from the desired trajectory and linearization of indexes or constraints. A discussion of these three methods is presented in terms of tracking error, constraint violation, singularity distance, among others through experiments performed on a Baxter collaborative robot.Métodos de rastreamento de trajetória para manipuladores redundantes restritos são apresentados nesta tese, onde o efetuador de um manipulador serial redundante tem que rastrear uma trajetória desejada enquanto alguns pontos em sua cadeia cinemática satisfazem uma ou mais restrições. Além disso, dois índices de manipulabilidade são levados em consideração a fim de otimizar a trajetória para evitar singularidades. O primeiro índice é definido em função do jacobiano geométrico do manipulador na configuração restrita. O segundo índice é baseado no Jacobiano restrito, o qual mapeia velocidades no espaço das juntas para a espaço da tarefa, levando em conta as restrições holonômicas. Três métodos para resolver o problema de rastreamento de trajetória são discutidos. Os dois primeiros, controle cinemático e programação quadrática (QP), são amplamente discutidos na literatura. O terceiro, programação quadrática sequencial (SQP), é uma nova abordagem, diferentemente do controle cinemático ou QP, tem como vantagens (apesar de algumas deficiências) não depender explicitamente da pseudo-inversa de jacobianos, derivadas da trajetória desejada e linearização de índices ou restrições. Uma discussão desses três métodos é apresentada em termos de erro de rastreamento, violação da restrição, distância de singularidades, entre outros através de experimentos realizados em um robô colaborativo Baxter

Design and Development of a Surgical Robot for Needle-Based Medical Interventions

Lung cancer is the leading cause of cancer related deaths. If diagnosed in a timely manner, the treatment of choice is surgical resection of the cancerous lesions followed by radiotherapy. However, surgical resection may be too invasive for some patients due to old age or weakness. An alternative is minimally invasive needle-based interventions for cancer diagnosis and treatment. This project describes the design, analysis, development and experimental evaluation of a modular, compact, patient-mounted robotic manipulator for lung cancer diagnosis and treatment. In this regard, a novel parallel Remote Centre of Motion (RCM) mechanism is proposed for minimally invasive delivery of needle-based interventions. The proposed robot provides four degrees of freedom (DOFs) to orient and move a surgical needle within a spherical coordinate system. There is an analytical solution for the kinematics of the proposed parallel mechanism and the end-effectors motion is well-conditioned within the required workspace. The RCM is located beneath the skin surface to minimize the invasiveness of the surgical procedure while providing the required workspace to target the cancerous lesions. In addition, the proposed robot benefits from a design capable of measuring the interaction forces between the needle and the tissue. The experimental evaluation of the robot has proved its capability to accurately orient and move a surgical needle within the required workspace. Although this robotic system has been designed for the treatment of lung cancer, it is capable of performing other procedures in the thoracic or abdominal cavity such as liver cancer diagnosis and treatment

Position planning for collaborating robots and its application in neurosurgery

Primjena robotskih manipulatora u medicini danas je vrlo aktualno područje istraživanja. Unatoč tome još uvijek postoji velik broj problema koji se javljaju kod pripreme većine robotiziranih operacijskih postupaka. Jedan od glavnih je pozicioniranje robota u odnosu na pacijenta. Kod postavljanja robota u odnosu na unaprijed poznate radne točke potrebno je osigurati efikasnu poziciju robota iz koje se sve zadane kretnje mogu izvršiti bez kinematskih problema i kolizija. U radu je predstavljena metoda za planiranje prostornog razmještaja robota prikladna za primjenu u neurokirurgiji. Razvijena metoda počiva na višeciljnoj optimizaciji funkcije cilja koja je sastavljena od kriterija koji objedinjuju prostornu upravljivost robota sa izbjegavanjem kolizija. Primjena razvijene metode validirana je na dvoručnom sustavu robota.Applications of robot manipulators in surgery are nowadays a very actual field of research. Still, there are a number of technical problems when setting and preparing robotical systems for various operation procedures. One of them is the robot-patient placement. When placing robots in respect to known target working positions it is crucial to assure feasible positioning where all required motions can be executed with no kinematic or collision problems. A planning method for robot placement suitable for neurosurgical operations is presented in this paper. The planning method is based on a multi-objective cost function which is composed of criteria that balance dexterity properties with a novel collision avoiding parameter. Use of the planning approach is implemented and validated on a dual arm robot setup

Shared control for natural motion and safety in hands-on robotic surgery

Hands-on robotic surgery is where the surgeon controls the tool's motion by applying forces and torques to the robot holding the tool, allowing the robot-environment interaction to be felt though the tool itself. To further improve results, shared control strategies are used to combine the strengths of the surgeon with those of the robot. One such strategy is active constraints, which prevent motion into regions deemed unsafe or unnecessary. While research in active constraints on rigid anatomy has been well-established, limited work on dynamic active constraints (DACs) for deformable soft tissue has been performed, particularly on strategies which handle multiple sensing modalities. In addition, attaching the tool to the robot imposes the end effector dynamics onto the surgeon, reducing dexterity and increasing fatigue. Current control policies on these systems only compensate for gravity, ignoring other dynamic effects. This thesis presents several research contributions to shared control in hands-on robotic surgery, which create a more natural motion for the surgeon and expand the usage of DACs to point clouds. A novel null-space based optimization technique has been developed which minimizes the end effector friction, mass, and inertia of redundant robots, creating a more natural motion, one which is closer to the feeling of the tool unattached to the robot. By operating in the null-space, the surgeon is left in full control of the procedure. A novel DACs approach has also been developed, which operates on point clouds. This allows its application to various sensing technologies, such as 3D cameras or CT scans and, therefore, various surgeries. Experimental validation in point-to-point motion trials and a virtual reality ultrasound scenario demonstrate a reduction in work when maneuvering the tool and improvements in accuracy and speed when performing virtual ultrasound scans. Overall, the results suggest that these techniques could increase the ease of use for the surgeon and improve patient safety.Open Acces

Anthropomorphic Dual-Arm Coordinated Control for a Single-Port Surgical Robot Based on Dual-Step Optimization

Effective teleoperation of the small-scale and highly-integrated robots for single-port surgery (SPS) imposes unique control and human-robot interaction challenges. Traditional isometric teleoperation schemes mainly focus on end-to-end trajectory mapping, which is problematic when applied to SPS robotic control, especially for dual-arm coordinated operation. Inspired by the human arm configuration in boxing maneuvers, an optimized anthropomorphic coordinated control strategy based on a dual-step optimization approach is proposed. Theoretical derivation and solvability of the problem are addressed, and the effectiveness of the method is further demonstrated in detailed simulation and in-vitro experiments. The proposed control strategy has been shown to perform dexterous SPS bimanual manipulation more effectively, involving less instrument-interference and is free from singularities, thereby improving the safety and efficiency of SPS operations

Design-centric Method for an Augmented Reality Robotic Surgery

Master'sMASTER OF ENGINEERIN