Passive bilateral teleoperation framework for assisted robotic tasks

This paper addresses the problem of stable bilateral teleoperation, through a time-delayed communication channel, for a n-DOF system without position drift. The classical wave variables-based scheme encodes velocity and force whilst the proposed scheme encodes position and integral of force. The resulting passive communication scheme is used in an extended teleoperation framework, in which the human operator can define geometrical motion restrictions in order to increase his awareness and his feeling of immersion, hence improving his ability to perform complex tasks. The framework also provides the operator with video feedback and 3D rendering of the remote scene.Peer Reviewe

Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

Haptics in Robot-Assisted Surgery: Challenges and Benefits

Robotic surgery is transforming the current surgical practice, not only by improving the conventional surgical methods but also by introducing innovative robot-enhanced approaches that broaden the capabilities of clinicians. Being mainly of man-machine collaborative type, surgical robots are seen as media that transfer pre- and intra-operative information to the operator and reproduce his/her motion, with appropriate filtering, scaling, or limitation, to physically interact with the patient. The field, however, is far from maturity and, more critically, is still a subject of controversy in medical communities. Limited or absent haptic feedback is reputed to be among reasons that impede further spread of surgical robots. In this paper objectives and challenges of deploying haptic technologies in surgical robotics is discussed and a systematic review is performed on works that have studied the effects of providing haptic information to the users in major branches of robotic surgery. It has been tried to encompass both classical works and the state of the art approaches, aiming at delivering a comprehensive and balanced survey both for researchers starting their work in this field and for the experts

Enhancing bilateral teleoperation using camera-based online virtual fixtures generation

In this paper we present an interactive system to enhance bilateral teleoperation through online virtual fixtures generation and task switching. This is achieved using a stereo camera system which provides accurate information of the surrounding environment of the robot and of the tasks that have to be performed in it. The use of the proposed approach aims at improving the performances of bilateral teleoperation systems by reducing the human operator workload and increasing both the implementation and the execution efficiency. In fact, using our method virtual guidances do not need to be programmed a priori but they can be instead automatically generated and updated making the system suitable for unstructured environments. We strengthen the proposed method using passivity control in order to safely switch between different tasks while teleoperating under active constraints. A series of experiments emulating real industrial scenarios are used to show that the switch between multiple tasks can be passively and safely achieved and handled by the system

Prevalence of haptic feedback in robot-mediated surgery : a systematic review of literature

© 2017 Springer-Verlag. This is a post-peer-review, pre-copyedit version of an article published in Journal of Robotic Surgery. The final authenticated version is available online at: https://doi.org/10.1007/s11701-017-0763-4With the successful uptake and inclusion of robotic systems in minimally invasive surgery and with the increasing application of robotic surgery (RS) in numerous surgical specialities worldwide, there is now a need to develop and enhance the technology further. One such improvement is the implementation and amalgamation of haptic feedback technology into RS which will permit the operating surgeon on the console to receive haptic information on the type of tissue being operated on. The main advantage of using this is to allow the operating surgeon to feel and control the amount of force applied to different tissues during surgery thus minimising the risk of tissue damage due to both the direct and indirect effects of excessive tissue force or tension being applied during RS. We performed a two-rater systematic review to identify the latest developments and potential avenues of improving technology in the application and implementation of haptic feedback technology to the operating surgeon on the console during RS. This review provides a summary of technological enhancements in RS, considering different stages of work, from proof of concept to cadaver tissue testing, surgery in animals, and finally real implementation in surgical practice. We identify that at the time of this review, while there is a unanimous agreement regarding need for haptic and tactile feedback, there are no solutions or products available that address this need. There is a scope and need for new developments in haptic augmentation for robot-mediated surgery with the aim of improving patient care and robotic surgical technology further.Peer reviewe

Control of Cooperative Haptics-Enabled Teleoperation Systems with Application to Minimally Invasive Surgery

Robot-Assisted Minimally Invasive Surgical (RAMIS) systems frequently have a structure of cooperative teleoperator systems where multiple master-slave pairs are used to collaboratively execute a task. Although multiple studies indicate that haptic feedback improves the realism of tool-tissue interaction to the surgeon and leads to better performance for surgical procedures, current telesurgical systems typically do not provide force feedback, mainly because of the inherent stability issues. The research presented in this thesis is directed towards the development of control algorithms for force reflecting cooperative surgical teleoperator systems with improved stability and transparency characteristics. In the case of cooperative force reflecting teleoperation over networks, conventional passivity based approaches may have limited applicability due to potentially non-passive slave-slave interactions and irregular communication delays imposed by the network. In this thesis, an alternative small gain framework for the design of cooperative network-based force reflecting teleoperator systems is developed. Using the small gain framework, control algorithms for cooperative force-reflecting teleoperator systems are designed that guarantee stability in the presence of multiple network-induced communication constraints. Furthermore, the design conservatism typically associated with the small-gain approach is eliminated by using the Projection-Based Force Reflection (PBFR) algorithms. Stability results are established for networked cooperative teleoperator systems under different types of force reflection algorithms in the presence of irregular communication delays. The proposed control approach is consequently implemented on a dual-arm (two masters/two slaves) robotic MIS testbed. The testbed consists of two Haptic Wand devices as masters and two PA10-7C robots as the slave manipulators equipped with da Vinci laparoscopic surgical instruments. The performance of the proposed control approach is evaluated in three different cooperative surgical tasks, which are knot tightening, pegboard transfer, and object manipulation. The experimental results obtained indicate that the PBFR algorithms demonstrate statistically significant performance improvement in comparison with the conventional direct force reflection algorithms. One possible shortcoming of using PBFR algorithms is that implementation of these algorithms may lead to attenuation of the high-frequency component of the contact force which is important, in particular, for haptic perception of stiff surfaces. In this thesis, a solution to this problem is proposed which is based on the idea of separating the different frequency bands in the force reflection signal and consequently applying the projection-based principle to the low-frequency component, while reflecting the high-frequency component directly. The experimental results demonstrate that substantial improvement in transient fidelity of the force feedback is achieved using the proposed method without negative effects on the stability of the system

Passivity-Based adaptive bilateral teleoperation control for uncertain manipulators without jerk measurements

In this work, we consider the bilateral teleoperation problem of cooperative robotic systems in a Single-Master Multi-Slave (SM/MS) configuration, which is able to perform load transportation tasks in the presence of parametric uncertainty in the robot kinematic and dynamic models. The teleoperation architecture is based on the two-layer approach placed in a hierarchical structure, whose top and bottom layers are responsible for ensuring the transparency and stability properties respectively. The load transportation problem is tackled by using the formation control approach wherein the desired translational velocity and interaction force are provided to the master robot by the user, while the object is manipulated with a bounded constant force by the slave robots. Firstly, we develop an adaptive kinematic-based control scheme based on a composite adaptation law to solve the cooperative control problem for robots with uncertain kinematics. Secondly, the dynamic adaptive control for cooperative robots is implemented by means of a cascade control strategy, which does not require the measurement of the time derivative of force (which requires jerk measurements). The combination of the Lyapunov stability theory and the passivity formalism are used to establish the stability and convergence property of the closed-loop control system. Simulations and experimental results illustrate the performance and feasibility of the proposed control scheme.No presente trabalho, considera-se o problema de teleoperação bilateral de um sistema robótico cooperativo do tipo single-master e multiple-slaves (SM/MS) capaz de realizar tarefas de transporte de carga na presença de incertezas paramétricas no modelo cinemático e dinâmico dos robôs. A arquitetura de teleoperação está baseada na abordagem de duas camadas em estrutura hierárquica, onde as camadas superior e inferior são responsáveis por assegurar as propriedades de transparência e estabilidade respectivamente. O problema de transporte de carga é formulado usando a abordagem de controle de formação onde a velocidade de translação desejada e a força de interação são fornecidas ao robô mestre pelo operador, enquanto o objeto é manipulado pelos robôs escravos com uma força constante limitada. Primeiramente, desenvolve-se um esquema de controle adaptativo cinemático baseado em uma lei de adaptação composta para solucionar o problema de controle cooperativo de robôs com cinemática incerta. Em seguida, o controle adaptativo dinâmico de robôs cooperativos é implementado por meio de uma estratégia de controle em cascata, que não requer a medição da derivada da força (o qual requer a derivada da aceleração ou jerk). A teoria de estabilidade de Lyapunov e o formalismo de passividade são usados para estabelecer as propriedades de estabilidade e a convergência do sistema de controle em malha-fechada. Resultados de simulações numéricas ilustram o desempenho e viabilidade da estratégia de controle proposta

Control of Networked Robotic Systems

With the infrastructure of ubiquitous networks around the world, the study of robotic systems over communication networks has attracted widespread attention. This area is denominated as networked robotic systems. By exploiting the fruitful technological developments in networking and computing, networked robotic systems are endowed with potential and capabilities for several applications. Robots within a network are capable of connecting with control stations, human operators, sensors, and other robots via digital communication over possibly noisy channels/media. The issues of time delays in communication and data losses have emerged as a pivotal issue that have stymied practical deployment. The aim of this dissertation is to develop control algorithms and architectures for networked robotic systems that guarantee stability with improved overall performance in the presence of time delays in communication. The first topic addressed in this dissertation is controlled synchronization that is utilized for networked robotic systems to achieve collective behaviors. Exploiting passivity property of individual robotic systems, the proposed control schemes and interconnections are shown to ensure stability and convergence of synchronizing errors. The robustness of the control algorithms to constant and time-varying communication delays is also studied. In addition to time delays, the number of communication links, which prevents scalability of networked robotic systems, is another challenging issue. Thus, a synchronizing control with practically feasible constraints of network topology is developed. The problem of networked robotic systems interacting with human operators is then studied subsequently. This research investigates a teleoperation system with heterogeneous robots under asymmetric and unknown communication delays. Sub-task controllers are proposed for redundant slave robot to autonomously achieve additional tasks, such as singularity avoidance, joint angle limits, and collision avoidance. The developed control algorithms can enhance the efficiency of teleoperation systems, thereby ameliorating the performance degradation due to cognitive limitations of human operator and incomplete information about the environment. Compared to traditional robotic systems, control of robotic manipulators over networks has significant advantages; for example, increased flexibility and ease of maintenance. With the utilization of scattering variables, this research demonstrates that transmitting scattering variables over delayed communications can stabilize an otherwise unstable system. An architecture utilizing delayed position feedback in conjunction with scattering variables is developed for the case of time-varying communication delays. The proposed control architecture improves tracking performance and stabilizes robotic manipulators with input-output communication delays. The aforementioned control algorithms and architectures for networked robotic systems are validated via numerical examples and experiments