Control of Cooperative Haptics-Enabled Teleoperation Systems with Application to Minimally Invasive Surgery

Robot-Assisted Minimally Invasive Surgical (RAMIS) systems frequently have a structure of cooperative teleoperator systems where multiple master-slave pairs are used to collaboratively execute a task. Although multiple studies indicate that haptic feedback improves the realism of tool-tissue interaction to the surgeon and leads to better performance for surgical procedures, current telesurgical systems typically do not provide force feedback, mainly because of the inherent stability issues. The research presented in this thesis is directed towards the development of control algorithms for force reflecting cooperative surgical teleoperator systems with improved stability and transparency characteristics. In the case of cooperative force reflecting teleoperation over networks, conventional passivity based approaches may have limited applicability due to potentially non-passive slave-slave interactions and irregular communication delays imposed by the network. In this thesis, an alternative small gain framework for the design of cooperative network-based force reflecting teleoperator systems is developed. Using the small gain framework, control algorithms for cooperative force-reflecting teleoperator systems are designed that guarantee stability in the presence of multiple network-induced communication constraints. Furthermore, the design conservatism typically associated with the small-gain approach is eliminated by using the Projection-Based Force Reflection (PBFR) algorithms. Stability results are established for networked cooperative teleoperator systems under different types of force reflection algorithms in the presence of irregular communication delays. The proposed control approach is consequently implemented on a dual-arm (two masters/two slaves) robotic MIS testbed. The testbed consists of two Haptic Wand devices as masters and two PA10-7C robots as the slave manipulators equipped with da Vinci laparoscopic surgical instruments. The performance of the proposed control approach is evaluated in three different cooperative surgical tasks, which are knot tightening, pegboard transfer, and object manipulation. The experimental results obtained indicate that the PBFR algorithms demonstrate statistically significant performance improvement in comparison with the conventional direct force reflection algorithms. One possible shortcoming of using PBFR algorithms is that implementation of these algorithms may lead to attenuation of the high-frequency component of the contact force which is important, in particular, for haptic perception of stiff surfaces. In this thesis, a solution to this problem is proposed which is based on the idea of separating the different frequency bands in the force reflection signal and consequently applying the projection-based principle to the low-frequency component, while reflecting the high-frequency component directly. The experimental results demonstrate that substantial improvement in transient fidelity of the force feedback is achieved using the proposed method without negative effects on the stability of the system

Posture-Dependent Projection-Based Force Reflection Algorithms for Bilateral Teleoperators

It was previously established that the projection-based force reflection (PBFR) algorithms improve the overall stability of a force reflecting teleoperation system. The idea behind the PBFR algorithms is to identify the component of the reflected force which is compensated by interaction with the operator\u27s hand, and subsequently attenuate the residual component of the reflected force. If there is no a priori information regarding the behaviour of the human operator, the PBFR gain is selected equal to sufficiently small constant in order to guarantee stability for a wide range of human operator responses. Small PBRF gains, however, may deteriorate the transparency of a teleoperator system. In this thesis, a new method for selecting the PBFR gain is introduced which depends on human postures. Using an online human posture estimation, the introduced posture-dependent PBFR algorithm has been applied to a teleoperation system with force feedback. It is experimentally demonstrated that the developed method for selection of the PBFR gain based on human postures improves the transparency of the teleoperator system while the stability is preserved. Finally, preliminary results that deal with an extension of the developed methods towards a more realistic model of the human arm with 4 degrees of freedom and three dimensional movements are presented

Generalized Scattering-Based Stabilization of Nonlinear Interconnected Systems

The research presented in this thesis is aimed at development of new methods and techniques for stability analysis and stabilization of interconnections of nonlinear systems, in particular, in the presence of communication delays. Based on the conic systems\u27 formalism, we extend the notion of conicity for the non-planar case where the dimension of the cone\u27s central subspace may be greater than one. One of the advantages of the notion of non-planar conicity is that any dissipative system with a quadratic supply rate can be represented as a non-planar conic system; specifically, its central subspace and radius can be calculated using an algorithm developed in this thesis. For a feedback interconnection of two non-planar conic systems, a graph separation condition for finite-gain L2-stability is established in terms of central subspaces and radii of the subsystems\u27 non-planar cones. Subsequently, a generalized version of the scattering transformation is developed which is applicable to non-planar conic systems. The transformation allows for rendering the dynamics of a non-planar conic system into a prescribed cone with compatible dimensions; the corresponding design algorithm is presented. The ability of the generalized scattering transformation to change the parameters of a system\u27s cone can be used for stabilization of interconnections of non-planar conic systems. For interconnections without communication delays, stabilization is achieved through the design of a scattering transformation that guarantees the fulfilment of the graph separation stability condition. For interconnected systems with communication delays, scattering transformations are designed on both sides of communication channel in a way that guarantees the overall stability through fulfilment of the small gain stability condition. Application to stabilization of bilateral teleoperators with multiple heterogeneous communication delays is briefly discussed. Next, the coupled stability problem is addressed based on the proposed scattering based stabilization techniques. The coupled stability problem is one of the most fundamental problems in robotics. It requires to guarantee stability of a controlled manipulator in contact with an environment whose dynamics are unknown, or at least not known precisely. We present a scattering-based design procedure that guarantees coupled stability while at the same time does not affect the robot\u27s trajectory tracking performance in free space. A detailed design example is presented that demonstrates the capabilities of the scattering-based design approach, as well as its advantages in comparison with more conventional passivity-based approaches. Finally, the generalized scattering-based technique is applied to the problem of stabilization of complex interconnections of dissipative systems with quadratic supply rates in the presence of multiple heterogeneous constant time delays. Our approach is to design local scattering transformations that guarantee the fulfilment of a multi-dimensional small-gain stability condition for the interconnected system. A numerical example is presented that illustrates the capabilities of the proposed design method

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

A Framework for Stable Robot-Environment Interaction Based on the Generalized Scattering Transformation

This thesis deals with development and experimental evaluation of control algorithms for stabilization of robot-environment interaction based on the conic systems formalism and scattering transformation techniques. A framework for stable robot-environment interaction is presented and evaluated on a real physical system. The proposed algorithm fundamentally generalizes the conventional passivity-based approaches to the coupled stability problem. In particular, it allows for stabilization of not necessarily passive robot-environment interaction. The framework is based on the recently developed non-planar conic systems formalism and generalized scattering-based stabilization methods. A comprehensive theoretical background on the scattering transformation techniques, planar and non-planar conic systems is presented. The dynamics of the robot are estimated using data-driven techniques, which allows the equations for the dynamics of a robot to be obtained in an explicit form. The generalized scattering transformation is used in combination with the Lyapunov-based adaptive trajectory tracking control. It is shown that the original interconnected system is not stable due to its non-passive nature; however, the application of the proposed stabilization algorithm allows stability to be achieved, without affecting the robot’s trajectory tracking performance in free space

Model Based Teleoperation to Eliminate Feedback Delay NSF Grant BCS89-01352 Second Report

We are conducting research in the area of teleoperation with feedback delay. Delay occurs with earth-based teleoperation in space and with surface-based teleoperation with untethered submersibles when acoustic communication links are involved. The delay in obtaining position and force feedback from remote slave arms makes teleoperation extremely difficult leading to very low productivity. We have combined computer graphics with manipulator programming to provide a solution to the problem. A teleoperator master arm is interfaced to a graphics based simulator of the remote environment. The system is then coupled with a robot manipulator at the remote, delayed site. The operator\u27s actions are monitored to provide both kinesthetic and visual feedback and to generate symbolic motion commands to the remote slave. The slave robot then executes these symbolic commands delayed in time. While much of a task proceeds error free, when an error does occur, the slave system transmits data back to the master environment which is then reset to the error state from which the operator continues the task

Human Machine Interfaces for Teleoperators and Virtual Environments

In Mar. 1990, a meeting organized around the general theme of teleoperation research into virtual environment display technology was conducted. This is a collection of conference-related fragments that will give a glimpse of the potential of the following fields and how they interplay: sensorimotor performance; human-machine interfaces; teleoperation; virtual environments; performance measurement and evaluation methods; and design principles and predictive models

Teleprogramming: Overcoming Communication Delays in Remote Manipulation (Dissertation Proposal)

Modern industrial processes (nuclear, chemical industry), public service needs (firefighting, rescuing), and research interests (undersea, outer space exploration) have established a clear need to perform work remotely. Whereas a purely autonomous manipulative capability would solve the problem, its realization is beyond the state of the art in robotics [Stark et al.,1988]. Some of the problems plaguing the development of autonomous systems are: a) anticipation, detection, and correction of the multitude of possible error conditions arising during task execution, b) development of general strategy planning techniques transcending any particular limited task domain, c) providing the robot system with real-time adaptive behavior to accommodate changes in the remote environment, d) allowing for on-line learning and performance improvement through experience , etc. The classical approach to tackle some of these problems has been to introduce problem solvers and expert systems as part of the remote robot workcell control system. However, such systems tend to be limited in scope (to remain intellectually and implementationally manageable), too slow to be useful in real-time robot task execution, and generally fail to adequately represent and model the complexities of the real world environment. These problems become particularly severe when only partial information about the remote environment is available

Haptic Sensory Feedback for Improved Interface to Smart Prosthetics.

Grip force feedback is not available in modern myoelectric upper-limb prostheses, yet its benefits are well known in object manipulation tasks performed through cable-driven body-powered prostheses. To evaluate the efficacy of grip force feedback in a myoelectric prosthesis, direct head-to-head comparisons should be made with body-powered prostheses, as well as with proposed designs that provide grip force feedback through haptic displays such as vibrotactile arrays. Direct comparisons, however, are difficult because myoelectric control for a trans-radial amputee uses residual muscles in the forearm, body-power generally refers interaction to the shoulder, and haptic displays often involve additional information encoding transformations. Currently, no unifying theory exists to cover both information encoding as well as the body part used for control or display. The work developed in this dissertation presents a systematic hypothesis-driven approach to evaluating both information encoding and body part used in the display of grip force feedback. Drawing upon principles from psychophysics, teleoperation, and sensory substitution, we use a series of human subject experiments to quantify the value of grip force feedback for an amputee wearing a trans-radial myoelectric prosthesis. Our findings demonstrate that both able-bodied individuals and amputees scale and coordinate their grip force for the anticipated weight of an object, that control and grip force feedback should be located on the same body site to improve stiffness recognition, and that grip force feedback is more useful than vision feedback in stiffness recognition through a prosthesis.PhDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/108792/1/jdelaine_1.pd