Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Legged robotic locomotion with variable impedance joints

Humans have a complex musculoskeletal arrangement which gives them great behavioural flexibility. As well as simply moving their legs, they can modulate the impedance of them. Variable impedance has become a large field in robotics, and tailoring the impedance of a robot to a particular task can improve efficiency, stability, and potentially safety. Locomotion of a bipedal robot is a perfect example of a task for which variable impedance may provide such advantages, since it is a dynamic movement which involves periodic ground impacts. This thesis explores the creation of two novel bipedal robots with variable impedance joints. These robots aim to achieve some of the benefits of compliance, while retaining the behavioural flexibility to be truly versatile machines. The field of variable impedance actuators is explored and evaluated, before the design of the robots is presented. Of the two robots, BLUE (Bipedal Locomotion at the University of Edinburgh) has a 700mm hip rotation height, and is a saggital plane biped. miniBLUE has a hip rotation height of 465mm, and includes additional joints to allow hip adduction and abduction. Rapid prototyping techniques were utilised in the creation of both robots, and both robots are based around a custom, high performance electronics and communication architecture. The human walking cycle is analysed and a simple, parameterised representation developed. Walking trajectories gathered from human motion capture data, and generated from high level gait determinants are evaluated in dynamic simulation, and then on BLUE. With the robot being capable of locomotion, we explore the effect of varying stiffness on efficiency, and find that changing the stiffness can have an effect on the energy efficiency of the movement. Finally, we introduce a system for goal-based teleoperation of the robots, in which parameters are extracted from a user in a motion capture suit and replicated by the robot. In this way, the robot produces the same overall locomotion as the human, but with joint trajectories and stiffnesses that are more suited for its dynamics

Steering control for haptic feedback and active safety functions

Steering feedback is an important element that defines driver–vehicle interaction. It strongly affects driving performance and is primarily dependent on the steering actuator\u27s control strategy. Typically, the control method is open loop, that is without any reference tracking; and its drawbacks are hardware dependent steering feedback response and attenuated driver–environment transparency. This thesis investigates a closed-loop control method for electric power assisted steering and steer-by-wire systems. The advantages of this method, compared to open loop, are better hardware impedance compensation, system independent response, explicit transparency control and direct interface to active safety functions.The closed-loop architecture, outlined in this thesis, includes a reference model, a feedback controller and a disturbance observer. The feedback controller forms the inner loop and it ensures: reference tracking, hardware impedance compensation and robustness against the coupling uncertainties. Two different causalities are studied: torque and position control. The two are objectively compared from the perspective of (uncoupled and coupled) stability, tracking performance, robustness, and transparency.The reference model forms the outer loop and defines a torque or position reference variable, depending on the causality. Different haptic feedback functions are implemented to control the following parameters: inertia, damping, Coulomb friction and transparency. Transparency control in this application is particularly novel, which is sequentially achieved. For non-transparent steering feedback, an environment model is developed such that the reference variable is a function of virtual dynamics. Consequently, the driver–steering interaction is independent from the actual environment. Whereas, for the driver–environment transparency, the environment interaction is estimated using an observer; and then the estimated signal is fed back to the reference model. Furthermore, an optimization-based transparency algorithm is proposed. This renders the closed-loop system transparent in case of environmental uncertainty, even if the initial condition is non-transparent.The steering related active safety functions can be directly realized using the closed-loop steering feedback controller. This implies, but is not limited to, an angle overlay from the vehicle motion control functions and a torque overlay from the haptic support functions.Throughout the thesis, both experimental and the theoretical findings are corroborated. This includes a real-time implementation of the torque and position control strategies. In general, it can be concluded that position control lacks performance and robustness due to high and/or varying system inertia. Though the problem is somewhat mitigated by a robust H-infinity controller, the high frequency haptic performance remains compromised. Whereas, the required objectives are simultaneously achieved using a torque controller

Control of Cooperative Haptics-Enabled Teleoperation Systems with Application to Minimally Invasive Surgery

Robot-Assisted Minimally Invasive Surgical (RAMIS) systems frequently have a structure of cooperative teleoperator systems where multiple master-slave pairs are used to collaboratively execute a task. Although multiple studies indicate that haptic feedback improves the realism of tool-tissue interaction to the surgeon and leads to better performance for surgical procedures, current telesurgical systems typically do not provide force feedback, mainly because of the inherent stability issues. The research presented in this thesis is directed towards the development of control algorithms for force reflecting cooperative surgical teleoperator systems with improved stability and transparency characteristics. In the case of cooperative force reflecting teleoperation over networks, conventional passivity based approaches may have limited applicability due to potentially non-passive slave-slave interactions and irregular communication delays imposed by the network. In this thesis, an alternative small gain framework for the design of cooperative network-based force reflecting teleoperator systems is developed. Using the small gain framework, control algorithms for cooperative force-reflecting teleoperator systems are designed that guarantee stability in the presence of multiple network-induced communication constraints. Furthermore, the design conservatism typically associated with the small-gain approach is eliminated by using the Projection-Based Force Reflection (PBFR) algorithms. Stability results are established for networked cooperative teleoperator systems under different types of force reflection algorithms in the presence of irregular communication delays. The proposed control approach is consequently implemented on a dual-arm (two masters/two slaves) robotic MIS testbed. The testbed consists of two Haptic Wand devices as masters and two PA10-7C robots as the slave manipulators equipped with da Vinci laparoscopic surgical instruments. The performance of the proposed control approach is evaluated in three different cooperative surgical tasks, which are knot tightening, pegboard transfer, and object manipulation. The experimental results obtained indicate that the PBFR algorithms demonstrate statistically significant performance improvement in comparison with the conventional direct force reflection algorithms. One possible shortcoming of using PBFR algorithms is that implementation of these algorithms may lead to attenuation of the high-frequency component of the contact force which is important, in particular, for haptic perception of stiff surfaces. In this thesis, a solution to this problem is proposed which is based on the idea of separating the different frequency bands in the force reflection signal and consequently applying the projection-based principle to the low-frequency component, while reflecting the high-frequency component directly. The experimental results demonstrate that substantial improvement in transient fidelity of the force feedback is achieved using the proposed method without negative effects on the stability of the system

Generalized Scattering-Based Stabilization of Nonlinear Interconnected Systems

The research presented in this thesis is aimed at development of new methods and techniques for stability analysis and stabilization of interconnections of nonlinear systems, in particular, in the presence of communication delays. Based on the conic systems\u27 formalism, we extend the notion of conicity for the non-planar case where the dimension of the cone\u27s central subspace may be greater than one. One of the advantages of the notion of non-planar conicity is that any dissipative system with a quadratic supply rate can be represented as a non-planar conic system; specifically, its central subspace and radius can be calculated using an algorithm developed in this thesis. For a feedback interconnection of two non-planar conic systems, a graph separation condition for finite-gain L2-stability is established in terms of central subspaces and radii of the subsystems\u27 non-planar cones. Subsequently, a generalized version of the scattering transformation is developed which is applicable to non-planar conic systems. The transformation allows for rendering the dynamics of a non-planar conic system into a prescribed cone with compatible dimensions; the corresponding design algorithm is presented. The ability of the generalized scattering transformation to change the parameters of a system\u27s cone can be used for stabilization of interconnections of non-planar conic systems. For interconnections without communication delays, stabilization is achieved through the design of a scattering transformation that guarantees the fulfilment of the graph separation stability condition. For interconnected systems with communication delays, scattering transformations are designed on both sides of communication channel in a way that guarantees the overall stability through fulfilment of the small gain stability condition. Application to stabilization of bilateral teleoperators with multiple heterogeneous communication delays is briefly discussed. Next, the coupled stability problem is addressed based on the proposed scattering based stabilization techniques. The coupled stability problem is one of the most fundamental problems in robotics. It requires to guarantee stability of a controlled manipulator in contact with an environment whose dynamics are unknown, or at least not known precisely. We present a scattering-based design procedure that guarantees coupled stability while at the same time does not affect the robot\u27s trajectory tracking performance in free space. A detailed design example is presented that demonstrates the capabilities of the scattering-based design approach, as well as its advantages in comparison with more conventional passivity-based approaches. Finally, the generalized scattering-based technique is applied to the problem of stabilization of complex interconnections of dissipative systems with quadratic supply rates in the presence of multiple heterogeneous constant time delays. Our approach is to design local scattering transformations that guarantee the fulfilment of a multi-dimensional small-gain stability condition for the interconnected system. A numerical example is presented that illustrates the capabilities of the proposed design method

A Stable and Transparent Framework for Adaptive Shared Control of Robots

In mixed-initiative haptic shared control of robots, humans and automatic control system work in parallel. The commands to the robot are a weighted sum of forces from these two agents. This thesis develops control methods to improve the force feedback performance for mixed-initiative shared teleoperation and to adapt the control authority between human and automatic control system in a stable manner even in the presence of communication delays. All methods are validated on real robotic hardware

Robotic manipulators for single access surgery

This thesis explores the development of cooperative robotic manipulators for enhancing surgical precision and patient outcomes in single-access surgery and, specifically, Transanal Endoscopic Microsurgery (TEM). During these procedures, surgeons manipulate a heavy set of instruments via a mechanical clamp inserted in the patient’s body through a surgical port, resulting in imprecise movements, increased patient risks, and increased operating time. Therefore, an articulated robotic manipulator with passive joints is initially introduced, featuring built-in position and force sensors in each joint and electronic joint brakes for instant lock/release capability. The articulated manipulator concept is further improved with motorised joints, evolving into an active tool holder. The joints allow the incorporation of advanced robotic capabilities such as ultra-lightweight gravity compensation and hands-on kinematic reconfiguration, which can optimise the placement of the tool holder in the operating theatre. Due to the enhanced sensing capabilities, the application of the active robotic manipulator was further explored in conjunction with advanced image guidance approaches such as endomicroscopy. Recent advances in probe-based optical imaging such as confocal endomicroscopy is making inroads in clinical uses. However, the challenging manipulation of imaging probes hinders their practical adoption. Therefore, a combination of the fully cooperative robotic manipulator with a high-speed scanning endomicroscopy instrument is presented, simplifying the incorporation of optical biopsy techniques in routine surgical workflows. Finally, another embodiment of a cooperative robotic manipulator is presented as an input interface to control a highly-articulated robotic instrument for TEM. This master-slave interface alleviates the drawbacks of traditional master-slave devices, e.g., using clutching mechanics to compensate for the mismatch between slave and master workspaces, and the lack of intuitive manipulation feedback, e.g. joint limits, to the user. To address those drawbacks a joint-space robotic manipulator is proposed emulating the kinematic structure of the flexible robotic instrument under control.Open Acces

Technology for the Future: In-Space Technology Experiments Program, part 2

The purpose of the Office of Aeronautics and Space Technology (OAST) In-Space Technology Experiments Program In-STEP 1988 Workshop was to identify and prioritize technologies that are critical for future national space programs and require validation in the space environment, and review current NASA (In-Reach) and industry/ university (Out-Reach) experiments. A prioritized list of the critical technology needs was developed for the following eight disciplines: structures; environmental effects; power systems and thermal management; fluid management and propulsion systems; automation and robotics; sensors and information systems; in-space systems; and humans in space. This is part two of two parts and contains the critical technology presentations for the eight theme elements and a summary listing of critical space technology needs for each theme

Proceedings of the NASA Conference on Space Telerobotics, volume 4

Papers presented at the NASA Conference on Space Telerobotics are compiled. The theme of the conference was man-machine collaboration in space. The conference provided a forum for researchers and engineers to exchange ideas on the research and development required for the application of telerobotic technology to the space systems planned for the 1990's and beyond. Volume 4 contains papers related to the following subject areas: manipulator control; telemanipulation; flight experiments (systems and simulators); sensor-based planning; robot kinematics, dynamics, and control; robot task planning and assembly; and research activities at the NASA Langley Research Center