Position / force control of systems subjected to communicaton delays and interruptions in bilateral teleoperation

Thesis (Master)--Izmir Institute of Technology, Mechanical Engineering, Izmir, 2012Includes bibliographical references (leaves: 65-68)Text in English; Abstract: Turkish and Englishix, 76 leavesTeleoperation technology allows to remotely operate robotic (slave) systems located in hazardous, risky and distant environments. The human operator sends commands through the controller (master) system to execute the tasks from a distance. The operator is provided with necessary (visual, audio or haptic) feedback to accomplish the mission remotely. In bilateral teleoperation, continuous feedback from the remote environment is generated. Thus, the operator can handle the task as if the operator is in the remote environment relying on the relevant feedback. Since teleoperation deals with systems controlled from a distance, time delays and package losses in transmission of information are present. These communication failures affect the human perception and system stability, and thus, the ability of operator to handle the task successfully. The objective of this thesis is to investigate and develop a control algorithm, which utilizes model mediated teleoperation integrating parallel position/force controllers, to compensate for the instability issues and excessive forcing applied to the environment arising from communication failures. Model mediation technique is extended for three-degrees-of-freedom teleoperation and a parallel position/force controller, impedance controller, is integrated in the control algorithm. The proposed control method is experimentally tested by using Matlab Simulink blocksets for real-time experimentation in which haptic desktop devices, Novint Falcon and Phantom Desktop are configured as master and slave subsystems of the bilateral teleoperation. The results of these tests indicate that the stability and passivity of proposed bilateral teleoperation systems are preserved during constant and variable time delays and data losses while the position and force tracking test results provide acceptable performance with bounded errors

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

Hybrid Stiff/Compliant Workspace Control for Robotized Minimally Invasive Surgery

Abstract-This paper presents a novel control architecture for hybrid stiff and compliant control for minimally invasive surgery which satisfies the constraints of zero lateral velocity at the entry point for serial manipulators. For minimally invasive surgery it is required that there is no sideways motion at the point where the robots enter the abdomen. This is necessary to avoid any damage to the patient's body when the robot moves. We solve this at a kinematic level, i.e., we find a Jacobian matrix that maps the velocities in joint space to the end-effector velocities and at the same time guarantees that certain velocities at the entry point are zero. Because the new velocity variables are defined in the end-effector workspace we can use these for hybrid motion/force control. The approach is verified experimentally by implementing hybrid stiff and compliant control of the end effector and we show that the insertion point constraints are always satisfied

Model Driven Robotic Assistance for Human-Robot Collaboration

While robots routinely perform complex assembly tasks in highly structured factory environments, it is challenging to apply completely autonomous robotic systems in less structured manipulation tasks, such as surgery and machine assembly/repair, due to the limitations of machine intelligence, sensor data interpretation and environment modeling. A practical, yet effective approach to accomplish these tasks is through human-robot collaboration, in which the human operator and the robot form a partnership and complement each other in performing a complex task. We recognize that humans excel at determining task goals and recognizing constraints, if given sufficient feedback about the interaction between the tool (e.g., end-effector of the robot) and the environment. Robots are precise, unaffected by fatigue and able to work in environments not suitable for humans. We hypothesize that by providing the operator with adequate information about the task, through visual and force (haptic) feedback, the operator can: (1) define the task model, in terms of task goals and virtual fixture constraints through an interactive, or immersive augmented reality interface, and (2) have the robot actively assist the operator to enhance the execution time, quality and precision of the tasks. We validate our approaches through the implementations of both cooperative (i.e., hands-on) control and telerobotic systems, for image-guided robotic neurosurgery and telerobotic manipulation tasks for satellite servicing under significant time delay

A learning-based shared control architecture for interactive task execution

Shared control is a key technology for various robotic applications in which a robotic system and a human operator are meant to collaborate efficiently. In order to achieve efficient task execution in shared control, it is essential to predict the desired behavior for a given situation or context to simplify the control task for the human operator. To do this prediction, we use Learning from Demonstration (LfD), which is a popular approach for transferring human skills to robots. We encode the demonstrated behavior as trajectory distributions and generalize the learned distributions to new situations. The goal of this paper is to present a shared control framework that uses learned expert distributions to gain more autonomy. Our approach controls the balance between the controller’s autonomy and the human preference based on the distributions of the demonstrated trajectories. Moreover, the learned distributions are autonomously refined from collaborative task executions, resulting in a master-slave system with increasing autonomy that requires less user input with an increasing number of task executions. We experimentally validated that our shared control approach enables efficient task executions. Moreover, the conducted experiments demonstrated that the developed system improves its performances through interactive task executions with our shared control

An Adaptive Tool-Based Telerobot Control System

Modern telerobotics concepts seek to improve the work efficiency and quality of remote operations. The unstructured nature of typical remote operational environments makes autonomous operation of telerobotic systems difficult to achieve. Thus, human operators must always remain in the control loop for safety reasons. Remote operations involve tooling interactions with task environment. These interactions can be strong enough to promote unstable operation sometimes leading to system failures. Interestingly, manipulator/tooling dynamic interactions have not been studied in detail. This dissertation introduces a human-machine cooperative telerobotic (HMCTR) system architecture that has the ability to incorporate tooling interaction control and other computer assistance functions into the overall control system. A universal tooling interaction force prediction model has been created and implemented using grey system theory. Finally, a grey prediction force/position parallel fuzzy controller has been developed that compensates for the tooling interaction forces. Detailed experiments using a full-scale telerobotics testbed indicate: (i) the feasibility of the developed methodologies, and (ii) dramatic improvements in the stability of manipulator – based on band saw cutting operations. These results are foundational toward the further enhancement and development of telerobot

Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Passivity-Based adaptive bilateral teleoperation control for uncertain manipulators without jerk measurements

In this work, we consider the bilateral teleoperation problem of cooperative robotic systems in a Single-Master Multi-Slave (SM/MS) configuration, which is able to perform load transportation tasks in the presence of parametric uncertainty in the robot kinematic and dynamic models. The teleoperation architecture is based on the two-layer approach placed in a hierarchical structure, whose top and bottom layers are responsible for ensuring the transparency and stability properties respectively. The load transportation problem is tackled by using the formation control approach wherein the desired translational velocity and interaction force are provided to the master robot by the user, while the object is manipulated with a bounded constant force by the slave robots. Firstly, we develop an adaptive kinematic-based control scheme based on a composite adaptation law to solve the cooperative control problem for robots with uncertain kinematics. Secondly, the dynamic adaptive control for cooperative robots is implemented by means of a cascade control strategy, which does not require the measurement of the time derivative of force (which requires jerk measurements). The combination of the Lyapunov stability theory and the passivity formalism are used to establish the stability and convergence property of the closed-loop control system. Simulations and experimental results illustrate the performance and feasibility of the proposed control scheme.No presente trabalho, considera-se o problema de teleoperação bilateral de um sistema robótico cooperativo do tipo single-master e multiple-slaves (SM/MS) capaz de realizar tarefas de transporte de carga na presença de incertezas paramétricas no modelo cinemático e dinâmico dos robôs. A arquitetura de teleoperação está baseada na abordagem de duas camadas em estrutura hierárquica, onde as camadas superior e inferior são responsáveis por assegurar as propriedades de transparência e estabilidade respectivamente. O problema de transporte de carga é formulado usando a abordagem de controle de formação onde a velocidade de translação desejada e a força de interação são fornecidas ao robô mestre pelo operador, enquanto o objeto é manipulado pelos robôs escravos com uma força constante limitada. Primeiramente, desenvolve-se um esquema de controle adaptativo cinemático baseado em uma lei de adaptação composta para solucionar o problema de controle cooperativo de robôs com cinemática incerta. Em seguida, o controle adaptativo dinâmico de robôs cooperativos é implementado por meio de uma estratégia de controle em cascata, que não requer a medição da derivada da força (o qual requer a derivada da aceleração ou jerk). A teoria de estabilidade de Lyapunov e o formalismo de passividade são usados para estabelecer as propriedades de estabilidade e a convergência do sistema de controle em malha-fechada. Resultados de simulações numéricas ilustram o desempenho e viabilidade da estratégia de controle proposta

Haptic Device Design and Teleoperation Control Algorithms for Mobile Manipulators

The increasing need of teleoperated robotic systems implies more and more often to use, as slave devices, mobile platforms (terrestrial, aerial or underwater) with integrated manipulation capabilities, provided e.g. by robotic arms with proper grasping/manipulation tools. Despite this, the research activity in teleoperation of robotic systems has mainly focused on the control of either fixed-base manipulators or mobile robots, non considering the integration of these two types of systems in a single device. Such a combined robotic devices are usually referred to as mobile manipulators: systems composed by both a robotic manipulator and a mobile platform (on which the arm is mounted) whose purpose is to enlarge the manipulator’s workspace. The combination of a mobile platform and a serial manipulator creates redundancy: a particular point in the space can be reached by moving the manipulator, by moving the mobile platform, or by a combined motion of both. A synchronized motion of both devices need then to be addressed. Although specific haptic devices explicitly oriented to the control of mobile manipulators need to be designed, there are no commercial solution yet. For this reason it is often necessary to control such as combined systems with traditional haptic devices not specifically oriented to the control of mobile manipulators. The research activity presented in this Ph.D. thesis focuses in the first place on the design of a teleoperation control scheme which allows the simultaneous control of both the manipulator and the mobile platform by means of a single haptic device characterized by fixed base and an open kinematic chain. Secondly the design of a novel cable-drive haptic devices has been faced. Investigating the use of twisted strings actuation in force rendering is the most interesting challenge of the latter activity