Teleoperation control based on combination of wave variable and neural networks

In this paper, a novel control scheme is developed for a teleoperation system, combining the radial basis function (RBF) neural networks (NNs) and wave variable technique to simultaneously compensate for the effects caused by communication delays and dynamics uncertainties. The teleoperation system is set up with a TouchX joystick as the master device and a simulated Baxter robot arm as the slave robot. The haptic feedback is provided to the human operator to sense the interaction force between the slave robot and the environment when manipulating the stylus of the joystick. To utilize the workspace of the telerobot as much as possible, a matching process is carried out between the master and the slave based on their kinematics models. The closed loop inverse kinematics method and RBF NN approximation technique are seamlessly integrated in the control design. To overcome the potential instability problem in the presence of delayed communication channels, wave variables and their corrections are effectively embedded into the control system, and Lyapunov-based analysis is performed to theoretically establish the closed-loop stability. Comparative experiments have been conducted for a trajectory tracking task, under the different conditions of various communication delays. Experimental results show that in terms of tracking performance and force reflection, the proposed control approach shows superior performance over the conventional methods

Adaptive Neural Network Fixed-Time Control Design for Bilateral Teleoperation With Time Delay.

In this article, subject to time-varying delay and uncertainties in dynamics, we propose a novel adaptive fixed-time control strategy for a class of nonlinear bilateral teleoperation systems. First, an adaptive control scheme is applied to estimate the upper bound of delay, which can resolve the predicament that delay has significant impacts on the stability of bilateral teleoperation systems. Then, radial basis function neural networks (RBFNNs) are utilized for estimating uncertainties in bilateral teleoperation systems, including dynamics, operator, and environmental models. Novel adaptation laws are introduced to address systems' uncertainties in the fixed-time convergence settings. Next, a novel adaptive fixed-time neural network control scheme is proposed. Based on the Lyapunov stability theory, the bilateral teleoperation systems are proved to be stable in fixed time. Finally, simulations and experiments are presented to verify the validity of the control algorithm

Output Feedback Bilateral Teleoperation with Force Estimation in the Presence of Time Delays

This thesis presents a novel bilateral teleoperation algorithm for n degree of freedom nonlinear manipulators connected through time delays. Teleoperation has many practical uses, as there are many benefits that come from being able to operate machines from a distance. For instance, the ability to send a remote controlled robotic vehicle into a hazardous environment can be a great asset in many industrial applications. As well, the field of remote medicine can benefit from these technologies. A highly skilled surgeon could perform surgery on a patient who is located in another city, or even country. Earth to space operations and deep sea exploration are other areas where teleoperation is quite useful. Central to the approach presented in this work is the use of second order sliding mode unknown input observers for estimating the external forces acting on the manipulators. The use of these observers removes the need for both velocity and force sensors, leading to a lower cost hardware setup that provides all of the advantages of a position-force teleoperation algorithm. Stability results for this new algorithm are presented for several cases. Stability of each of the master and slave sides of the teleoperation system is demonstrated, showing that the master and slave are both stabilized by their respective controllers when the unknown input observers are used for state and force estimation. Additionally, closed loop stability results for the teleoperation system connected to a variety of slave side environments are presented. Delay-independent stability results for a linear spring-damper environment as well as a general finite-gain stable nonlinear environment are given. Delay-dependent stability results for the case where the slave environment is a liner spring-damper and the delays are commensurate are also presented. As well, stability results for the closed loop under the assumption that the human operator is modeled as a finite-gain stable nonlinear environment are given. Following the theoretical presentation, numerical simulations illustrating the algorithm are presented, and experimental results verifying the practical application of the approach are given

Trust-Based Control of (Semi)Autonomous Mobile Robotic Systems

Despite great achievements made in (semi)autonomous robotic systems, human participa-tion is still an essential part, especially for decision-making about the autonomy allocation of robots in complex and uncertain environments. However, human decisions may not be optimal due to limited cognitive capacities and subjective human factors. In human-robot interaction (HRI), trust is a major factor that determines humans use of autonomy. Over/under trust may lead to dispro-portionate autonomy allocation, resulting in decreased task performance and/or increased human workload. In this work, we develop automated decision-making aids utilizing computational trust models to help human operators achieve a more effective and unbiased allocation. Our proposed decision aids resemble the way that humans make an autonomy allocation decision, however, are unbiased and aim to reduce human workload, improve the overall performance, and result in higher acceptance by a human. We consider two types of autonomy control schemes for (semi)autonomous mobile robotic systems. The first type is a two-level control scheme which includes switches between either manual or autonomous control modes. For this type, we propose automated decision aids via a computational trust and self-confidence model. We provide analytical tools to investigate the steady-state effects of the proposed autonomy allocation scheme on robot performance and human workload. We also develop an autonomous decision pattern correction algorithm using a nonlinear model predictive control to help the human gradually adapt to a better allocation pattern. The second type is a mixed-initiative bilateral teleoperation control scheme which requires mixing of autonomous and manual control. For this type, we utilize computational two-way trust models. Here, mixed-initiative is enabled by scaling the manual and autonomous control inputs with a function of computational human-to-robot trust. The haptic force feedback cue sent by the robot is dynamically scaled with a function of computational robot-to-human trust to reduce humans physical workload. Using the proposed control schemes, our human-in-the-loop tests show that the trust-based automated decision aids generally improve the overall robot performance and reduce the operator workload compared to a manual allocation scheme. The proposed decision aids are also generally preferred and trusted by the participants. Finally, the trust-based control schemes are extended to the single-operator-multi-robot applications. A theoretical control framework is developed for these applications and the stability and convergence issues under the switching scheme between different robots are addressed via passivity based measures

Visual Servoing in Robotics

Visual servoing is a well-known approach to guide robots using visual information. Image processing, robotics, and control theory are combined in order to control the motion of a robot depending on the visual information extracted from the images captured by one or several cameras. With respect to vision issues, a number of issues are currently being addressed by ongoing research, such as the use of different types of image features (or different types of cameras such as RGBD cameras), image processing at high velocity, and convergence properties. As shown in this book, the use of new control schemes allows the system to behave more robustly, efficiently, or compliantly, with fewer delays. Related issues such as optimal and robust approaches, direct control, path tracking, or sensor fusion are also addressed. Additionally, we can currently find visual servoing systems being applied in a number of different domains. This book considers various aspects of visual servoing systems, such as the design of new strategies for their application to parallel robots, mobile manipulators, teleoperation, and the application of this type of control system in new areas

Towards Intelligent Telerobotics: Visualization and Control of Remote Robot

Human-machine cooperative or co-robotics has been recognized as the next generation of robotics. In contrast to current systems that use limited-reasoning strategies or address problems in narrow contexts, new co-robot systems will be characterized by their flexibility, resourcefulness, varied modeling or reasoning approaches, and use of real-world data in real time, demonstrating a level of intelligence and adaptability seen in humans and animals. The research I focused is in the two sub-field of co-robotics: teleoperation and telepresence. We firstly explore the ways of teleoperation using mixed reality techniques. I proposed a new type of display: hybrid-reality display (HRD) system, which utilizes commodity projection device to project captured video frame onto 3D replica of the actual target surface. It provides a direct alignment between the frame of reference for the human subject and that of the displayed image. The advantage of this approach lies in the fact that no wearing device needed for the users, providing minimal intrusiveness and accommodating users eyes during focusing. The field-of-view is also significantly increased. From a user-centered design standpoint, the HRD is motivated by teleoperation accidents, incidents, and user research in military reconnaissance etc. Teleoperation in these environments is compromised by the Keyhole Effect, which results from the limited field of view of reference. The technique contribution of the proposed HRD system is the multi-system calibration which mainly involves motion sensor, projector, cameras and robotic arm. Due to the purpose of the system, the accuracy of calibration should also be restricted within millimeter level. The followed up research of HRD is focused on high accuracy 3D reconstruction of the replica via commodity devices for better alignment of video frame. Conventional 3D scanner lacks either depth resolution or be very expensive. We proposed a structured light scanning based 3D sensing system with accuracy within 1 millimeter while robust to global illumination and surface reflection. Extensive user study prove the performance of our proposed algorithm. In order to compensate the unsynchronization between the local station and remote station due to latency introduced during data sensing and communication, 1-step-ahead predictive control algorithm is presented. The latency between human control and robot movement can be formulated as a linear equation group with a smooth coefficient ranging from 0 to 1. This predictive control algorithm can be further formulated by optimizing a cost function. We then explore the aspect of telepresence. Many hardware designs have been developed to allow a camera to be placed optically directly behind the screen. The purpose of such setups is to enable two-way video teleconferencing that maintains eye-contact. However, the image from the see-through camera usually exhibits a number of imaging artifacts such as low signal to noise ratio, incorrect color balance, and lost of details. Thus we develop a novel image enhancement framework that utilizes an auxiliary color+depth camera that is mounted on the side of the screen. By fusing the information from both cameras, we are able to significantly improve the quality of the see-through image. Experimental results have demonstrated that our fusion method compares favorably against traditional image enhancement/warping methods that uses only a single image

Enhancement of the Tracking Performance for Robot Manipulator by Using the Feed-forward Scheme and Reasonable Switching Mechanism

Robot manipulator has become an exciting topic for many researchers during several decades. They have investigated the advanced algorithms such as sliding mode control, neural network, or genetic scheme to implement these developments. However, they lacked the integration of these algorithms to explore many potential expansions. Simultaneously, the complicated system requires a lot of computational costs, which is not always supported. Therefore, this paper presents a novel design of switching mechanisms to control the robot manipulator. This investigation is expected to achieve superior performance by flexibly adjusting various strategies for better selection. The Proportional-Integral-Derivative (PID) scheme is well-known, easy to implement, and ensures rapid computation while it might not have much control effect. The advanced interval type-2 fuzzy sliding mode control properly deals with nonlinear factors and disturbances. Consequently, the PID scheme is switched when the tracking error is less than the threshold or is far from the target. Otherwise, the interval type-2 fuzzy sliding mode control scheme is activated to cope with unknown factors. The main contributions of this paper are (i) the recommendation of a suitable switching mechanism to drive the robot manipulator, (ii) the successful integration of the interval type-2 fuzzy sliding mode control to track the desired trajectory, and (iii) the launching of several tests to validate the proposed controller with robot model. From these achievements, it would be stated that the proposed approach is effective in tracking performance, robust in disturbance-rejection, and feasible in practical implementation

Proceedings of the NASA Conference on Space Telerobotics, volume 4

Papers presented at the NASA Conference on Space Telerobotics are compiled. The theme of the conference was man-machine collaboration in space. The conference provided a forum for researchers and engineers to exchange ideas on the research and development required for the application of telerobotic technology to the space systems planned for the 1990's and beyond. Volume 4 contains papers related to the following subject areas: manipulator control; telemanipulation; flight experiments (systems and simulators); sensor-based planning; robot kinematics, dynamics, and control; robot task planning and assembly; and research activities at the NASA Langley Research Center

Nonlinear Control Techniques for Robot Manipulators

This Masters thesis describes the design and implementation of control strategies for the following topics of research: i) Whole Arm Grasping Control for Redundant Robot Manipulators, ii) Neural Network Grasping Controller for Continuum Robots and, iii) Coordination Control for Haptic and Teleoperator Systems. An approach to whole arm grasping of objects using redundant robot manipulators is presented. A kinematic control which facilitates the encoding of both the end-effector position, as well as body self-motion positioning information as a desired trajectory signal for the manipulator joints is developed. An approach is presented to whole arm grasping control for continuum robots. The grasping controller is developed in two stages; high level path planning for the grasping objective, and a low level joint controller using a neural network feedforward component to compensate for dynamic uncertainties. Lastly, two controllers are developed for nonlinear haptic and teleoperator systems for coordination of the master and slave systems