A Tele-Operated Display With a Predictive Display Algorithm

Tele-operated display systems with head mounted displays (HMD) are becoming popular as visual feedback systems for tele-operation systems. However, the users are suffered from time-varying bidirectional delays caused by the latency and limited bandwidth of wireless communication networks. Here, we develop a tele-operated display system and a predictive display algorithm allowing comfortable use of HMDs by operators of tele-operation systems. Inspired by the kinematic model of the human head-neck complex, we built a robot neck-camera system to capture the field of view in any desired orientation. To reduce the negative effects of the time-varying bidirectional communication delay and operation delay of the robot neck, we developed a predictive display algorithm based on a kinematic model of the human/robot neck-camera system, and a geometrical model of a camera. Experimental results showed that the system provide predicted images with high frame rate to the user

Predictive input delay compensation for motion control systems

This paper presents an analytical approach for the prediction of future motion to be used in input delay compensation of time-delayed motion control systems. The method makes use of the current and previous input values given to a nominally behaving system in order to realize the prediction of the future motion of that system. The generation of the future input is made through an integration which is realized in discrete time setting. Once the future input signal is created, it is used as the reference input of the remote system to enforce an input time delayed system, conduct a delay-free motion. Following the theoretical formulation, the proposed method is tested in experiments and the validity of the approach is verified

Control systems with network delay

In this paper motion control systems with delay in measurement and control channels are discussed and a new structure of the observer-predictor is proposed. The feature of the proposed system is enforcement of the convergence in both the estimation and the prediction of the plant output in the presence of the variable, unknown delay in both measurement and in the control channels. The estimation is based on the available data – undelayed control input, the delayed measurement of position or velocity and the nominal parameters of the plant and it does not require apriori knowledge of the delay. The stability and convergence is proven and selection of observer and the controller parameters is discussed. Experimental results are shown to illustrate the theoretical prediction

Network Latency in Teleoperation of Connected and Autonomous Vehicles:A Review of Trends, Challenges, and Mitigation Strategies

With remarkable advancements in the development of connected and autonomous vehicles (CAVs), the integration of teleoperation has become crucial for improving safety and operational efficiency. However, teleoperation faces substantial challenges, with network latency being a critical factor influencing its performance. This survey paper explores the impact of network latency along with state-of-the-art mitigation/compensation approaches. It examines cascading effects on teleoperation communication links (i.e., uplink and downlink) and how delays in data transmission affect the real-time perception and decision-making of operators. By elucidating the challenges and available mitigation strategies, the paper offers valuable insights for researchers, engineers, and practitioners working towards the seamless integration of teleoperation in the evolving landscape of CAVs

Teleoperation of passivity-based model reference robust control over the internet

This dissertation offers a survey of a known theoretical approach and novel experimental results in establishing a live communication medium through the internet to host a virtual communication environment for use in Passivity-Based Model Reference Robust Control systems with delays. The controller which is used as a carrier to support a robust communication between input-to-state stability is designed as a control strategy that passively compensates for position errors that arise during contact tasks and strives to achieve delay-independent stability for controlling of aircrafts or other mobile objects. Furthermore the controller is used for nonlinear systems, coordination of multiple agents, bilateral teleoperation, and collision avoidance thus maintaining a communication link with an upper bound of constant delay is crucial for robustness and stability of the overall system. For utilizing such framework an elucidation can be formulated by preparing site survey for analyzing not only the geographical distances separating the nodes in which the teleoperation will occur but also the communication parameters that define the virtual topography that the data will travel through. This survey will first define the feasibility of the overall operation since the teleoperation will be used to sustain a delay based controller over the internet thus obtaining a hypothetical upper bound for the delay via site survey is crucial not only for the communication system but also the delay is required for the design of the passivity-based model reference robust control. Following delay calculation and measurement via site survey, bandwidth tests for unidirectional and bidirectional communication is inspected to ensure that the speed is viable to maintain a real-time connection. Furthermore from obtaining the results it becomes crucial to measure the consistency of the delay throughout a sampled period to guarantee that the upper bound is not breached at any point within the communication to jeopardize the robustness of the controller. Following delay analysis a geographical and topological overview of the communication is also briefly examined via a trace-route to understand the underlying nodes and their contribution to the delay and round-trip consistency. To accommodate the communication channel for the controller the input and output data from both nodes need to be encapsulated within a transmission control protocol via a multithreaded design of a robust program within the C language. The program will construct a multithreaded client-server relationship in which the control data is transmitted. For added stability and higher level of security the channel is then encapsulated via an internet protocol security by utilizing a protocol suite for protecting the communication by authentication and encrypting each packet of the session using negotiation of cryptographic keys during each session

A Delay Compensation Framework Based on Eye-Movement for Teleoperated Ground Vehicles

An eye-movement-based predicted trajectory guidance control (ePTGC) is proposed to mitigate the maneuverability degradation of a teleoperated ground vehicle caused by communication delays. Human sensitivity to delays is the main reason for the performance degradation of a ground vehicle teleoperation system. The proposed framework extracts human intention from eye-movement. Then, it combines it with contextual constraints to generate an intention-compliant guidance trajectory, which is then employed to control the vehicle directly. The advantage of this approach is that the teleoperator is removed from the direct control loop by using the generated trajectories to guide vehicle, thus reducing the adverse sensitivity to delay. The delay can be compensated as long as the prediction horizon exceeds the delay. A human-in-loop simulation platform is designed to evaluate the teleoperation performance of the proposed method at different delay levels. The results are analyzed by repeated measures ANOVA, which shows that the proposed method significantly improves maneuverability and cognitive burden at large delay levels (>200 ms). The overall performance is also much better than the PTGC which does not employ the eye-movement feature.Comment: 9 pages, 11 figure

Position referenced force augmentation in teleoperated hydraulic manipulators operating under delayed and lossy networks: a pilot study.

Position error between motions of the master and slave end-effectors is inevitable as it originates from hard-to-avoid imperfections in controller design and model uncertainty. Moreover, when a slave manipulator is controlled through a delayed and lossy communication channel, the error between the desired motion originating from the master device and the actual movement of the slave manipulator end-effector is further exacerbated. This paper introduces a force feedback scheme to alleviate this problem by simply guiding the operator to slow down the haptic device motion and, in turn, allows the slave manipulator to follow the desired trajectory closely. Using this scheme, the master haptic device generates a force, which is proportional to the position error at the slave end-effector, and opposite to the operator's intended motion at the master site. Indeed, this force is a signal or cue to the operator for reducing the hand speed when position error, due to delayed and lossy network, appears at the slave site. Effectiveness of the proposed scheme is validated by performing experiments on a hydraulic telemanipulator setup developed for performing live-line maintenance. Experiments are conducted when the system operates under both dedicated and wireless networks. Results show that the scheme performs well in reducing the position error between the haptic device and the slave end-effector. Specifically, by utilizing the proposed force, the mean position error, for the case presented here, reduces by at least 92% as compared to the condition without the proposed force augmentation scheme. The scheme is easy to implement, as the only required on-line measurement is the angular displacement of the slave manipulator joints

Long future frame prediction using optical flow informed deep neural networks for enhancement of robotic teleoperation in high latency environments

High latency in teleoperation has a significant negative impact on operator performance. While deep learning has revolutionized many domains recently, it has not previously been applied to teleoperation enhancement. We propose a novel approach to predict video frames deep into the future using neural networks informed by synthetically generated optical flow information. This can be employed in teleoperated robotic systems that rely on video feeds for operator situational awareness. We have used the image-to-image translation technique as a basis for the prediction of future frames. The Pix2Pix conditional generative adversarial network (cGAN) has been selected as a base network. Optical flow components reflecting real-time control inputs are added to the standard RGB channels of the input image. We have experimented with three data sets of 20,000 input images each that were generated using our custom-designed teleoperation simulator with a 500-ms delay added between the input and target frames. Structural Similarity Index Measures (SSIMs) of 0.60 and Multi-SSIMs of 0.68 were achieved when training the cGAN with three-channel RGB image data. With the five-channel input data (incorporating optical flow) these values improved to 0.67 and 0.74, respectively. Applying Fleiss\u27 κ gave a score of 0.40 for three-channel RGB data, and 0.55 for five-channel optical flow-added data. We are confident the predicted synthetic frames are of sufficient quality and reliability to be presented to teleoperators as a video feed that will enhance teleoperation. To the best of our knowledge, we are the first to attempt to reduce the impacts of latency through future frame prediction using deep neural networks

Toward safe and stable time-delayed mobile robot teleoperation through sampling-based path planning

This work proposes a teleoperation architecture for mobile robots in partially unknown environments under the presence of variable time delay. The system is provided with artificial intelligence represented by a probabilistic path planner that, in combination with a prediction module, assists the operator while guaranteeing a collision-free motion. For this purpose, a certain level of autonomy is given to the system. The structure was tested in indoor environments for different kinds of operators. A maximum time delay of 2s was successfully coped with. © 2011 Cambridge University Press