Teleoperating a mobile manipulator and a free-flying camera from a single haptic device

© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThe paper presents a novel teleoperation system that allows the simultaneous and continuous command of a ground mobile manipulator and a free flying camera, implemented using an UAV, from which the operator can monitor the task execution in real-time. The proposed decoupled position and orientation workspace mapping allows the teleoperation from a single haptic device with bounded workspace of a complex robot with unbounded workspace. When the operator is reaching the position and orientation boundaries of the haptic workspace, linear and angular velocity components are respectively added to the inputs of the mobile manipulator and the flying camera. A user study on a virtual environment has been conducted to evaluate the performance and the workload on the user before and after proper training. Analysis on the data shows that the system complexity is not an obstacle for an efficient performance. This is a first step towards the implementation of a teleoperation system with a real mobile manipulator and a low-cost quadrotor as the free-flying camera.Accepted versio

The use of modern tools for modelling and simulation of UAV with Haptic

Unmanned Aerial Vehicle (UAV) is a research field in robotics which is in high demand in recent years, although there still exist many unanswered questions. In contrast, to the human operated aerial vehicles, it is still far less used to the fact that people are dubious about flying in or flying an unmanned vehicle. It is all about giving the control right to the computer (which is the Artificial Intelligence) for making decisions based on the situation like human do but this has not been easy to make people understand that it’s safe and to continue the enhancement on it. These days there are many types of UAVs available in the market for consumer use, for applications like photography to play games, to map routes, to monitor buildings, for security purposes and much more. Plus, these UAVs are also being widely used by the military for surveillance and for security reasons. One of the most commonly used consumer product is a quadcopter or quadrotor. The research carried out used modern tools (i.e., SolidWorks, Java Net Beans and MATLAB/Simulink) to model controls system for Quadcopter UAV with haptic control system to control the quadcopter in a virtual simulation environment and in real time environment. A mathematical model for the controlling the quadcopter in simulations and real time environments were introduced. Where, the design methodology for the quadcopter was defined. This methodology was then enhanced to develop a virtual simulation and real time environments for simulations and experiments. Furthermore, the haptic control was then implemented with designed control system to control the quadcopter in virtual simulation and real time experiments. By using the mathematical model of quadcopter, PID & PD control techniques were used to model the control setup for the quadcopter altitude and motion controls as work progressed. Firstly, the dynamic model is developed using a simple set of equations which evolves further by using complex control & mathematical model with precise function of actuators and aerodynamic coefficients Figure5-7. The presented results are satisfying and shows that flight experiments and simulations of the quadcopter control using haptics is a novel area of research which helps perform operations more successfully and give more control to the operator when operating in difficult environments. By using haptic accidents can be minimised and the functional performance of the operator and the UAV will be significantly enhanced. This concept and area of research of haptic control can be further developed accordingly to the needs of specific applications

Application of Simultaneous Localization and Mapping Algorithms for Haptic Teleoperation of Aerial Vehicles

In this thesis, a new type of haptic teleoperator system for remote control of Unmanned Aerial Vehicles (UAVs) has been developed, where the Simultaneous Localization and Mapping (SLAM) algorithms are implemented for the purpose of generating the haptic feedback. Specifically, the haptic feedback is provided to the human operator through interaction with artificial potential field built around the obstacles in the virtual environment which is located at the master site of the teleoperator system. The obstacles in the virtual environment replicate essential features of the actual remote environment where the UAV executes its tasks. The state of the virtual environment is generated and updated in real time using Extended Kalman Filter SLAM algorithms based on measurements performed by the UAV in the actual remote environment. Two methods for building haptic feedback from SLAM algorithms have been developed. The basic SLAM-based haptic feedback algorithm uses fixed size potential field around the obstacles, while the robust SLAM-based haptic feedback algorithm changes the size of potential field around the obstacle depending on the amount of uncertainty in obstacle location, which is represented by the covariance estimate provided by EKF. Simulations and experimental results are presented that evaluate the performance of the proposed teleoperator system

The use of modern tools for modelling and simulation of UAV with Haptic

Unmanned Aerial Vehicle (UAV) is a research field in robotics which is in high demand in recent years, although there still exist many unanswered questions. In contrast, to the human operated aerial vehicles, it is still far less used to the fact that people are dubious about flying in or flying an unmanned vehicle. It is all about giving the control right to the computer (which is the Artificial Intelligence) for making decisions based on the situation like human do but this has not been easy to make people understand that it’s safe and to continue the enhancement on it. These days there are many types of UAVs available in the market for consumer use, for applications like photography to play games, to map routes, to monitor buildings, for security purposes and much more. Plus, these UAVs are also being widely used by the military for surveillance and for security reasons. One of the most commonly used consumer product is a quadcopter or quadrotor. The research carried out used modern tools (i.e., SolidWorks, Java Net Beans and MATLAB/Simulink) to model controls system for Quadcopter UAV with haptic control system to control the quadcopter in a virtual simulation environment and in real time environment. A mathematical model for the controlling the quadcopter in simulations and real time environments were introduced. Where, the design methodology for the quadcopter was defined. This methodology was then enhanced to develop a virtual simulation and real time environments for simulations and experiments. Furthermore, the haptic control was then implemented with designed control system to control the quadcopter in virtual simulation and real time experiments. By using the mathematical model of quadcopter, PID & PD control techniques were used to model the control setup for the quadcopter altitude and motion controls as work progressed. Firstly, the dynamic model is developed using a simple set of equations which evolves further by using complex control & mathematical model with precise function of actuators and aerodynamic coefficients Figure5-7. The presented results are satisfying and shows that flight experiments and simulations of the quadcopter control using haptics is a novel area of research which helps perform operations more successfully and give more control to the operator when operating in difficult environments. By using haptic accidents can be minimised and the functional performance of the operator and the UAV will be significantly enhanced. This concept and area of research of haptic control can be further developed accordingly to the needs of specific applications

Haptic Feedback Effects on Human Control of a UAV in a Remote Teleoperation Flight Task

The remote manual teleoperation of an unmanned aerial vehicle (UAV) by a human operator creates a human-in-the loop system that is of great concern. In a remote teleoperation task, a human pilot must make control decisions based upon sensory information provided by the governed system. Often, this information consists of limited visual feedback provided by onboard cameras that do not provide an operator with an accurate portrayal of their immediate surroundings compromising the safety of the mobile robot. Due to this shortfall, haptic force feedback is often provided to the human in an effort to increase their perceptual awareness of the surrounding world. To investigate the effects of this additional sensory information provided to the human op-erator, we consider two haptic force feedback strategies. They were designed to provide either an attractive force to influence control behavior towards a reference trajectory along a flight path, or a repulsive force directing operators away from obstacles to prevent collision. Subject tests were con-ducted where human operators manually operated a remote UAV through a corridor environment under the conditions of the two strategies. For comparison, the conditions of no haptic feedback and the liner combination of both attractive and repulsive strategies were included in the study. Experi-mental results dictate that haptic force feedback in general (including both attractive and repulsive force feedback) improves the average distance from surrounding obstacles up to 21%. Further statis-tical comparison of repulsive and attractive feedback modalities reveal that even though a repulsive strategy is based directly on obstacles, an attractive strategy towards a reference trajectory is more suitable across all performance metrics. To further examine the effects of haptic aides in a UAV teleoperation task, the behavior of the human system as part of the control loop was also investigated. Through a novel device placed on the end effector of the haptic device, human-haptic interaction forces were captured and further analyzed. With this information, system identification techniques were carried out to determine the plausibility of deriving a human control model for the system. By defining lateral motion as a one-dimensional compensatory tracking task the results show that general human control behavior can be identified where lead compensation in invoked to counteract second-order UAV dynamics

The Effects of Remotely Piloted Aircraft Command and Control Latency during Within-Visual-Range Air-To-Air Combat

The type of military missions conducted by remotely piloted aircraft continues to expand into all facets of operations including air-to-air combat. While future within-visual-range air-to-air combat will be piloted by artificial intelligence, remotely piloted aircraft will likely first see combat. The purpose of this study was to quantify the effect of latency on one-versus-one, within-visual-range air-to-air combat success during both high-speed and low-speed engagements. The research employed a repeated-measures experimental design to test the various hypothesis associated with command and control latency. Participants experienced in air-to-air combat were subjected to various latency inputs during one-versus-one simulated combat using a virtual-reality simulator and scored on the combat success of each engagement. This research was pursued in coordination with the Air Force Research Laboratory and the United States Air Force Warfare Center. The dependent variable, combat score, was derived through post-simulation analysis and scored for each engagement. The independent variables included the input control latency (time) and the starting velocity of the engagement (high-speed and low-speed). The input latency included six different delays (0.0, 0.25, 0.50, 0.75, 1.0, and 1.25 seconds) between pilot input and simulator response. Each latency was repeated for a high-speed and low-speed engagement. A two-way repeated-measures analysis of variance was used to determine whether there was a statistically significant difference in means between the various treatments on combat success and determine if there was an interaction between latency and fight speed. The results indicated that there was a statistically significant difference between combat success at the various latency levels and engagement velocity. There was a significant interaction effect between latency and engagement speed, indicating that the outcome was dependent on both variables. As the latency increased, a significant decrease in combat success occurred, decreasing from .539 with no latency, to .133 at 1.250 seconds of latency during high-speed combat. During low-speed combat, the combat success decreased from .659 with no latency, to .189 at 1.250 seconds of latency. The largest incremental decrease occurred between 1.00 and 1.25 seconds of latency for high-speed and between 0.75 and 1.00 at low-speed. The overall decrease in combat success during a high-speed engagement was less than during the low-speed engagements. The results of this study quantified the decrease in combat success during within-visual range air-to-air combat and concluded that, when latency is encountered, a high-speed (two-circle) engagement is desired to minimize adverse latency effects. The research informs aircraft and communication designers of the decrease in expected combat success caused by latency. This simulation configuration can be utilized for future research leading to methods and tactics to decrease the effects of latency

Teams organization and performance analysis in autonomous human-robot teams

This paper proposes a theory of human control of robot teams based on considering how people coordinate across different task allocations. Our current work focuses on domains such as foraging in which robots perform largely independent tasks. The present study addresses the interaction between automation and organization of human teams in controlling large robot teams performing an Urban Search and Rescue (USAR) task. We identify three subtasks: perceptual search-visual search for victims, assistance-teleoperation to assist robot, and navigation-path planning and coordination. For the studies reported here, navigation was selected for automation because it involves weak dependencies among robots making it more complex and because it was shown in an earlier experiment to be the most difficult. This paper reports an extended analysis of the two conditions from a larger four condition study. In these two "shared pool" conditions Twenty four simulated robots were controlled by teams of 2 participants. Sixty paid participants (30 teams) were recruited to perform the shared pool tasks in which participants shared control of the 24 UGVs and viewed the same screens. Groups in the manual control condition issued waypoints to navigate their robots. In the autonomy condition robots generated their own waypoints using distributed path planning. We identify three self-organizing team strategies in the shared pool condition: joint control operators share full authority over robots, mixed control in which one operator takes primary control while the other acts as an assistant, and split control in which operators divide the robots with each controlling a sub-team. Automating path planning improved system performance. Effects of team organization favored operator teams who shared authority for the pool of robots. © 2010 ACM

Haptic Teleoperation of UAV Equipped with Gamma-Ray Spectrometer for Detection and Identification of Radio-Active Materials in Industrial Plants

Large scale factories such as steel, wood, construction, recycling plants and landfills involve the procurement of raw material which may include radiating parts, that must be monitored, because potentially dangerous for workers. Manufacturing operations are carried out in unstructured environments, where fully autonomous unmanned aerial vehicle (UAV) inspection is hardly applicable. In this work we report on the development of a haptic teleoperated UAV for localization of radiation sources in industrial plants. Radiation sources can be localized and identified thanks to a novel CZT-based custom gamma-ray detector integrated on the UAV, providing light, compact, spectroscopic, and low power operation. UAV operation with a human in the loop allows an expert operator to focus on selected candidate areas, thereby optimizing short flight mission in face of the constrained acquisition times required by nuclear inspection. To cope with the reduced situational awareness of the remote operator, force feedback is exploited as an additional sensory channel. The developed prototype has been demonstrated both in relevant and operational environments