Modeling the Human Visuo-Motor System for Remote-Control Operation

University of Minnesota Ph.D. dissertation. 2018. Major: Computer Science. Advisors: Nikolaos Papanikolopoulos, Berenice Mettler. 1 computer file (PDF); 172 pages.Successful operation of a teleoperated miniature rotorcraft relies on capabilities including guidance, trajectory following, feedback control, and environmental perception. For many operating scenarios fragile automation systems are unable to provide adequate performance. In contrast, human-in-the-loop systems demonstrate an ability to adapt to changing and complex environments, stability in control response, high level goal selection and planning, and the ability to perceive and process large amounts of information. Modeling the perceptual processes of the human operator provides the foundation necessary for a systems based approach to the design of control and display systems used by remotely operated vehicles. In this work we consider flight tasks for remotely controlled miniature rotorcraft operating in indoor environments. Operation of agile robotic systems in three dimensional spaces requires a detailed understanding of the perceptual aspects of the problem as well as knowledge of the task and models of the operator response. When modeling the human-in-the-loop the dynamics of the vehicle, environment, and human perception-action are tightly coupled in space and time. The dynamic response of the overall system emerges from the interplay of perception and action. The main questions to be answered in this work are: i) what approach does the human operator implement when generating a control and guidance response? ii) how is information about the vehicle and environment extracted by the human? iii) can the gaze patterns of the pilot be decoded to provide information for estimation and control? In relation to existing research this work differs by focusing on fast acting dynamic systems in multiple dimensions and investigating how the gaze can be exploited to provide action-relevant information. To study human-in-the-loop systems the development and integration of the experimental infrastructure is described. Utilizing the infrastructure, a theoretical framework for computational modeling of the human pilot’s perception-action is proposed and verified experimentally. The benefits of the human visuo-motor model are demonstrated through application examples where the perceptual and control functions of a teleoperation system are augmented to reduce workload and provide a more natural human-machine interface

Objective performance metrics for improved space telerobotics training

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 47-50).NASA astronauts undergo many hours of formal training and self-study to gain proficiency in space teleoperation tasks. After each lesson, instructors score an astronaut's performance in several broad skill categories, including 'General Situational Awareness', 'Maneuvers/Task Performance', and 'Hand- Controller Techniques'. A plus, check, or minus indicates that the student is ahead of, at, or behind the expected skill level. The scoring of the final evaluation for a robotics training course is also largely subjective, with the instructor designating an integer score for the student between 1 (Unsatisfactory) and 5 (Strong) in the same skill categories. This thesis research project was designed to: (1) consider the variety of quantitative metrics that could be embedded into a space robotics training simulation, and (2) investigate at what point and by what means it is most constructive for performance assessment to be revealed to an operator-in-training. We reviewed the current largely qualitative space robotics performance metrics, as well as new quantitative kinematic metrics of manual control skills-including those explored thus far only in laboratory experiments-and additional measures of executive function and supervisory control performance. Kinematic metrics include quantitative measures such as rate of change of linear and rotational acceleration. Potential measures of executive function and supervisory control include camera selection and clearance monitoring. To instantiate our ideas, we chose a specific "fly-to" space telerobotics task taught in the early phases of NASA Generic Robotics Training (GRT) and developed a pilot training experiment (n=16) using our virtual robotics training workstation. Our goal was to evaluate potential performance metrics designed to encourage use of multi-axis control, and to compare real-time ("live") performance feedback alternatives (live visual vs. live aural vs. none). Movement time decreased and multi-axis and bimanual control use gradually increased across trials. All subjects had the opportunity to view post-trial performance feedback including these metrics. Although our subjects overwhelmingly preferred the live, visual feedback condition, no reliable additional effects of live feedback condition were found, except perhaps among the more experienced subjects. However, the experiment demonstrated that embedded performance metrics potentially could quantify and improve some important aspects of GRT evaluations.Supported by the National Space Biomedical Research Institute through NASA NCC9-58by Rachel Emily Forman.S.M

Mars Rotorcraft: Possibilities, Limitations, and Implications For Human/Robotic Exploration

Several research investigations have examined the challenges and opportunities in the use of small robotic rotorcraft for the exploration of Mars. To date, only vehicles smaller than 150 kg have been studied. This paper proposes to examine the question of maximum Mars rotorcraft size, range, and payload/cargo capacity. Implications for the issue of whether or not (from an extreme design standpoint) a manned Mars rotorcraft is viable are also discussed

Aerospace medicine and biology: A continuing bibliography with indexes (supplement 315)

This bibliography lists 211 reports, articles and other documents introduced into the NASA scientific and technical information system in September, 1988

Aeronautical engineering: A continuing bibliography with indexes (supplement 268)

This bibliography lists 406 reports, articles, and other documents introduced into the NASA scientific and technical information system in July, 1991. Subject coverage includes: design, construction and testing of aircraft and aircraft engines; aircraft components, equipment and systems; ground support systems; and theoretical and applied aspects of aerodynamics and general fluid dynamics

Aerospace medicine and biology: A continuing bibliography with indexes (supplement 391)

This bibliography lists 75 reports, articles, and other documents introduced into the NASA Scientific and Technical Information System during Aug. 1994. Subject coverage includes: aerospace medicine and physiology, life support systems and man/system technology, protective clothing, exobiology and extraterrestrial life, planetary biology, and flight crew behavior and performance

Mixed-reality for unmanned aerial vehicle operations in near earth environments

Future applications will bring unmanned aerial vehicles (UAVs) to near Earth environments such as urban areas, causing a change in the way UAVs are currently operated. Of concern is that UAV accidents still occur at a much higher rate than the accident rate for commercial airliners. A number of these accidents can be attributed to a UAV pilot's low situation awareness (SA) due to the limitations of UAV operating interfaces. The main limitation is the physical separation between the vehicle and the pilot. This eliminates any motion and exteroceptive sensory feedback to the pilot. These limitation on top of a small eld of view from the onboard camera results in low SA, making near Earth operations di cult and dangerous. Autonomy has been proposed as a solution for near Earth tasks but state of the art arti cial intelligence still requires very structured and well de ned goals to allow safe autonomous operations. Therefore, there is a need to better train pilots to operate UAVs in near Earth environments and to augment their performance for increased safety and minimization of accidents.In this work, simulation software, motion platform technology, and UAV sensor suites were integrated to produce mixed-reality systems that address current limitations of UAV piloting interfaces. The mixed reality de nition is extended in this work to encompass not only the visual aspects but to also include a motion aspect. A training and evaluation system for UAV operations in near Earth environments was developed. Modi cations were made to ight simulator software to recreate current UAV operating modalities (internal and external). The training and evaluation system has been combined with Drexel's Sensor Integrated Systems Test Rig (SISTR) to allow simulated missions while incorporating real world environmental e ects andUAV sensor hardware.To address the lack of motion feedback to a UAV pilot, a system was developed that integrates a motion simulator into UAV operations. The system is designed such that during ight, the angular rate of a UAV is captured by an onboard inertial measurement unit (IMU) and is relayed to a pilot controlling the vehicle from inside the motion simulator.Efforts to further increase pilot SA led to the development of a mixed reality chase view piloting interface. Chase view is similar to a view of being towed behind the aircraft. It combines real world onboard camera images with a virtual representation of the vehicle and the surrounding operating environment. A series of UAV piloting experiments were performed using the training and evaluation systems described earlier. Subjects' behavioral performance while using the onboard camera view and the mixed reality chase view interface during missions was analyzed. Subjects' cognitive workload during missions was also assessed using subjective measures such as NASA task load index and non-subjective brain activity measurements using a functional Infrared Spectroscopy (fNIR) system. Behavioral analysis showed that the chase view interface improved pilot performance in near Earth ights and increased their situational awareness. fNIR analysis showed that a subjects cognitive workload was signi cantly less while using the chase view interface. Real world ight tests were conducted in a near Earth environment with buildings and obstacles to evaluate the chase view interface with real world data. The interface performed very well with real world, real time data in close range scenarios.The mixed reality approaches presented follow studies on human factors performance and cognitive loading. The resulting designs serve as test beds for studying UAV pilot performance, creating training programs, and developing tools to augment UAV operations and minimize UAV accidents during operations in near Earth environments.Ph.D., Mechanical Engineering -- Drexel University, 201

Haptic Feedback Effects on Human Control of a UAV in a Remote Teleoperation Flight Task

The remote manual teleoperation of an unmanned aerial vehicle (UAV) by a human operator creates a human-in-the loop system that is of great concern. In a remote teleoperation task, a human pilot must make control decisions based upon sensory information provided by the governed system. Often, this information consists of limited visual feedback provided by onboard cameras that do not provide an operator with an accurate portrayal of their immediate surroundings compromising the safety of the mobile robot. Due to this shortfall, haptic force feedback is often provided to the human in an effort to increase their perceptual awareness of the surrounding world. To investigate the effects of this additional sensory information provided to the human op-erator, we consider two haptic force feedback strategies. They were designed to provide either an attractive force to influence control behavior towards a reference trajectory along a flight path, or a repulsive force directing operators away from obstacles to prevent collision. Subject tests were con-ducted where human operators manually operated a remote UAV through a corridor environment under the conditions of the two strategies. For comparison, the conditions of no haptic feedback and the liner combination of both attractive and repulsive strategies were included in the study. Experi-mental results dictate that haptic force feedback in general (including both attractive and repulsive force feedback) improves the average distance from surrounding obstacles up to 21%. Further statis-tical comparison of repulsive and attractive feedback modalities reveal that even though a repulsive strategy is based directly on obstacles, an attractive strategy towards a reference trajectory is more suitable across all performance metrics. To further examine the effects of haptic aides in a UAV teleoperation task, the behavior of the human system as part of the control loop was also investigated. Through a novel device placed on the end effector of the haptic device, human-haptic interaction forces were captured and further analyzed. With this information, system identification techniques were carried out to determine the plausibility of deriving a human control model for the system. By defining lateral motion as a one-dimensional compensatory tracking task the results show that general human control behavior can be identified where lead compensation in invoked to counteract second-order UAV dynamics

The use of modern tools for modelling and simulation of UAV with Haptic

Unmanned Aerial Vehicle (UAV) is a research field in robotics which is in high demand in recent years, although there still exist many unanswered questions. In contrast, to the human operated aerial vehicles, it is still far less used to the fact that people are dubious about flying in or flying an unmanned vehicle. It is all about giving the control right to the computer (which is the Artificial Intelligence) for making decisions based on the situation like human do but this has not been easy to make people understand that it’s safe and to continue the enhancement on it. These days there are many types of UAVs available in the market for consumer use, for applications like photography to play games, to map routes, to monitor buildings, for security purposes and much more. Plus, these UAVs are also being widely used by the military for surveillance and for security reasons. One of the most commonly used consumer product is a quadcopter or quadrotor. The research carried out used modern tools (i.e., SolidWorks, Java Net Beans and MATLAB/Simulink) to model controls system for Quadcopter UAV with haptic control system to control the quadcopter in a virtual simulation environment and in real time environment. A mathematical model for the controlling the quadcopter in simulations and real time environments were introduced. Where, the design methodology for the quadcopter was defined. This methodology was then enhanced to develop a virtual simulation and real time environments for simulations and experiments. Furthermore, the haptic control was then implemented with designed control system to control the quadcopter in virtual simulation and real time experiments. By using the mathematical model of quadcopter, PID & PD control techniques were used to model the control setup for the quadcopter altitude and motion controls as work progressed. Firstly, the dynamic model is developed using a simple set of equations which evolves further by using complex control & mathematical model with precise function of actuators and aerodynamic coefficients Figure5-7. The presented results are satisfying and shows that flight experiments and simulations of the quadcopter control using haptics is a novel area of research which helps perform operations more successfully and give more control to the operator when operating in difficult environments. By using haptic accidents can be minimised and the functional performance of the operator and the UAV will be significantly enhanced. This concept and area of research of haptic control can be further developed accordingly to the needs of specific applications

Aerospace medicine and biology: A continuing bibliography with indexes (supplement 333)

This bibliography lists 122 reports, articles and other documents introduced into the NASA Scientific and Technical Information System during January, 1990. Subject coverage includes: aerospace medicine and psychology, life support systems and controlled environments, safety equipment, exobiology and extraterrestrial life, and flight crew behavior and performance