Towards Sensor Enhanced Virtual Reality Teleoperation in a Dynamic Environment

A teleoperation interface is introduced featuring an integrated virtual reality based simulation augmented by sensors and image processing capabilities on-board the remotely operated vehicle. The virtual reality system addresses the typical limitations of video-based teleoperation caused by signal lag and limited field of view, allowing the operator to navigate in a continuous fashion. The vehicle incorporates an on-board computer and a stereo vision system to facilitate obstacle detection. It also enables temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle’s teleoperated state. Finally, the system provides real time update to the virtual environment based on anomalies encountered by the vehicle. System architecture and preliminary implementation results are discussed, and future work focused on incorporating dynamic moving objects in the environment is described

Sensor Augmented Virtual Reality Based Teleoperation Using Mixed Autonomy

A multimodal teleoperation interface is introduced, featuring an integrated virtual reality (VR) based simulation augmented by sensors and image processing capabilities onboard the remotely operated vehicle. The proposed virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multimodal control interface. VR addresses the typical limitations of video based teleoperation caused by signal lag and limited field of view, allowing the operator to navigate in a continuous fashion. The vehicle incorporates an onboard computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and a real-state tracking system enable temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle’s teleoperated state. The system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The VR based multimodal teleoperation interface is expected to be more adaptable and intuitive when compared with other interfaces

Effects of Visual Interaction Methods on Simulated Unmanned Aircraft Operator Situational Awareness

The limited field of view of static egocentric visual displays employed in unmanned aircraft controls introduces the soda straw effect on operators, which significantly affects their ability to capture and maintain situational awareness by not depicting peripheral visual data. The problem with insufficient operator situational awareness is the resulting increased potential for error and oversight during operation of unmanned aircraft, leading to accidents and mishaps costing United States taxpayers between $4 million to$54 million per year. The purpose of this quantitative experimental completely randomized design study was to examine and compare use of dynamic eyepoint to static visual interaction in a simulated stationary egocentric environment to determine which, if any, resulted in higher situational awareness. The theoretical framework for the study established the premise that the amount of visual information available could affect the situational awareness of an operator and that increasing visual information through dynamic eyepoint manipulation may result in higher situational awareness than static visualization. Four experimental dynamic visual interaction methods were examined (analog joystick, head tracker, uninterrupted hat/point of view switch, and incremental hat/point of view switch) and compared to a single static method (the control treatment). The five methods were used in experimental testing with 150 participants to determine if the use of a dynamic eyepoint significantly increased the situational awareness of a user within a stationary egocentric environment, indicating that employing dynamic control would reduce the occurrence or consequences of the soda straw effect. The primary difference between the four dynamic visual interaction methods was their unique manipulation approaches to control the pitch and yaw of the simulated eyepoint. The identification of dynamic visual interaction increasing user SA may lead to the further refinement of human-machine-interface (HMI), teleoperation, and unmanned aircraft control principles, with the pursuit and performance of related research

Virtual reality based multi-modal teleoperation using mixed autonomy

The thesis presents a multi modal teleoperation interface featuring an integrated virtual reality based simulation aumented by sensors and image processing capabilities onboard the remotely operated vehicle. The virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multi modal control interface. Virtual reality addresses the typical limitations of video-based teleoperation caused by signal lag and limited field of view thereby allowing the operator to navigate in a continuous fashion. The vehicle incorporates an on-board computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and real state tracking system enables temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle\u27s teleoperated state. As both the vehicle and the operator share absolute autonomy in stages, the operation is referred to as mixed autonomous. Finally, the system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The system effectively balances the autonomy between the human operator and on board vehicle intelligence. The reliability results of individual components along with overall system implementation and the results of the user study helps show that the VR based multi modal teleoperation interface is more adaptable and intuitive when compared to other interfaces