Virtual Reality to Simulate Visual Tasks for Robotic Systems

Virtual reality (VR) can be used as a tool to analyze the interactions between the visual system of a robotic agent and the environment, with the aim of designing the algorithms to solve the visual tasks necessary to properly behave into the 3D world. The novelty of our approach lies in the use of the VR as a tool to simulate the behavior of vision systems. The visual system of a robot (e.g., an autonomous vehicle, an active vision system, or a driving assistance system) and its interplay with the environment can be modeled through the geometrical relationships between the virtual stereo cameras and the virtual 3D world. Differently from conventional applications, where VR is used for the perceptual rendering of the visual information to a human observer, in the proposed approach, a virtual world is rendered to simulate the actual projections on the cameras of a robotic system. In this way, machine vision algorithms can be quantitatively validated by using the ground truth data provided by the knowledge of both the structure of the environment and the vision system

Fidelity metrics for virtual environment simulations based on spatial memory awareness states

This paper describes a methodology based on human judgments of memory awareness states for assessing the simulation fidelity of a virtual environment (VE) in relation to its real scene counterpart. To demonstrate the distinction between task performance-based approaches and additional human evaluation of cognitive awareness states, a photorealistic VE was created. Resulting scenes displayed on a headmounted display (HMD) with or without head tracking and desktop monitor were then compared to the real-world task situation they represented, investigating spatial memory after exposure. Participants described how they completed their spatial recollections by selecting one of four choices of awareness states after retrieval in an initial test and a retention test a week after exposure to the environment. These reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection and also included guesses, even if informed. Experimental results revealed variations in the distribution of participants’ awareness states across conditions while, in certain cases, task performance failed to reveal any. Experimental conditions that incorporated head tracking were not associated with visually induced recollections. Generally, simulation of task performance does not necessarily lead to simulation of the awareness states involved when completing a memory task. The general premise of this research focuses on how tasks are achieved, rather than only on what is achieved. The extent to which judgments of human memory recall, memory awareness states, and presence in the physical and VE are similar provides a fidelity metric of the simulation in question

FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s). While a vehicle is in flight in the FlightGoggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex extrinsic dynamics are generated organically through the natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest.Comment: Initial version appeared at IROS 2019. Supplementary material can be found at https://flightgoggles.mit.edu. Revision includes description of new FlightGoggles features, such as a photogrammetric model of the MIT Stata Center, new rendering settings, and a Python AP

Stereo Viewing and Virtual Reality Technologies in Mobile Robot Teleguide

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.” DOI: 10.1109/TRO.2009.2028765The use of 3-D stereoscopic visualization may provide a user with higher comprehension of remote environments in teleoperation when compared with 2-D viewing, in particular, a higher perception of environment depth characteristics, spatial localization, remote ambient layout, faster system learning, and decision performance. Works in the paper have demonstrated how stereo vision contributes to the improvement of the perception of some depth cues, often for abstract tasks, while it is hard to find works addressing stereoscopic visualization in mobile robot teleguide applications. This paper intends to contribute to this aspect by investigating the stereoscopic robot teleguide under different conditions, including typical navigation scenarios and the use of synthetic and real images. This paper also investigates how user performance may vary when employing different display technologies. Results from a set of test trials run on seven virtual reality systems, from laptop to large panorama and from head-mounted display to Cave automatic virtual environment (CAVE), emphasized few aspects that represent a base for further investigations as well as a guide when designing specific systems for telepresence.Peer reviewe

VR systems for memory assessment and depth perception

La evolución de la tecnología de Realidad Virtual (RV) ha contribuido en todos los campos, incluyendo la psicología. Esta evolución implica mejoras tanto en hardware como en software, que permiten experiencias más inmersivas. En un entorno de RV los usuarios pueden percibir la sensación de "presencia" y sentirse "inmersos". Estas sensaciones son posibles utilizando HMDs. Hoy en día, el desarrollo de los HMDs se ha centrado en mejorar sus características técnicas para ofrecer inmersión total. En psicología, los entornos de RV son una herramienta de investigación. Hay algunas aplicaciones para evaluar la memoria espacial que utilizan métodos básicos de interacción. Sin embargo, sistemas de RV que incorporen estereoscopía y movimiento físico todavía no se han explotado en psicología. En esta tesis, se ha desarrollado un nuevo sistema de RV que combina características inmersivas, interactivas y de movimiento. El sistema de RV (tarea en un laberinto virtual) se ha utilizado para evaluar la memoria espacial y la percepción de profundidad. Se han integrado dos tipos diferentes de interacción: una basada en locomoción que consistió en pedalear en una bicicleta fija (condición1) y otra estacionaria usando un gamepad (condición2). El sistema integró dos tipos de visualización: 1) Oculus Rift (OR); 2) Una gran pantalla estéreo. Se diseñaron dos estudios. El primer estudio (N=89) evaluó la memoria espacial a corto plazo usando el OR y los dos tipos de interacción. Los resultados indican que existían diferencias significativas entre ambas condiciones. Los participantes que utilizaron la condición2 obtuvieron mejor rendimiento que los que utilizaron la tarea en la condición1. Sin embargo, no se encontraron diferencias significativas en las puntuaciones de satisfacción e interacción entre ambas condiciones. El desempeño en la tarea correlacionó con el desempeño en las pruebas neuropsicológicas clásicas, revelando la verosimilitud entre ellas. El segundo estudio (N=59) incluyó participantes con y sin estereopsis. Este estudio evaluó la percepción de profundidad comparando los dos sistemas de visualización. Los participantes realizaron la tarea usando la condición2. Los resultados mostraron que las diferentes características del sistema de visualización no influyeron en el rendimiento en la tarea entre los participantes con y sin estereopsis. Se encontraron diferencias significativas a favor del HMD entre las dos condiciones y entre los dos grupos de participantes respecto a la percepción de profundidad. Los participantes que no tenían estereopsis y no podían percibir la profundidad cuando utilizaban otros sistemas de visualización, tuvieron la ilusión de percepción de profundidad cuando utilizaron el OR. El estudio sugiere que para las personas que no tienen estereopsis, el seguimiento de la cabeza influye en gran medida en la experiencia 3D. Los resultados estadísticos de ambos estudios han demostrado que el sistema de RV desarrollado es una herramienta apropiada para evaluar la memoria espacial a corto plazo y la percepción de profundidad. Por lo tanto, los sistemas de RV que combinan inmersión total, interacción y movimiento pueden ser una herramienta útil para la evaluación de procesos cognitivos humanos como la memoria. De estos estudios se han extraído las siguientes conclusiones generales: 1) La tecnología de RV y la inmersión proporcionada por los actuales HMDs son herramientas adecuadas para aplicaciones psicológicas, en particular, la evaluación de la memoria espacial a corto plazo; 2) Un sistema de RV como el presentado podría ser utilizado como herramienta para evaluar o entrenar adultos en habilidades relacionadas con la memoria espacial a corto plazo; 3) Los dos tipos de interacción utilizados para la navegación en el laberinto virtual podrían ser útiles para su uso con diferentes colectivos; 4) El OR permite que los usuarios sin estereopsis puedan percibir lThe evolution of Virtual Reality (VR) technology has contributed in all fields, including psychology. This evolution involves improvements in hardware and software allowing more immersive experiences. In a VR environment users can perceive the sensation of "presence" and feel "immersed". These sensations are possible using VR devices as HMDs. Nowadays, the development of the HMDs has focused on improving their technical features to offer full immersion. In psychology, VR environments are research tools because they allow the use of new paradigms that are not possible to employ in a real environment. There are some applications for assessing spatial memory that use basic methods of HCI. However, VR systems that incorporate stereoscopy and physical movement have not yet been exploited in psychology. In this thesis, a novel VR system combining immersive, interactive and motion features was developed. This system was used for the assessment of the spatial memory and the evaluation of depth perception. For this system, a virtual maze task was designed and implemented. In this system, two different types of interaction were integrated: a locomotion-based interaction pedaling a fixed bicycle (condition1), and a stationary interaction using a gamepad (condition2). This system integrated two types of display systems: 1) The Oculus Rift; 2) A large stereo screen. Two studies were designed to determine the efficacy of the VR system using physical movement and immersion. The first study (N=89) assessed the spatial short term memory using the Oculus Rift and the two types of interaction The results showed that there were statistically significant differences between both conditions. The participants who performed the condition2 got better performance than participants who performed the condition1. However, there were no statistically significant differences in satisfaction and interaction scores between both conditions. The performance on the task correlated with the performance on other classical neuropsychological tests, revealing a verisimilitude between them. The second study (N=59) involved participants who had and who had not stereopsis. This study assessed the depth perception by comparing the two display systems. The participants performed the task using the condition2. The results showed that the different features of the display system did not influence the performance on the task between the participants with and without stereopsis. Statistically significant differences were found in favor of the HMD between the two conditions and between the two groups of participants regard to depth perception. The participants who did not have stereopsis and could not perceive the depth when they used other display systems (e.g. CAVE); however, they had the illusion of depth perception when they used the Oculus Rift. The study suggests that for the people who did not have stereopsis, the head tracking largely influences the 3D experience. The statistical results of both studies have proven that the VR system developed for this research is an appropriate tool to assess the spatial short-term memory and the depth perception. Therefore, the VR systems that combine full immersion, interaction and movement can be a helpful tool for the assessment of human cognitive processes as the memory. General conclusions from these studies are: 1) The VR technology and immersion provided by current HMDs are appropriate tools for psychological applications, in particular, the assessment of spatial short-term memory; 2) A VR system like the one presented in this thesis could be used as a tool to assess or train adults in skills related to spatial short-term memory; 3) The two types of interaction (condition1 and condition2) used for navigation within the virtual maze could be helpful to use with different collectives; 4) The Oculus Rift allows that the users without stereopsis can perceive the depth perception of 3D objects and have rich 3D experiences.L'evolució de la tecnologia de Realitat Virtual (RV) ha contribuït en tots els camps, incloent la psicologia. Aquesta evolució implica millores en el maquinari i el programari que permeten experiències més immersives. En un entorn de RV, els usuaris poden percebre la sensació de "presència" i sentir-se "immersos". Aquestes sensacions són possibles utilitzant HMDs. Avui dia, el desenvolupament dels HMDs s'ha centrat a millorar les seves característiques tècniques per oferir immersió plena. En la psicologia, els entorns de RV són eines de recerca. Hi ha algunes aplicacions per avaluar la memòria espacial que utilitzen mètodes bàsics d'interacció. Tanmateix, sistemes de RV que incorporen estereoscòpia i moviment físic no s'han explotat en psicologia. En aquesta tesi, s'ha desenvolupat un sistema de RV novell que combina immersió, interacció i moviment. El sistema (tasca en un laberint virtual) s'ha utilitzat per a l'avaluació de la memòria espacial i la percepció de profunditat. S'han integrat dos tipus d'interacció: una interacció basada en locomoció pedalejant una bicicleta fixa (condició1), i l'altra una interacció estacionària usant un gamepad (condició2). S'han integrat dos tipus de sistemes de pantalla: 1) L'Oculus Rift; 2) Una gran pantalla estereoscòpica. Dos estudis van ser dissenyats. El primer estudi (N=89) va avaluar la memòria a curt termini i espacial utilitzant l'Oculus Rift i els dos tipus d'interacció. Els resultats indiquen que hi havia diferències significatives entre les dues condicions. Els participants que van utilitzar la condició2 van obtenir millor rendiment que els participants que van utilitzar la condició1. Tanmateix, no hi havia diferències significatives dins satisfacció i puntuacions d'interacció entre les dues condicions. El rendiment de la tasca va correlacionar amb el rendiment en les proves neuropsicològiques clàssiques, revelant versemblança entre elles. El segon estudi (N=59) va implicar participants que van tenir i que van haver-hi no estereopsis. Aquest estudi va avaluar la percepció de profunditat comparant els dos sistemes de pantalla. Els participants realitzen la tasca utilitzant la condició2. Els resultats van mostrar que les diferents característiques del sistema de pantalla no va influir en el rendiment en la tasca entre els participants qui tenien i els qui no tenien estereopsis. Diferències significatives van ser trobades a favor del HMD entre les dues condicions i entre els dos grups de participants. Els participants que no van tenir estereopsis i no podien percebre la profunditat quan van utilitzar altres sistemes de pantalla (per exemple, CAVE), van tenir la il.lusió de percepció de profunditat quan van utilitzar l'Oculus Rift. L'estudi suggereix que per les persones que no van tenir estereopsis, el seguiment del cap influeix en gran mesura en l'experiència 3D. Els resultats estadístics dels dos estudis han provat que el sistema de RV desenvolupat per aquesta recerca és una eina apropiada per avaluar la memòria espacial a curt termini i la percepció de profunditat. Per això, els sistemes de RV que combinen immersió plena, interacció i moviment poden ser una eina útil per la avaluació de processos cognitius humans com la memòria Les conclusions generals que s'han extret d'aquests estudis, són les següents: 1) La tecnologia de RV i la immersió proporcionada pels HMDs són eines apropiades per aplicacions psicològiques, en particular, la avaluació de memòria espacial a curt termini; 2) Un sistema de RV com el presentat podria ser utilitzat com a eina per avaluar o entrenar adults en habilitats relacionades amb la memòria espacial a curt termini; 3) Els dos tipus d'interacció utilitzats per navegació dins del laberint virtual podrien ser útils per al seu ús amb diferent col.lectius; 3) L'Oculus Rift permet que els usuaris que no tenen estereopsis puguen percebre la percepció de profunditat dels objectes 3D i tenirCárdenas Delgado, SE. (2017). VR systems for memory assessment and depth perception [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/94629TESI

Contributions to virtual reality

153 p.The thesis contributes in three Virtual Reality areas: ¿ Visual perception: a calibration algorithm is proposed to estimate stereo projection parameters in head-mounted displays, so that correct shapes and distances can be perceived, and calibration and control procedures are proposed to obtain desired accommodation stimuli at different virtual distances.¿ Immersive scenarios: the thesis analyzes several use cases demanding varying degrees of immersion and special, innovative visualization solutions are proposed to fulfil their requirements. Contributions focus on machinery simulators, weather radar volumetric visualization and manual arc welding simulation.¿ Ubiquitous visualization: contributions are presented to scenarios where users access interactive 3D applications remotely. The thesis follows the evolution of Web3D standards and technologies to propose original visualization solutions for volume rendering of weather radar data, e-learning on energy efficiency, virtual e-commerce and visual product configurators