Contributions to virtual reality

153 p.The thesis contributes in three Virtual Reality areas: ¿ Visual perception: a calibration algorithm is proposed to estimate stereo projection parameters in head-mounted displays, so that correct shapes and distances can be perceived, and calibration and control procedures are proposed to obtain desired accommodation stimuli at different virtual distances.¿ Immersive scenarios: the thesis analyzes several use cases demanding varying degrees of immersion and special, innovative visualization solutions are proposed to fulfil their requirements. Contributions focus on machinery simulators, weather radar volumetric visualization and manual arc welding simulation.¿ Ubiquitous visualization: contributions are presented to scenarios where users access interactive 3D applications remotely. The thesis follows the evolution of Web3D standards and technologies to propose original visualization solutions for volume rendering of weather radar data, e-learning on energy efficiency, virtual e-commerce and visual product configurators

3D-Stereoscopic Immersive Analytics Projects at Monash University and University of Konstanz

Immersive Analytics investigates how novel interaction and display technologies may support analytical reasoning and decision making. The Immersive Analytics initiative of Monash University started early 2014. Over the last few years, a number of projects have been developed or extended in this context to meet the requirements of semi- or full-immersive stereoscopic environments. Different technologies are used for this purpose: CAVE2™ (a 330 degree large-scale visualization environment which can be used for educative and scientific group presentations, analyses and discussions), stereoscopic Powerwalls (miniCAVEs, representing a segment of the CAVE2 and used for development and communication), Fishtanks, and/or HMDs (such as Oculus, VIVE, and mobile HMD approaches). Apart from CAVE2™ all systems are or will be employed on both the Monash University and the University of Konstanz side, especially to investigate collaborative Immersive Analytics. In addition, sensiLab extends most of the previous approaches by involving all senses, 3D visualization is combined with multi-sensory feedback, 3D printing, robotics in a scientific-artistic-creative environment

Apple Vision Pro for Healthcare: "The Ultimate Display"? -- Entering the Wonderland of Precision Medicine

At the Worldwide Developers Conference (WWDC) in June 2023, Apple introduced the Vision Pro. The Vision Pro is a Mixed Reality (MR) headset, more specifically it is a Virtual Reality (VR) device with an additional Video See-Through (VST) capability. The VST capability turns the Vision Pro also into an Augmented Reality (AR) device. The AR feature is enabled by streaming the real world via cameras to the (VR) screens in front of the user's eyes. This is of course not unique and similar to other devices, like the Varjo XR-3. Nevertheless, the Vision Pro has some interesting features, like an inside-out screen that can show the headset wearers' eyes to "outsiders" or a button on the top, called "Digital Crown", that allows you to seamlessly blend digital content with your physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to the "Ultimate Display", which Ivan Sutherland had already sketched in 1965. Not available to the public yet, like the Ultimate Display, we want to take a look into the crystal ball in this perspective to see if it can overcome some clinical challenges that - especially - AR still faces in the medical domain, but also go beyond and discuss if the Vision Pro could support clinicians in essential tasks to spend more time with their patients.Comment: This is a Preprint under CC BY. This work was supported by NIH/NIAID R01AI172875, NIH/NCATS UL1 TR001427, the REACT-EU project KITE and enFaced 2.0 (FWF KLI 1044). B. Puladi was funded by the Medical Faculty of the RWTH Aachen University as part of the Clinician Scientist Program. C. Gsaxner was funded by the Advanced Research Opportunities Program from the RWTH Aachen Universit

Contributions to virtual reality

153 p.The thesis contributes in three Virtual Reality areas: ¿ Visual perception: a calibration algorithm is proposed to estimate stereo projection parameters in head-mounted displays, so that correct shapes and distances can be perceived, and calibration and control procedures are proposed to obtain desired accommodation stimuli at different virtual distances.¿ Immersive scenarios: the thesis analyzes several use cases demanding varying degrees of immersion and special, innovative visualization solutions are proposed to fulfil their requirements. Contributions focus on machinery simulators, weather radar volumetric visualization and manual arc welding simulation.¿ Ubiquitous visualization: contributions are presented to scenarios where users access interactive 3D applications remotely. The thesis follows the evolution of Web3D standards and technologies to propose original visualization solutions for volume rendering of weather radar data, e-learning on energy efficiency, virtual e-commerce and visual product configurators

VR systems for memory assessment and depth perception

La evolución de la tecnología de Realidad Virtual (RV) ha contribuido en todos los campos, incluyendo la psicología. Esta evolución implica mejoras tanto en hardware como en software, que permiten experiencias más inmersivas. En un entorno de RV los usuarios pueden percibir la sensación de "presencia" y sentirse "inmersos". Estas sensaciones son posibles utilizando HMDs. Hoy en día, el desarrollo de los HMDs se ha centrado en mejorar sus características técnicas para ofrecer inmersión total. En psicología, los entornos de RV son una herramienta de investigación. Hay algunas aplicaciones para evaluar la memoria espacial que utilizan métodos básicos de interacción. Sin embargo, sistemas de RV que incorporen estereoscopía y movimiento físico todavía no se han explotado en psicología. En esta tesis, se ha desarrollado un nuevo sistema de RV que combina características inmersivas, interactivas y de movimiento. El sistema de RV (tarea en un laberinto virtual) se ha utilizado para evaluar la memoria espacial y la percepción de profundidad. Se han integrado dos tipos diferentes de interacción: una basada en locomoción que consistió en pedalear en una bicicleta fija (condición1) y otra estacionaria usando un gamepad (condición2). El sistema integró dos tipos de visualización: 1) Oculus Rift (OR); 2) Una gran pantalla estéreo. Se diseñaron dos estudios. El primer estudio (N=89) evaluó la memoria espacial a corto plazo usando el OR y los dos tipos de interacción. Los resultados indican que existían diferencias significativas entre ambas condiciones. Los participantes que utilizaron la condición2 obtuvieron mejor rendimiento que los que utilizaron la tarea en la condición1. Sin embargo, no se encontraron diferencias significativas en las puntuaciones de satisfacción e interacción entre ambas condiciones. El desempeño en la tarea correlacionó con el desempeño en las pruebas neuropsicológicas clásicas, revelando la verosimilitud entre ellas. El segundo estudio (N=59) incluyó participantes con y sin estereopsis. Este estudio evaluó la percepción de profundidad comparando los dos sistemas de visualización. Los participantes realizaron la tarea usando la condición2. Los resultados mostraron que las diferentes características del sistema de visualización no influyeron en el rendimiento en la tarea entre los participantes con y sin estereopsis. Se encontraron diferencias significativas a favor del HMD entre las dos condiciones y entre los dos grupos de participantes respecto a la percepción de profundidad. Los participantes que no tenían estereopsis y no podían percibir la profundidad cuando utilizaban otros sistemas de visualización, tuvieron la ilusión de percepción de profundidad cuando utilizaron el OR. El estudio sugiere que para las personas que no tienen estereopsis, el seguimiento de la cabeza influye en gran medida en la experiencia 3D. Los resultados estadísticos de ambos estudios han demostrado que el sistema de RV desarrollado es una herramienta apropiada para evaluar la memoria espacial a corto plazo y la percepción de profundidad. Por lo tanto, los sistemas de RV que combinan inmersión total, interacción y movimiento pueden ser una herramienta útil para la evaluación de procesos cognitivos humanos como la memoria. De estos estudios se han extraído las siguientes conclusiones generales: 1) La tecnología de RV y la inmersión proporcionada por los actuales HMDs son herramientas adecuadas para aplicaciones psicológicas, en particular, la evaluación de la memoria espacial a corto plazo; 2) Un sistema de RV como el presentado podría ser utilizado como herramienta para evaluar o entrenar adultos en habilidades relacionadas con la memoria espacial a corto plazo; 3) Los dos tipos de interacción utilizados para la navegación en el laberinto virtual podrían ser útiles para su uso con diferentes colectivos; 4) El OR permite que los usuarios sin estereopsis puedan percibir lThe evolution of Virtual Reality (VR) technology has contributed in all fields, including psychology. This evolution involves improvements in hardware and software allowing more immersive experiences. In a VR environment users can perceive the sensation of "presence" and feel "immersed". These sensations are possible using VR devices as HMDs. Nowadays, the development of the HMDs has focused on improving their technical features to offer full immersion. In psychology, VR environments are research tools because they allow the use of new paradigms that are not possible to employ in a real environment. There are some applications for assessing spatial memory that use basic methods of HCI. However, VR systems that incorporate stereoscopy and physical movement have not yet been exploited in psychology. In this thesis, a novel VR system combining immersive, interactive and motion features was developed. This system was used for the assessment of the spatial memory and the evaluation of depth perception. For this system, a virtual maze task was designed and implemented. In this system, two different types of interaction were integrated: a locomotion-based interaction pedaling a fixed bicycle (condition1), and a stationary interaction using a gamepad (condition2). This system integrated two types of display systems: 1) The Oculus Rift; 2) A large stereo screen. Two studies were designed to determine the efficacy of the VR system using physical movement and immersion. The first study (N=89) assessed the spatial short term memory using the Oculus Rift and the two types of interaction The results showed that there were statistically significant differences between both conditions. The participants who performed the condition2 got better performance than participants who performed the condition1. However, there were no statistically significant differences in satisfaction and interaction scores between both conditions. The performance on the task correlated with the performance on other classical neuropsychological tests, revealing a verisimilitude between them. The second study (N=59) involved participants who had and who had not stereopsis. This study assessed the depth perception by comparing the two display systems. The participants performed the task using the condition2. The results showed that the different features of the display system did not influence the performance on the task between the participants with and without stereopsis. Statistically significant differences were found in favor of the HMD between the two conditions and between the two groups of participants regard to depth perception. The participants who did not have stereopsis and could not perceive the depth when they used other display systems (e.g. CAVE); however, they had the illusion of depth perception when they used the Oculus Rift. The study suggests that for the people who did not have stereopsis, the head tracking largely influences the 3D experience. The statistical results of both studies have proven that the VR system developed for this research is an appropriate tool to assess the spatial short-term memory and the depth perception. Therefore, the VR systems that combine full immersion, interaction and movement can be a helpful tool for the assessment of human cognitive processes as the memory. General conclusions from these studies are: 1) The VR technology and immersion provided by current HMDs are appropriate tools for psychological applications, in particular, the assessment of spatial short-term memory; 2) A VR system like the one presented in this thesis could be used as a tool to assess or train adults in skills related to spatial short-term memory; 3) The two types of interaction (condition1 and condition2) used for navigation within the virtual maze could be helpful to use with different collectives; 4) The Oculus Rift allows that the users without stereopsis can perceive the depth perception of 3D objects and have rich 3D experiences.L'evolució de la tecnologia de Realitat Virtual (RV) ha contribuït en tots els camps, incloent la psicologia. Aquesta evolució implica millores en el maquinari i el programari que permeten experiències més immersives. En un entorn de RV, els usuaris poden percebre la sensació de "presència" i sentir-se "immersos". Aquestes sensacions són possibles utilitzant HMDs. Avui dia, el desenvolupament dels HMDs s'ha centrat a millorar les seves característiques tècniques per oferir immersió plena. En la psicologia, els entorns de RV són eines de recerca. Hi ha algunes aplicacions per avaluar la memòria espacial que utilitzen mètodes bàsics d'interacció. Tanmateix, sistemes de RV que incorporen estereoscòpia i moviment físic no s'han explotat en psicologia. En aquesta tesi, s'ha desenvolupat un sistema de RV novell que combina immersió, interacció i moviment. El sistema (tasca en un laberint virtual) s'ha utilitzat per a l'avaluació de la memòria espacial i la percepció de profunditat. S'han integrat dos tipus d'interacció: una interacció basada en locomoció pedalejant una bicicleta fixa (condició1), i l'altra una interacció estacionària usant un gamepad (condició2). S'han integrat dos tipus de sistemes de pantalla: 1) L'Oculus Rift; 2) Una gran pantalla estereoscòpica. Dos estudis van ser dissenyats. El primer estudi (N=89) va avaluar la memòria a curt termini i espacial utilitzant l'Oculus Rift i els dos tipus d'interacció. Els resultats indiquen que hi havia diferències significatives entre les dues condicions. Els participants que van utilitzar la condició2 van obtenir millor rendiment que els participants que van utilitzar la condició1. Tanmateix, no hi havia diferències significatives dins satisfacció i puntuacions d'interacció entre les dues condicions. El rendiment de la tasca va correlacionar amb el rendiment en les proves neuropsicològiques clàssiques, revelant versemblança entre elles. El segon estudi (N=59) va implicar participants que van tenir i que van haver-hi no estereopsis. Aquest estudi va avaluar la percepció de profunditat comparant els dos sistemes de pantalla. Els participants realitzen la tasca utilitzant la condició2. Els resultats van mostrar que les diferents característiques del sistema de pantalla no va influir en el rendiment en la tasca entre els participants qui tenien i els qui no tenien estereopsis. Diferències significatives van ser trobades a favor del HMD entre les dues condicions i entre els dos grups de participants. Els participants que no van tenir estereopsis i no podien percebre la profunditat quan van utilitzar altres sistemes de pantalla (per exemple, CAVE), van tenir la il.lusió de percepció de profunditat quan van utilitzar l'Oculus Rift. L'estudi suggereix que per les persones que no van tenir estereopsis, el seguiment del cap influeix en gran mesura en l'experiència 3D. Els resultats estadístics dels dos estudis han provat que el sistema de RV desenvolupat per aquesta recerca és una eina apropiada per avaluar la memòria espacial a curt termini i la percepció de profunditat. Per això, els sistemes de RV que combinen immersió plena, interacció i moviment poden ser una eina útil per la avaluació de processos cognitius humans com la memòria Les conclusions generals que s'han extret d'aquests estudis, són les següents: 1) La tecnologia de RV i la immersió proporcionada pels HMDs són eines apropiades per aplicacions psicològiques, en particular, la avaluació de memòria espacial a curt termini; 2) Un sistema de RV com el presentat podria ser utilitzat com a eina per avaluar o entrenar adults en habilitats relacionades amb la memòria espacial a curt termini; 3) Els dos tipus d'interacció utilitzats per navegació dins del laberint virtual podrien ser útils per al seu ús amb diferent col.lectius; 3) L'Oculus Rift permet que els usuaris que no tenen estereopsis puguen percebre la percepció de profunditat dels objectes 3D i tenirCárdenas Delgado, SE. (2017). VR systems for memory assessment and depth perception [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/94629TESI

Designing real-time, continuous emotion annotation techniques for 360° VR videos

With the increasing availability of head-mounted displays (HMDs) that show immersive 360° VR content, it is important to understand to what extent these immersive experiences can evoke emotions. Typically to collect emotion ground truth labels, users rate videos through post-experience self-reports that are discrete in nature. However, post-stimuli self-reports are temporally imprecise, especially after watching 360° videos. In this work, we design six continuous emotion annotation techniques for the Oculus Rift HMD aimed at minimizing workload and distraction. Based on a co-design session with six experts, we contribute HaloLight and DotSize, two continuous annotation methods deemed unobtrusive and easy to understand. We discuss the next challenges for evaluating the usability of these techniques, and reliability of continuous annotations

Naturalistic visualization of reaching movements using head-mounted displays improves movement quality compared to conventional computer screens and proves high usability.

BACKGROUND The relearning of movements after brain injury can be optimized by providing intensive, meaningful, and motivating training using virtual reality (VR). However, most current solutions use two-dimensional (2D) screens, where patients interact via symbolic representations of their limbs (e.g., a cursor). These 2D screens lack depth cues, potentially deteriorating movement quality and increasing cognitive load. Head-mounted displays (HMDs) have great potential to provide naturalistic movement visualization by incorporating improved depth cues, reduce visuospatial transformations by rendering movements in the space where they are performed, and preserve eye-hand coordination by showing an avatar-with immersive VR (IVR)-or the user's real body-with augmented reality (AR). However, elderly populations might not find these novel technologies usable, hampering potential motor and cognitive benefits. METHODS We compared movement quality, cognitive load, motivation, and system usability in twenty elderly participants (>59 years old) while performing a dual motor-cognitive task with different visualization technologies: IVR HMD, AR HMD, and a 2D screen. We evaluated participants' self-reported cognitive load, motivation, and usability using questionnaires. We also conducted a pilot study with five brain-injured patients comparing the visualization technologies while using an assistive device. RESULTS Elderly participants performed straighter, shorter duration, and smoother movements when the task was visualized with the HMDs than screen. The IVR HMD led to shorter duration movements than AR. Movement onsets were shorter with IVR than AR, and shorter for both HMDs than the screen, potentially indicating facilitated reaction times due to reduced cognitive load. No differences were found in the questionnaires regarding cognitive load, motivation, or usability between technologies in elderly participants. Both HMDs proved high usability in our small sample of patients. CONCLUSIONS HMDs are a promising technology to be incorporated into neurorehabilitation, as their more naturalistic movement visualization improves movement quality compared to conventional screens. HMDs demonstrate high usability, without decreasing participants' motivation, and might potentially lower cognitive load. Our preliminary clinical results suggest that brain-injured patients may especially benefit from more immersive technologies. However, larger patient samples are needed to draw stronger conclusions.*