D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos

Understanding human visual behavior within virtual reality environments is crucial to fully leverage their potential. While previous research has provided rich visual data from human observers, existing gaze datasets often suffer from the absence of multimodal stimuli. Moreover, no dataset has yet gathered eye gaze trajectories (i.e., scanpaths) for dynamic content with directional ambisonic sound, which is a critical aspect of sound perception by humans. To address this gap, we introduce D-SAV360, a dataset of 4,609 head and eye scanpaths for 360° videos with first-order ambisonics. This dataset enables a more comprehensive study of multimodal interaction on visual behavior in virtual reality environments. We analyze our collected scanpaths from a total of 87 participants viewing 85 different videos and show that various factors such as viewing mode, content type, and gender significantly impact eye movement statistics. We demonstrate the potential of D-SAV360 as a benchmarking resource for state-of-the-art attention prediction models and discuss its possible applications in further research. By providing a comprehensive dataset of eye movement data for dynamic, multimodal virtual environments, our work can facilitate future investigations of visual behavior and attention in virtual reality

Holographic reality: enhancing the artificial reality experience throuhg interactive 3D holography

Holography was made know by several science-fiction productions, however this technology dates back to the year 1940. Despite the considerable age of this discovery, this technology remains inaccessible to the average consumer. The main goal of this manuscript is to advance the state of the art in interactive holography, providing an accessible and low-cost solution. The final product intends to nudge the HCI com munity to explore potential applications, in particular to be aquatic centric and environmentally friendly. Two main user studies are performed, in order to determine the impact of the proposed solution by a sample audience. Provided user studies include a first prototype as a Tangible User Interface - TUI for Holographic Reality - HR Second study included the Holographic Mounted Display - HMD for proposed HR interface, further analyzing the interactive holographic experience without hand-held devices. Both of these studies were further compared with an Augmented Reality setting. Obtained results demonstrate a significantly higher score for the HMD approach. This suggests it is the better solution, most likely due to the added simplicity and immersiveness features it has. However the TUI study did score higher in several key parameters, and should be considered for future studies. Comparing with an AR experience, the HMD study scores slightly lower, but manages to surpass AR in several parameters. Several approaches were outlined and evaluated, depicting different methods for the creation of Interactive Holographic Reality experiences. In spite of the low maturity of holographic technology, it can be concluded it is comparable and can keep up to other more developed and mature artificial reality settings, further supporting the need for the existence of the Holographic Reality conceptA tecnologia holográfica tornou-se conhecida através da ficção científica, contudo esta tecnologia remonta até ao ano 1940. Apesar da considerável idade desta descoberta, esta tecnologia continua a não ser acessíveil para o consumidor. O objetivo deste manuscrito é avançar o estado de arte da Holografia Interactiva, e fornecer uma solução de baixo custo. O objetivo do produto final é persuadir a comunidade HCI para a exploração de aplicações desta tecnologia, em particular em contextos aquáticos e pró-ambientais. Dois estudos principais foram efetuados, de modo a determinar qual o impacto da solução pro posta numa amostra. Os estudos fornecidos incluem um protótipo inicial baseado numa Interface Tangível e Realidade Holográfica e um dispositivo tangível. O segundo estudo inclui uma interface baseada num dispositivo head-mounted e em Realidade Holográfica, de modo a analisar e avaliar a experiência interativa e holográfica. Ambos os estudos são comparados com uma experiência semelhante, em Realidade Aumentada. Os resultados obtidos demonstram que o estudo HMD recebeu uma avaliação significante mel hor, em comparação com a abordagem TUI. Isto sugere que uma abordagem "head-mounted" tende a ser melhor solução, muito provavelmente devido às vantagens que possui em relação à simplicidade e imersividade que oferece. Contudo, o estudo TUI recebeu pontuações mais altas em alguns parâmetros chave, e deve ser considerados para a implementação de futuros estudos. Comparando com uma experiência de realidade aumentada, o estudo HMD recebeu uma avaliação ligeiramente menor, mas por uma margem mínima, e ultrapassando a AR em alguns parâmetros. Várias abordagens foram deliniadas e avaliadas, com diferentes métodos para a criação de experiências de Realidade Holográfica. Apesar da pouca maturidade da tecnologia holográfica, podemos concluir que a mesma é comparável e consegue acompanhar outros tipos de realidade artificial, que são muito mais desenvolvidos, o que suporta a necessidade da existência do conceito de Realidade Holográfica

2D and 3D computer vision analysis of gaze, gender and age

Human-Computer Interaction (HCI) has been an active research area for over four decades. Research studies and commercial designs in this area have been largely facilitated by the visual modality which brings diversified functionality and improved usability to HCI interfaces by employing various computer vision techniques. This thesis explores a number of facial cues, such as gender, age and gaze, by performing 2D and 3D based computer vision analysis. The ultimate aim is to create a natural HCI strategy that can fulfil user expectations, augment user satisfaction and enrich user experience by understanding user characteristics and behaviours. To this end, salient features have been extracted and analysed from 2D and 3D face representations; 3D reconstruction algorithms and their compatible real-world imaging systems have been investigated; case study HCI systems have been designed to demonstrate the reliability, robustness, and applicability of the proposed method.More specifically, an unsupervised approach has been proposed to localise eye centres in images and videos accurately and efficiently. This is achieved by utilisation of two types of geometric features and eye models, complemented by an iris radius constraint and a selective oriented gradient filter specifically tailored to this modular scheme. This approach resolves challenges such as interfering facial edges, undesirable illumination conditions, head poses, and the presence of facial accessories and makeup. Tested on 3 publicly available databases (the BioID database, the GI4E database and the extended Yale Face Database b), and a self-collected database, this method outperforms all the methods in comparison and thus proves to be highly accurate and robust. Based on this approach, a gaze gesture recognition algorithm has been designed to increase the interactivity of HCI systems by encoding eye saccades into a communication channel similar to the role of hand gestures. As well as analysing eye/gaze data that represent user behaviours and reveal user intentions, this thesis also investigates the automatic recognition of user demographics such as gender and age. The Fisher Vector encoding algorithm is employed to construct visual vocabularies as salient features for gender and age classification. Algorithm evaluations on three publicly available databases (the FERET database, the LFW database and the FRCVv2 database) demonstrate the superior performance of the proposed method in both laboratory and unconstrained environments. In order to achieve enhanced robustness, a two-source photometric stereo method has been introduced to recover surface normals such that more invariant 3D facia features become available that can further boost classification accuracy and robustness. A 2D+3D imaging system has been designed for construction of a self-collected dataset including 2D and 3D facial data. Experiments show that utilisation of 3D facial features can increase gender classification rate by up to 6% (based on the self-collected dataset), and can increase age classification rate by up to 12% (based on the Photoface database). Finally, two case study HCI systems, a gaze gesture based map browser and a directed advertising billboard, have been designed by adopting all the proposed algorithms as well as the fully compatible imaging system. Benefits from the proposed algorithms naturally ensure that the case study systems can possess high robustness to head pose variation and illumination variation; and can achieve excellent real-time performance. Overall, the proposed HCI strategy enabled by reliably recognised facial cues can serve to spawn a wide array of innovative systems and to bring HCI to a more natural and intelligent state

Stereoscopic high dynamic range imaging

Two modern technologies show promise to dramatically increase immersion in virtual environments. Stereoscopic imaging captures two images representing the views of both eyes and allows for better depth perception. High dynamic range (HDR) imaging accurately represents real world lighting as opposed to traditional low dynamic range (LDR) imaging. HDR provides a better contrast and more natural looking scenes. The combination of the two technologies in order to gain advantages of both has been, until now, mostly unexplored due to the current limitations in the imaging pipeline. This thesis reviews both fields, proposes stereoscopic high dynamic range (SHDR) imaging pipeline outlining the challenges that need to be resolved to enable SHDR and focuses on capture and compression aspects of that pipeline. The problems of capturing SHDR images that would potentially require two HDR cameras and introduce ghosting, are mitigated by capturing an HDR and LDR pair and using it to generate SHDR images. A detailed user study compared four different methods of generating SHDR images. Results demonstrated that one of the methods may produce images perceptually indistinguishable from the ground truth. Insights obtained while developing static image operators guided the design of SHDR video techniques. Three methods for generating SHDR video from an HDR-LDR video pair are proposed and compared to the ground truth SHDR videos. Results showed little overall error and identified a method with the least error. Once captured, SHDR content needs to be efficiently compressed. Five SHDR compression methods that are backward compatible are presented. The proposed methods can encode SHDR content to little more than that of a traditional single LDR image (18% larger for one method) and the backward compatibility property encourages early adoption of the format. The work presented in this thesis has introduced and advanced capture and compression methods for the adoption of SHDR imaging. In general, this research paves the way for a novel field of SHDR imaging which should lead to improved and more realistic representation of captured scenes

VR systems for memory assessment and depth perception

La evolución de la tecnología de Realidad Virtual (RV) ha contribuido en todos los campos, incluyendo la psicología. Esta evolución implica mejoras tanto en hardware como en software, que permiten experiencias más inmersivas. En un entorno de RV los usuarios pueden percibir la sensación de "presencia" y sentirse "inmersos". Estas sensaciones son posibles utilizando HMDs. Hoy en día, el desarrollo de los HMDs se ha centrado en mejorar sus características técnicas para ofrecer inmersión total. En psicología, los entornos de RV son una herramienta de investigación. Hay algunas aplicaciones para evaluar la memoria espacial que utilizan métodos básicos de interacción. Sin embargo, sistemas de RV que incorporen estereoscopía y movimiento físico todavía no se han explotado en psicología. En esta tesis, se ha desarrollado un nuevo sistema de RV que combina características inmersivas, interactivas y de movimiento. El sistema de RV (tarea en un laberinto virtual) se ha utilizado para evaluar la memoria espacial y la percepción de profundidad. Se han integrado dos tipos diferentes de interacción: una basada en locomoción que consistió en pedalear en una bicicleta fija (condición1) y otra estacionaria usando un gamepad (condición2). El sistema integró dos tipos de visualización: 1) Oculus Rift (OR); 2) Una gran pantalla estéreo. Se diseñaron dos estudios. El primer estudio (N=89) evaluó la memoria espacial a corto plazo usando el OR y los dos tipos de interacción. Los resultados indican que existían diferencias significativas entre ambas condiciones. Los participantes que utilizaron la condición2 obtuvieron mejor rendimiento que los que utilizaron la tarea en la condición1. Sin embargo, no se encontraron diferencias significativas en las puntuaciones de satisfacción e interacción entre ambas condiciones. El desempeño en la tarea correlacionó con el desempeño en las pruebas neuropsicológicas clásicas, revelando la verosimilitud entre ellas. El segundo estudio (N=59) incluyó participantes con y sin estereopsis. Este estudio evaluó la percepción de profundidad comparando los dos sistemas de visualización. Los participantes realizaron la tarea usando la condición2. Los resultados mostraron que las diferentes características del sistema de visualización no influyeron en el rendimiento en la tarea entre los participantes con y sin estereopsis. Se encontraron diferencias significativas a favor del HMD entre las dos condiciones y entre los dos grupos de participantes respecto a la percepción de profundidad. Los participantes que no tenían estereopsis y no podían percibir la profundidad cuando utilizaban otros sistemas de visualización, tuvieron la ilusión de percepción de profundidad cuando utilizaron el OR. El estudio sugiere que para las personas que no tienen estereopsis, el seguimiento de la cabeza influye en gran medida en la experiencia 3D. Los resultados estadísticos de ambos estudios han demostrado que el sistema de RV desarrollado es una herramienta apropiada para evaluar la memoria espacial a corto plazo y la percepción de profundidad. Por lo tanto, los sistemas de RV que combinan inmersión total, interacción y movimiento pueden ser una herramienta útil para la evaluación de procesos cognitivos humanos como la memoria. De estos estudios se han extraído las siguientes conclusiones generales: 1) La tecnología de RV y la inmersión proporcionada por los actuales HMDs son herramientas adecuadas para aplicaciones psicológicas, en particular, la evaluación de la memoria espacial a corto plazo; 2) Un sistema de RV como el presentado podría ser utilizado como herramienta para evaluar o entrenar adultos en habilidades relacionadas con la memoria espacial a corto plazo; 3) Los dos tipos de interacción utilizados para la navegación en el laberinto virtual podrían ser útiles para su uso con diferentes colectivos; 4) El OR permite que los usuarios sin estereopsis puedan percibir lThe evolution of Virtual Reality (VR) technology has contributed in all fields, including psychology. This evolution involves improvements in hardware and software allowing more immersive experiences. In a VR environment users can perceive the sensation of "presence" and feel "immersed". These sensations are possible using VR devices as HMDs. Nowadays, the development of the HMDs has focused on improving their technical features to offer full immersion. In psychology, VR environments are research tools because they allow the use of new paradigms that are not possible to employ in a real environment. There are some applications for assessing spatial memory that use basic methods of HCI. However, VR systems that incorporate stereoscopy and physical movement have not yet been exploited in psychology. In this thesis, a novel VR system combining immersive, interactive and motion features was developed. This system was used for the assessment of the spatial memory and the evaluation of depth perception. For this system, a virtual maze task was designed and implemented. In this system, two different types of interaction were integrated: a locomotion-based interaction pedaling a fixed bicycle (condition1), and a stationary interaction using a gamepad (condition2). This system integrated two types of display systems: 1) The Oculus Rift; 2) A large stereo screen. Two studies were designed to determine the efficacy of the VR system using physical movement and immersion. The first study (N=89) assessed the spatial short term memory using the Oculus Rift and the two types of interaction The results showed that there were statistically significant differences between both conditions. The participants who performed the condition2 got better performance than participants who performed the condition1. However, there were no statistically significant differences in satisfaction and interaction scores between both conditions. The performance on the task correlated with the performance on other classical neuropsychological tests, revealing a verisimilitude between them. The second study (N=59) involved participants who had and who had not stereopsis. This study assessed the depth perception by comparing the two display systems. The participants performed the task using the condition2. The results showed that the different features of the display system did not influence the performance on the task between the participants with and without stereopsis. Statistically significant differences were found in favor of the HMD between the two conditions and between the two groups of participants regard to depth perception. The participants who did not have stereopsis and could not perceive the depth when they used other display systems (e.g. CAVE); however, they had the illusion of depth perception when they used the Oculus Rift. The study suggests that for the people who did not have stereopsis, the head tracking largely influences the 3D experience. The statistical results of both studies have proven that the VR system developed for this research is an appropriate tool to assess the spatial short-term memory and the depth perception. Therefore, the VR systems that combine full immersion, interaction and movement can be a helpful tool for the assessment of human cognitive processes as the memory. General conclusions from these studies are: 1) The VR technology and immersion provided by current HMDs are appropriate tools for psychological applications, in particular, the assessment of spatial short-term memory; 2) A VR system like the one presented in this thesis could be used as a tool to assess or train adults in skills related to spatial short-term memory; 3) The two types of interaction (condition1 and condition2) used for navigation within the virtual maze could be helpful to use with different collectives; 4) The Oculus Rift allows that the users without stereopsis can perceive the depth perception of 3D objects and have rich 3D experiences.L'evolució de la tecnologia de Realitat Virtual (RV) ha contribuït en tots els camps, incloent la psicologia. Aquesta evolució implica millores en el maquinari i el programari que permeten experiències més immersives. En un entorn de RV, els usuaris poden percebre la sensació de "presència" i sentir-se "immersos". Aquestes sensacions són possibles utilitzant HMDs. Avui dia, el desenvolupament dels HMDs s'ha centrat a millorar les seves característiques tècniques per oferir immersió plena. En la psicologia, els entorns de RV són eines de recerca. Hi ha algunes aplicacions per avaluar la memòria espacial que utilitzen mètodes bàsics d'interacció. Tanmateix, sistemes de RV que incorporen estereoscòpia i moviment físic no s'han explotat en psicologia. En aquesta tesi, s'ha desenvolupat un sistema de RV novell que combina immersió, interacció i moviment. El sistema (tasca en un laberint virtual) s'ha utilitzat per a l'avaluació de la memòria espacial i la percepció de profunditat. S'han integrat dos tipus d'interacció: una interacció basada en locomoció pedalejant una bicicleta fixa (condició1), i l'altra una interacció estacionària usant un gamepad (condició2). S'han integrat dos tipus de sistemes de pantalla: 1) L'Oculus Rift; 2) Una gran pantalla estereoscòpica. Dos estudis van ser dissenyats. El primer estudi (N=89) va avaluar la memòria a curt termini i espacial utilitzant l'Oculus Rift i els dos tipus d'interacció. Els resultats indiquen que hi havia diferències significatives entre les dues condicions. Els participants que van utilitzar la condició2 van obtenir millor rendiment que els participants que van utilitzar la condició1. Tanmateix, no hi havia diferències significatives dins satisfacció i puntuacions d'interacció entre les dues condicions. El rendiment de la tasca va correlacionar amb el rendiment en les proves neuropsicològiques clàssiques, revelant versemblança entre elles. El segon estudi (N=59) va implicar participants que van tenir i que van haver-hi no estereopsis. Aquest estudi va avaluar la percepció de profunditat comparant els dos sistemes de pantalla. Els participants realitzen la tasca utilitzant la condició2. Els resultats van mostrar que les diferents característiques del sistema de pantalla no va influir en el rendiment en la tasca entre els participants qui tenien i els qui no tenien estereopsis. Diferències significatives van ser trobades a favor del HMD entre les dues condicions i entre els dos grups de participants. Els participants que no van tenir estereopsis i no podien percebre la profunditat quan van utilitzar altres sistemes de pantalla (per exemple, CAVE), van tenir la il.lusió de percepció de profunditat quan van utilitzar l'Oculus Rift. L'estudi suggereix que per les persones que no van tenir estereopsis, el seguiment del cap influeix en gran mesura en l'experiència 3D. Els resultats estadístics dels dos estudis han provat que el sistema de RV desenvolupat per aquesta recerca és una eina apropiada per avaluar la memòria espacial a curt termini i la percepció de profunditat. Per això, els sistemes de RV que combinen immersió plena, interacció i moviment poden ser una eina útil per la avaluació de processos cognitius humans com la memòria Les conclusions generals que s'han extret d'aquests estudis, són les següents: 1) La tecnologia de RV i la immersió proporcionada pels HMDs són eines apropiades per aplicacions psicològiques, en particular, la avaluació de memòria espacial a curt termini; 2) Un sistema de RV com el presentat podria ser utilitzat com a eina per avaluar o entrenar adults en habilitats relacionades amb la memòria espacial a curt termini; 3) Els dos tipus d'interacció utilitzats per navegació dins del laberint virtual podrien ser útils per al seu ús amb diferent col.lectius; 3) L'Oculus Rift permet que els usuaris que no tenen estereopsis puguen percebre la percepció de profunditat dels objectes 3D i tenirCárdenas Delgado, SE. (2017). VR systems for memory assessment and depth perception [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/94629TESI

Une méthode pour l'évaluation de la qualité des images 3D stéréoscopiques.

Dans le contexte d'un intérêt grandissant pour les systèmes stéréoscopiques, mais sans méthodes reproductible pour estimer leur qualité, notre travail propose une contribution à la meilleure compréhension des mécanismes de perception et de jugement humains relatifs au concept multi-dimensionnel de qualité d'image stéréoscopique. Dans cette optique, notre démarche s'est basée sur un certain nombre d'outils : nous avons proposé un cadre adapté afin de structurer le processus d'analyse de la qualité des images stéréoscopiques, nous avons implémenté dans notre laboratoire un système expérimental afin de conduire plusieurs tests, nous avons crée trois bases de données d'images stéréoscopiques contenant des configurations précises et enfin nous avons conduit plusieurs expériences basées sur ces collections d'images. La grande quantité d'information obtenue par l'intermédiaire de ces expérimentations a été utilisée afin de construire un premier modèle mathématique permettant d'expliquer la perception globale de la qualité de la stéréoscopie en fonction des paramètres physiques des images étudiée.In a context of ever-growing interest in stereoscopic systems, but where no standardized algorithmic methods of stereoscopic quality assessment exist, our work stands as a step forward in the understanding of the human perception and judgment mechanisms related to the multidimensional concept of stereoscopic image quality. We used a series of tools in order to perform in-depth investigations in this direction: we proposed an adapted framework to structure the process of stereoscopic quality assessment, we implemented a stereoscopic system in our laboratory for performing various tests, we created three stereoscopic datasets with precise structures, and we performed several experimental studies using these datasets. The numerous experimental data obtained were used in order to propose a first mathematical framework for explaining the overall percept of stereoscopic quality in function of the physical parameters of the stereoscopic images under study.SAVOIE-SCD - Bib.électronique (730659901) / SudocGRENOBLE1/INP-Bib.électronique (384210012) / SudocGRENOBLE2/3-Bib.électronique (384219901) / SudocSudocFranceF