Towards Gaze Interaction in Immersive Virtual Reality: Evaluation of a Monocular Eye Tracking Set-Up

Pfeiffer T. Towards Gaze Interaction in Immersive Virtual Reality: Evaluation of a Monocular Eye Tracking Set-Up. In: Schumann M, Kuhlen T, eds. Virtuelle und Erweiterte Realität - Fünfter Workshop der GI-Fachgruppe VR/AR. Berichte aus der Informatik. Aachen: Shaker Verlag GmbH; 2008: 81-92.Of all senses, it is visual perception that is predominantly deluded in Virtual Realities. Yet, the eyes of the observer, despite the fact that they are the fastest perceivable moving body part, have gotten relatively little attention as an interaction modality. A solid integration of gaze, however, provides great opportunities for implicit and explicit human-computer interaction. We present our work on integrating a lightweight head-mounted eye tracking system in a CAVE-like Virtual Reality Set-Up and provide promising data from a user study on the achieved accuracy and latency

A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms

In this paper a review is presented of the research on eye gaze estimation techniques and applications, that has progressed in diverse ways over the past two decades. Several generic eye gaze use-cases are identified: desktop, TV, head-mounted, automotive and handheld devices. Analysis of the literature leads to the identification of several platform specific factors that influence gaze tracking accuracy. A key outcome from this review is the realization of a need to develop standardized methodologies for performance evaluation of gaze tracking systems and achieve consistency in their specification and comparative evaluation. To address this need, the concept of a methodological framework for practical evaluation of different gaze tracking systems is proposed.Comment: 25 pages, 13 figures, Accepted for publication in IEEE Access in July 201

Current Research in Human Physiology and Perception 2020

Šajā izdevumā apkopoti LU FMOF Optometrijas un redzes zinātnes nodaļas un tās sadarbības partneru aktuālie pētījumi par cilvēka fizioloģiju un uztveri. 2020. gadā. Pētījumi ir saistīti ar redzes zinātni un klīnisko optometriju. Daļa rakstu ir latviešu, daļa angļu valodā. // This collection of publications consists of research done in Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia. Reasearch topics are related with vision science and clinical optometry. Some of the articles are in Latvian, some of them are in English

3차원 의료 영상 판독 시선 정보의 대화형 시각적 분석 프레임워크

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2016. 2. 서진욱.We propose an interactive visual analytics framework for diagnostic gaze data on volumetric medical images. The framework is designed to compare gaze data from multiple readers with effective visualizations, which are tailored for volumetric gaze data with additional contextual information. Gaze pattern comparison is essential to understand how radiologists examine medical images and to identify factors influencing the examination. However, prior work on diagnostic gaze data using the medical images acquired from volumetric imaging systems (e.g., computed tomography or magnetic resonance imaging) showed a number of limitations in comparative analysis. In the diagnosis, radiologists scroll through a stack of images to get a 3D cognition of organs and lesions that resulting gaze patterns contain additional depth information compared to the gaze tracking study with 2D stimuli. As a result, the additional spatial dimension aggravated the complexity on visual representation of gaze data. A recent work proposed a visualization design based on direct volume rendering (DVR) for gaze patterns in volumetric imageshowever, effective and comprehensive gaze pattern comparison is still challenging due to lack of interactive visualization tools for comparative gaze analysis. In this paper, we first present an effective visual representation, and propose an interactive analytics framework for multiple volumetric gaze data. We also take the challenge integrating crucial contextual information such as pupil size and windowing (i.e., adjusting brightness and contrast of image) into the analysis process for more in-depth and ecologically valid findings. Among the interactive visualization components, a context-embedded interactive scatterplot (CIS) is especially designed to help users to examine abstract gaze data in diverse contexts by embedding medical imaging representations well-known to radiologists in it. We also present the results from case studies with chest and abdominal radiologistsChapter 1 Introduction 1 1.1 Background 1 1.2 Research Components 5 1.3 Radiological Practice 6 1.4 Organization of the Dissertation 8 Chapter 2 Related Work 9 2.1 Visualization Combining 2D and 3D 9 2.2 Eye Tracking Data Visualization 14 2.3 Comparative Data Analysis 16 2.4 Gaze Analysis in the Medical field 18 Chapter 3 GazeVis: Volumetric Gaze Data 21 3.1 Visualization of Stimuli and Gaze Data 23 3.1.1 Computation of Gaze Field 26 3.1.2 Visualization of Gaze Field 29 3.1.3 Gaze Field for Interactive Information Seeking 30 3.2 Interactions and Dynamic Queries 32 3.2.1 Interaction Design 32 3.2.2 Spatial Filtering 34 3.2.3 Temporal Filtering 34 3.2.4 Transfer Function Control 36 3.2.5 Gaussian Blur Control 38 3.3 Implementation 38 3.4 Evaluation with Radiologists 38 3.4.1 Case Study Protocol 39 3.4.2 Datasets 41 3.4.3 Apparatus 42 3.4.4 Chest Radiologists 42 3.4.5 Abdominal Radiologists 45 3.5 Discussion 49 3.5.1 Spatial Data Structure and Flexibility 49 3.5.2 Interacting with Contextual Data 51 Chapter 4 GazeDx: Interactive Gaze Analysis Framework 53 4.1 Design Rationale 54 4.2 Overviews for Comparative Gaze Analysis 57 4.2.1 Spatial Similarity 57 4.2.2 Qualitative Similarity Overview 58 4.2.3 Multi-level Temporal Overview 60 4.3 In-depth Comparison of Gaze Patterns 65 4.3.1 Detail Views for Individual Readers 65 4.3.2 Aggregation for Group Comparison 67 4.4 CIS: Context-embedded Interactive Scatterplot 68 4.4.1 Flexible Axis Configuration 68 4.4.2 Focus Attention with Familiar Representations 69 4.4.3 Scatterplot Matrix with CIS 72 4.5 Interactive Selection and Filtering 72 4.5.1 Selection by Freehand Drawing 73 4.5.2 Selection by Human Anatomy 74 4.6 Implementation 76 4.7 Case Studies 77 4.7.1 Case Study Protocol 78 4.7.2 Apparatus 80 4.7.3 Case Study 1: Chest Radiologists 81 4.7.4 Case Study 2: Abdominal Radiologists 85 4.8 Discussion 88 Chapter 5 Conclusion 91 Bibliography 94 Abstract in Korean 105Docto

3D head motion, point-of-regard and encoded gaze fixations in real scenes: next-generation portable video-based monocular eye tracking

Portable eye trackers allow us to see where a subject is looking when performing a natural task with free head and body movements. These eye trackers include headgear containing a camera directed at one of the subject\u27s eyes (the eye camera) and another camera (the scene camera) positioned above the same eye directed along the subject\u27s line-of-sight. The output video includes the scene video with a crosshair depicting where the subject is looking -- the point-of-regard (POR) -- that is updated for each frame. This video may be the desired final result or it may be further analyzed to obtain more specific information about the subject\u27s visual strategies. A list of the calculated POR positions in the scene video can also be analyzed. The goals of this project are to expand the information that we can obtain from a portable video-based monocular eye tracker and to minimize the amount of user interaction required to obtain and analyze this information. This work includes offline processing of both the eye and scene videos to obtain robust 2D PORs in scene video frames, identify gaze fixations from these PORs, obtain 3D head motion and ray trace fixations through volumes-of-interest (VOIs) to determine what is being fixated, when and where (3D POR). To avoid the redundancy of ray tracing a 2D POR in every video frame and to group these POR data meaningfully, a fixation-identification algorithm is employed to simplify the long list of 2D POR data into gaze fixations. In order to ray trace these fixations, the 3D motion -- position and orientation over time -- of the scene camera is computed. This camera motion is determined via an iterative structure and motion recovery algorithm that requires a calibrated camera and knowledge of the 3D location of at least four points in the scene (that can be selected from premeasured VOI vertices). The subjects 3D head motion is obtained directly from this camera motion. For the final stage of the algorithm, the 3D locations and dimensions of VOIs in the scene are required. This VOI information in world coordinates is converted to camera coordinates for ray tracing. A representative 2D POR position for each fixation is converted from image coordinates to the same camera coordinate system. Then, a ray is traced from the camera center through this position to determine which (if any) VOI is being fixated and where it is being fixated -- the 3D POR in the world. Results are presented for various real scenes. Novel visualizations of portable eye tracker data created using the results of our algorithm are also presented

Abstract 3D Eye Movement Analysis for VR Visual Inspection Training

This paper presents an improved 3D eye movement analysis algorithm for binocular eye tracking within Virtual Reality for visual inspection training. The user’s gaze direction, head position and orientation are tracked to allow recording of the user’s fixations within the environment. The paper summarizes methods for (1) integrating the eye tracker into a Virtual Reality framework, (2) calculating the user’s 3D gaze vector, and (3) calibrating the software to estimate the user’s inter-pupillary distance post-facto. New techniques are presented for eye movement analysis in 3D for improved signal noise suppression. The paper describes (1) the use of Finite Impulse Response (FIR) filters for eye movement analysis, (2) the utility of adaptive thresholding and fixation grouping, and (3) a heuristic method to recover lost eye movement data due to miscalibration. While the linear signal analysis approach is itself not new, its application to eye movement analysis in three dimensions advances traditional 2D approaches since it takes into account the 6 degrees of freedom of head movements and is resolution independent. Results indicate improved noise suppression over our previous signal analysis approach.

Ajustement d'affichage stéréoscopique par évaluation du point de regard 3D dans un environnement virtuel

RÉSUMÉ Ce projet de recherche se déroule au sein d'une voûte immersive de réalité virtuelle. Un tel environnement plonge l'observateur au cœur d'un monde généré par ordinateur et projeté sur 4 faces d'un cube d'environ 3 m de côté au milieu duquel il se tient. Néanmoins, l'affichage stéréoscopique provoque généralement de l'inconfort oculaire, voire de la fatigue visuelle. Le but poursuivi était d'améliorer l'expérience immersive en tâchant de diminuer la gêne visuelle ressentie, tout en rendant le monde plus réaliste et en renforçant le sentiment de présence. L'objectif fut donc défini de la manière suivante : concevoir et évaluer une méthode d'ajustement des paramètres de configuration du système de caméras, afin de trouver des valeurs optimales et vraisemblablement personnalisées pour chaque individu. Nos hypothèses soutenaient qu'ajuster ces valeurs en fonction de la position du point de regard permettrait de rendre la configuration plus performante. Des lunettes intégrant des caméras dirigées vers les yeux ont été utilisées comme système de suivi oculaire afin de calculer en temps réel le point de regard de l'observateur dans la scène. Cette donnée a ensuite servi à ajuster deux paramètres de la configuration du système de caméras : la distance séparant les deux caméras virtuelles utilisées pour obtenir le rendu stéréoscopique de la scène, et une profondeur de champ artificielle générée autour du point de regard. La connaissance de la position de ce dernier, et plus particulièrement des objets qui ont retenu l'attention de l'individu, fournit également des indices sur le traitement cognitif de l'information visuelle, c'est-à-dire la manière dont il a perçu et inspecté la scène. La première étape méthodologique a consisté à développer les outils qui permettent d'intégrer le système de suivi oculaire à la voûte de réalité virtuelle de l'Institut Philippe-Pinel de Montréal, de déterminer le point de regard grâce à une méthode de raycasting couplée à une procédure de calibration, et enfin de modifier les paramètres de la configuration en fonction. Subséquemment, une expérimentation a été conduite auprès de participants pour évaluer notre proposition par comparaison avec cinq autres configurations plus neutres ou classiques, intégrant un aspect éthique au projet. Les 6 configurations testées se différencient par la présence ou non d'un flou de profondeur de champ et la valeur de la distance inter-caméras (DIC) : fixe et choisie par le participant, fixe et correspondant à sa distance interpupillaire anatomique (DIP) ou variable en fonction de la position du point de regard. En dehors des méthodes de comparaison courantes répertoriées au cours de la revue de littérature, nous avons cherché une technique qui fournirait une vraie indication sur le comportement oculaire de l'observateur au cours d'une séance d'immersion. Pour cela, nous avons mis au point un test basé sur la reproduction d'une scène réelle en virtuel. Les 18 participants ont été recrutés sur la base du volontariat. Afin de limiter les risques de fatigue visuelle au cours de l'immersion, ceux-ci devaient être accoutumés à la réalité virtuelle et la stéréoscopie, c'est-à-dire avoir déjà joué à des jeux vidéos et assisté à des projections de films en 3D sans gêne. Ces participants ont dû effectuer différentes tâches, chacune associée à un objectif. Tout d'abord, ils devaient naviguer dans un environnement virtuel riche – un appartement comportant de nombreux objets – successivement avec les différentes configurations, et noter chacune selon le confort ressenti, le réalisme du rendu, le plaisir de navigation, la perception des distances et des profondeurs, et enfin le sentiment de présence dans le monde virtuel. Ils durent par la suite déterminer leurs limites de fusion en approchant ou reculant un objet virtuel. Celles-ci constituent une comparaison objective d'efficacité du rendu entre les différentes configurations, la diplopie générée par des difficultés de fusion des images stéréoscopiques étant l'une des sources principales d'inconfort. Finalement, les participants devaient suivre les déplacements d'un robot à la fois dans une scène réelle et dans sa reproduction la plus fidèle possible en virtuel. Les angles de vergence oculaire et les écarts angulaires à une cible disposée sur le robot constituent les indices de comparaison. La différence de variations de ces angles entre cas virtuels et réel permet de déterminer avec quelle configuration la position de l'objet, en particulier en profondeur, est rendue de la façon la plus réaliste. Les analyses statistiques et quantitatives réalisées sur les résultats de cette expérimentation permettent de mettre en lumière les effets des paramètres testés. Concernant la distance inter-caméras, il apparaît tout d'abord qu'une DIC variable ou correspondant à la DIP est significativement plus performante que la DIC fixe en ce qui concerne les limites de fusion stéréoscopique atteintes et la correspondance des mouvements oculaires entre réel et virtuel. La distance interpupillaire anatomique conduit quant à elle à des notations subjectives plus élevées, bien que la différence avec les autres valeurs pour ce paramètre ne soit pas significative. Enfin, notre proposition de DIC variable en fonction du point de regard conduit à des comportements oculaires plus réalistes en considérant le nombre de participants ayant obtenu le résultat le plus proche du réel avec cette valeur de paramètre. C'est en revanche la DIP qui obtient les meilleures moyennes pour ce test. En ce qui concerne le flou de profondeur de champ, les analyses révèlent que sa présence n'impacte pas significativement la capacité de fusion ou la similitude de comportement oculaire entre réel et virtuel. Elle s'est en revanche révélée significativement négative pour les notes attribuées dans le critère subjectif de perception des distances et des profondeurs, et a également influé sur l'inspection visuelle en augmentant de manière significative la durée moyenne des fixations et saccades. L'étude des résultats établit donc que nos propositions d'ajustement des paramètres n'ont pas amélioré significativement les performances du système de caméras et incite globalement à choisir une configuration dépourvue de flou et utilisant la distance interpupillaire anatomique. Cependant, si cette expérimentation n'a pas permis de valider nos hypothèses, elle ne les a pas non plus infirmées car certaines limitations techniques rencontrées ont plus que probablement eu une influence négative. En effet, des problèmes d'affichage intervenus en cours d'expérimentation ainsi qu'un faible nombre d'images générées par seconde, notamment en présence de flou de profondeur de champ, ont selon nous introduit des biais et conduit à des résultats qui ne sont pas imputables aux seules configurations. Nous suggérons en particulier de résoudre ces problèmes et de s'assurer, en le mesurant, que le temps de latence de mise à jour de l'affichage en fonction du point de regard n'est pas gênant pour le participant. Une seconde expérimentation permettra alors plus sûrement de conclure quant au potentiel des ajustements testés au cours de ce projet. Cette recherche a dans un premier temps consisté à intégrer un système de suivi oculaire à la voûte de réalité virtuelle de l'Institut Philippe-Pinel de Montréal. Les applications à court terme comprennent donc la possibilité pour les cliniciens de cet institut de connaître en temps réel le point de regard de l'observateur, et ainsi l'objet de son attention, avec une bonne précision. Grâce à la procédure de calibration mise en place, la mesure d'imprécision fut réduite à 2,65 cm au cours des tests préliminaires et à 6,47 cm en moyenne sur l'ensemble des participants pour des cibles situées à 2 m, soit 0,76° et 1,85° d'angle visuel. Les outils d'analyse des résultats développés, extrayant des données statistiques ou des représentations graphiques du signal visuel, leur serviront également au moment de l'interprétation des comportements oculaires en matière d'attention. Dans un second temps, nous avons conçu au cours de ce projet un cadre méthodologique novateur de comparaison des configurations stéréoscopiques. En effet, notre test utilisant la reproduction virtuelle d'une scène réelle et l'étude des mouvements de vergence pourra être employé dans d'autres recherches afin de mesurer efficacement la performance du rendu de la géométrie – positions, profondeurs, distances relatives et tailles – d'un environnement réel. Une seconde expérimentation réalisée après avoir effectué certaines corrections au niveau de l'installation de la voûte et de l'implémentation pourrait également révéler le potentiel réel de nos propositions d'ajustement. Enfin, il est important de noter que dans le contexte d'application de la réalité virtuelle à la psychiatrie légale, toute amélioration de la configuration du système de caméras ou du système d'affichage permet d'augmenter ses potentiels de diagnostic et de traitement, en renforçant le sentiment de présence des patients et en rapprochant les comportements exprimés au sein de la voûte de ceux dont ils feraient preuve dans le monde réel. L'intérêt est alors de valider l'utilisation d'environnements et d'avatars virtuels en lieu et place de personnes réelles, ce qui offre la possibilité de mettre en scène des situations davantage personnalisées, contrôlées et éthiquement acceptables. En mettant en lumière la configuration qui parmi les 6 testées et dans l'installation actuelle de la voûte est la plus confortable, la plus immersive et conduit en moyenne à un comportement oculaire plus naturel, notre projet permet aux cliniciens d'exploiter davantage les capacités de la réalité virtuelle.----------ABSTRACT This research project concerned a computer assisted virtual environment (CAVE). Such an environment makes the viewer dive into a computer-generated world, projected onto 4 faces of a room-sized cube in the middle of which he stands. Nevertheless, stereoscopic display often generates visual discomfort, or even visual fatigue. Our goal is to improve the immersive experience by trying to lower this visual discomfort, while making the world more realistic and improving the feeling of presence. The objective was thus defined as follows: design and evaluate an adjustment method for the configuration settings of the cameras system, so as to find the optimal and customized values for each individual. Our assumptions stated that adjusting these values according to the gaze point position would lead to a more efficient configuration. Glasses with cameras pointing towards the eyes were used as an eye tracking system so as to calculate in real time the viewer's gaze point in the scene. This information was then used to adjust two parameters of the cameras system configuration: the distance between the two virtual cameras used for stereoscopic rendering of the scene, and an artificial depth of field generated around the gaze point. The position of the latter, and more particularly the objects that have caught the viewer's attention, also provide clues about the cognitive processing of visual information, i.e. how he perceived and inspected the scene. The first methodological step was to develop tools in order to integrate the gaze tracking system to the CAVE of the Philippe-Pinel Institute of Montreal, to determine the gaze point through a method of raycasting coupled to a calibration procedure, and finally to change the configuration settings accordingly. Subsequently, an experiment was conducted with participants to evaluate our proposal in comparison with five other more neutral and classic configurations, incorporating an ethical dimension to the project. The 6 tested configurations differ in the presence or absence of depth of field blur and the value of the inter-camera distance (ICD): fixed and chosen by the participant, fixed and equal to its anatomical interpupillary distance (IPD), or variable depending on the gaze point. Apart from the common comparison methods listed in the literature review, we sought a technique that would provide a true indication of the viewer's ocular behavior during an immersion session. To this aim, we developed a test based on the virtual reproduction of a real scene. The 18 participants were recruited on a voluntary basis. To reduce the risk of eye strain during the immersion, they had to be familiar with virtual reality and stereoscopic displays, i.e. having played video games and watched 3D movies without experiencing visual discomfort. These participants were asked to perform different tasks, each associated with a goal. First, they had to navigate through a rich virtual environment – an apartment with many objects – successively with each configuration, and give it a rating depending on the felt comfort, rendering realism, fun, distances and depths perception, and finally the feeling of presence in the virtual world. Then, they had to determine their limits of fusion by moving a virtual object closer or further. These provide an objective comparison of rendering effectiveness between the different configurations, the diplopia generated by difficulties to merge stereoscopic images being one of the main sources of visual discomfort. Finally, participants were asked to visually follow a robot in both a real scene and its as accurate as possible virtual reproduction. The angles of ocular vergence and deviations from a target located on top of the robot were used as comparison measures. The difference of angles variations between real and virtual helps determine the configuration with which the position of the object, especially the depth, is rendered in the most realistic way. We carried out statistics and quantitative analysis on the experiment results in order to highlight the effects of the tested parameters. Concerning the inter-camera distance, it first appears that a variable ICD or one corresponding to the IPD is significantly more efficient than the fixed ICD, regarding the limits of stereoscopic fusion and the correspondence between eye movements in real and virtual. Moreover, the anatomical interpupillary distance leads to higher subjective ratings, although the difference with the other values for this parameter is not significant. Finally, our proposal of variable ICD depending on the gaze point leads to more realistic eye behavior when taking into account the number of participants who obtained the result closest to the real scene with this value of parameter. It is instead the IPD who achieves the best average for this test. The presence of depth of field has meanwhile appeared to have no significant impact on the ability to merge stereoscopic images, or on the similarity between ocular behavior in real and virtual. However, it revealed to impact significantly negatively the scores given in the subjective criteria of distances and depths perception, and also influenced visual inspection by significantly increasing the average duration of the fixations and saccades. Our results established that our proposals for adjusting parameters did not significantly improve the performance of the cameras system and encourages overall to select a configuration without blur and using anatomical interpupillary distance. However, if this experiment failed to prove our assumptions, it did not invalidate them because of technical limitations which had more than likely a negative influence. Indeed, display problems that occurred during the experiment as well as a small number of frames generated per second, especially in the presence of depth of field blur, may have introduced bias and led to results that can not be attributed solely to the configuration. We suggest in particular to solve these problems and to ensure, by measuring it, that the lag in updating the display based on the gaze point is not annoying for the participant. A second experiment will then be necessary before drawing a final conclusion on the tested adjustments potential. The first step of this research consisted in integrating an eye tracking system to the Philippe-Pinel Institute of Montreal CAVE. The short term applications therefore include the ability for the clinicians of this institute to know in real time the observer gaze point, and thus the object of his attention, with good accuracy. Thanks to our calibration procedure, measured inaccuracy was reduced to 2.65 cm over the preliminary tests and to 6.47 cm on average across all participants for targets at 2 m, i.e. 0.76° and 1.85° of visual angle. The results analysis tools developed, that extract statistical data or graphical representations from the visual signal, will also be of use to them when interpreting eye behavior in terms of attention. In a second step, we provide an innovative framework for comparing stereoscopic configurations. Indeed, our test using a virtual reproduction of a real scene and the study of vergence movements can be used in further research to effectively measure the performance of rendering the geometry – positions, depths, relative distances and sizes – of a real environment. A second experiment performed after making some corrections in the CAVE installation and in the implementation could also reveal the true potential of the adjustments we proposed. Finally, it is important to note that in the context of virtual reality application in forensic psychiatry, any improvement in the cameras configuration system or display system increases its diagnostic and treatment potentials, by enhancing the patients' feeling of presence and by bringing the behaviors expressed in the CAVE closer to the ones they would show in the real world. The interest is then to validate the use of virtual environments and avatars instead of real people, making possible to stage more specific, controlled and ethically acceptable situations. By highlighting the configuration among the 6 tested which is, in the current CAVE installation, the most comfortable, most immersive and which leads on average to more natural ocular movements, our project enables clinicians to further exploit the virtual reality potency