Deep into the Eyes: Applying Machine Learning to improve Eye-Tracking

Eye-tracking has been an active research area with applications in personal and behav- ioral studies, medical diagnosis, virtual reality, and mixed reality applications. Improving the robustness, generalizability, accuracy, and precision of eye-trackers while maintaining privacy is crucial. Unfortunately, many existing low-cost portable commercial eye trackers suffer from signal artifacts and a low signal-to-noise ratio. These trackers are highly depen- dent on low-level features such as pupil edges or diffused bright spots in order to precisely localize the pupil and corneal reflection. As a result, they are not reliable for studying eye movements that require high precision, such as microsaccades, smooth pursuit, and ver- gence. Additionally, these methods suffer from reflective artifacts, occlusion of the pupil boundary by the eyelid and often require a manual update of person-dependent parame- ters to identify the pupil region. In this dissertation, I demonstrate (I) a new method to improve precision while maintaining the accuracy of head-fixed eye trackers by combin- ing velocity information from iris textures across frames with position information, (II) a generalized semantic segmentation framework for identifying eye regions with a further extension to identify ellipse fits on the pupil and iris, (III) a data-driven rendering pipeline to generate a temporally contiguous synthetic dataset for use in many eye-tracking ap- plications, and (IV) a novel strategy to preserve privacy in eye videos captured as part of the eye-tracking process. My work also provides the foundation for future research by addressing critical questions like the suitability of using synthetic datasets to improve eye-tracking performance in real-world applications, and ways to improve the precision of future commercial eye trackers with improved camera specifications

Spatiotemporal Integration of Early Visual Processing: Visual Phenomena beyond the Critical Fusion Frequency

Department of Biomedical Engineering (Human Factors Engineering)Our visual system integrates continuous signals to construct visual representation. This integration process involves combining visual inputs across both space and time. This spatiotemporal integration is inherently linked to motion. The movements of our eyes play a fundamental role in shaping our visual perception by spatially shifting the images projected on our retinas. The significance of considering the effects of eye movements on visual perception, resulting from these spatial modulations, has been consistently emphasized. In the first part of the study, we focused on examining the influence of eye movements on the temporal sensitivity of the human visual system. Specifically, we assessed participants' ability to detect the flickering visual stimuli with and without the inclusion of eye movements. The findings revealed that the effects of eye movements varied depending on the spatial features of the stimuli. When eye movements were incorporated, participants were able to perceive flickering edges beyond their temporal limit, surpassing the critical fusion frequency (CFF). Furthermore, a significant positive correlation was observed between temporal sensitivity and the extent of eye movements executed during stimulus presentation. To further elucidate the alterations in visual perception caused by the spatial characteristics of visual stimuli and the shifted retinal image resulting from eye movements, we developed a spatiotemporal integration model. This model aimed to emulate the spatiotemporal integration process occurring in our early visual system and was employed to encompass the observations made in this study. In the second part of the study, we investigated the phenomena of visual persistence and the sense of reality associated with moving objects that move at a speed comparable to our eye movements (approximately 100 degrees per second). Introducing a Speedline, which connected two consecutively presented objects spatially, mitigated the smear effect on moving objects in terms of the range of visual persistence and perceived length. Additionally, the presence of the Speedline enhanced the resemblance between the moving objects in the display and the motion of objects in the real world, thereby amplifying the sense of reality in the object motion. Overall, this study offers valuable insights into how eye movements impact visual perception, considering the spatial features of objects. The predictions made by the developed model suggest that the observed behavioral patterns, such as elevated temporal sensitivity to stimuli containing spatial edges, are outcomes of the spatiotemporal integration process occurring in our early visual system. By investigating the intricate relationship between eye movements, object motion, spatial characteristics of visual stimuli, and visual perception, we deepen our understanding of how visual system processes and integrates visual information.clos

3D head motion, point-of-regard and encoded gaze fixations in real scenes: next-generation portable video-based monocular eye tracking

Portable eye trackers allow us to see where a subject is looking when performing a natural task with free head and body movements. These eye trackers include headgear containing a camera directed at one of the subject\u27s eyes (the eye camera) and another camera (the scene camera) positioned above the same eye directed along the subject\u27s line-of-sight. The output video includes the scene video with a crosshair depicting where the subject is looking -- the point-of-regard (POR) -- that is updated for each frame. This video may be the desired final result or it may be further analyzed to obtain more specific information about the subject\u27s visual strategies. A list of the calculated POR positions in the scene video can also be analyzed. The goals of this project are to expand the information that we can obtain from a portable video-based monocular eye tracker and to minimize the amount of user interaction required to obtain and analyze this information. This work includes offline processing of both the eye and scene videos to obtain robust 2D PORs in scene video frames, identify gaze fixations from these PORs, obtain 3D head motion and ray trace fixations through volumes-of-interest (VOIs) to determine what is being fixated, when and where (3D POR). To avoid the redundancy of ray tracing a 2D POR in every video frame and to group these POR data meaningfully, a fixation-identification algorithm is employed to simplify the long list of 2D POR data into gaze fixations. In order to ray trace these fixations, the 3D motion -- position and orientation over time -- of the scene camera is computed. This camera motion is determined via an iterative structure and motion recovery algorithm that requires a calibrated camera and knowledge of the 3D location of at least four points in the scene (that can be selected from premeasured VOI vertices). The subjects 3D head motion is obtained directly from this camera motion. For the final stage of the algorithm, the 3D locations and dimensions of VOIs in the scene are required. This VOI information in world coordinates is converted to camera coordinates for ray tracing. A representative 2D POR position for each fixation is converted from image coordinates to the same camera coordinate system. Then, a ray is traced from the camera center through this position to determine which (if any) VOI is being fixated and where it is being fixated -- the 3D POR in the world. Results are presented for various real scenes. Novel visualizations of portable eye tracker data created using the results of our algorithm are also presented

The multidimensional spectrum of imagination: Images, Dreams, Hallucinations, and Active, Imaginative Perception.

A theory of the structure and cognitive function of the human imagination that attempts to do justice to traditional intuitions about its psychological centrality is developed, largely through a detailed critique of the theory propounded by Colin McGinn. Like McGinn, I eschew the highly deflationary views of imagination, common amongst analytical philosophers, that treat it either as a conceptually incoherent notion, or as psychologically trivial. However, McGinn fails to develop his alternative account satisfactorily because (following Reid, Wittgenstein and Sartre) he draws an excessively sharp, qualitative distinction between imagination and perception, and because of his flawed, empirically ungrounded conception of hallucination. His arguments in defense of these views are rebutted in detail, and the traditional, passive, Cartesian view of visual perception, upon which several of them implicitly rely, is criticized in the light of findings from recent cognitive science and neuroscience. It is also argued that the apparent intuitiveness of the passive view of visual perception is a result of mere historical contingency. An understanding of perception (informed by modern visual science) as an inherently active process enables us to unify our accounts of perception, mental imagery, dreaming, hallucination, creativity, and other aspects of imagination within a single coherent theoretical framework

Engineering data compendium. Human perception and performance. User's guide

The concept underlying the Engineering Data Compendium was the product of a research and development program (Integrated Perceptual Information for Designers project) aimed at facilitating the application of basic research findings in human performance to the design and military crew systems. The principal objective was to develop a workable strategy for: (1) identifying and distilling information of potential value to system design from the existing research literature, and (2) presenting this technical information in a way that would aid its accessibility, interpretability, and applicability by systems designers. The present four volumes of the Engineering Data Compendium represent the first implementation of this strategy. This is the first volume, the User's Guide, containing a description of the program and instructions for its use

Gaze Guidance, Task-Based Eye Movement Prediction, and Real-World Task Inference using Eye Tracking

The ability to predict and guide viewer attention has important applications in computer graphics, image understanding, object detection, visual search and training. Human eye movements provide insight into the cognitive processes involved in task performance and there has been extensive research on what factors guide viewer attention in a scene. It has been shown, for example, that saliency in the image, scene context, and task at hand play significant roles in guiding attention. This dissertation presents and discusses research on visual attention with specific focus on the use of subtle visual cues to guide viewer gaze and the development of algorithms to predict the distribution of gaze about a scene. Specific contributions of this work include: a framework for gaze guidance to enable problem solving and spatial learning, a novel algorithm for task-based eye movement prediction, and a system for real-world task inference using eye tracking. A gaze guidance approach is presented that combines eye tracking with subtle image-space modulations to guide viewer gaze about a scene. Several experiments were conducted using this approach to examine its impact on short-term spatial information recall, task sequencing, training, and password recollection. A model of human visual attention prediction that uses saliency maps, scene feature maps and task-based eye movements to predict regions of interest was also developed. This model was used to automatically select target regions for active gaze guidance to improve search task performance. Finally, we develop a framework for inferring real-world tasks using image features and eye movement data. Overall, this dissertation naturally leads to an overarching framework, that combines all three contributions to provide a continuous feedback system to improve performance on repeated visual search tasks. This research has important applications in data visualization, problem solving, training, and online education

Down-Conditioning of Soleus Reflex Activity using Mechanical Stimuli and EMG Biofeedback

Spasticity is a common syndrome caused by various brain and neural injuries, which can severely impair walking ability and functional independence. To improve functional independence, conditioning protocols are available aimed at reducing spasticity by facilitating spinal neuroplasticity. This down-conditioning can be performed using different types of stimuli, electrical or mechanical, and reflex activity measures, EMG or impedance, used as biofeedback variable. Still, current results on effectiveness of these conditioning protocols are incomplete, making comparisons difficult. We aimed to show the within-session task- dependent and across-session long-term adaptation of a conditioning protocol based on mechanical stimuli and EMG biofeedback. However, in contrast to literature, preliminary results show that subjects were unable to successfully obtain task-dependent modulation of their soleus short-latency stretch reflex magnitude