Video-based eyetracking methods and algorithms in head-mounted displays

Head pose is utilized to approximate a user\u27s line-of-sight for real-time image rendering and interaction in most of the 3D visualization applications using head-mounted displays (HMD). The eye often reaches an object of interest before the completion of most head movements. It is highly desirable to integrate eye-tracking capability into HMDs in various applications. While the added complexity of an eyetracked-HMD (ETHMD) imposes challenges on designing a compact, portable, and robust system, the integration offers opportunities to improve eye tracking accuracy and robustness. In this paper, based on the modeling of an eye imaging and tracking system, we examine the challenges and identify parametric requirements for video-based pupil-glint tracking methods in an ET-HMD design, and predict how these parameters may affect the tracking accuracy, resolution, and robustness. We further present novel methods and associated algorithms that effectively improve eye-tracking accuracy and extend the tracking range

Optical Gaze Tracking with Spatially-Sparse Single-Pixel Detectors

Gaze tracking is an essential component of next generation displays for virtual reality and augmented reality applications. Traditional camera-based gaze trackers used in next generation displays are known to be lacking in one or multiple of the following metrics: power consumption, cost, computational complexity, estimation accuracy, latency, and form-factor. We propose the use of discrete photodiodes and light-emitting diodes (LEDs) as an alternative to traditional camera-based gaze tracking approaches while taking all of these metrics into consideration. We begin by developing a rendering-based simulation framework for understanding the relationship between light sources and a virtual model eyeball. Findings from this framework are used for the placement of LEDs and photodiodes. Our first prototype uses a neural network to obtain an average error rate of 2.67{\deg} at 400Hz while demanding only 16mW. By simplifying the implementation to using only LEDs, duplexed as light transceivers, and more minimal machine learning model, namely a light-weight supervised Gaussian process regression algorithm, we show that our second prototype is capable of an average error rate of 1.57{\deg} at 250 Hz using 800 mW.Comment: 10 pages, 8 figures, published in IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 202

Towards Energy Efficient Mobile Eye Tracking for AR Glasses through Optical Sensor Technology

After the introduction of smartphones and smartwatches, Augmented Reality (AR) glasses are considered the next breakthrough in the field of wearables. While the transition from smartphones to smartwatches was based mainly on established display technologies, the display technology of AR glasses presents a technological challenge. Many display technologies, such as retina projectors, are based on continuous adaptive control of the display based on the user’s pupil position. Furthermore, head-mounted systems require an adaptation and extension of established interaction concepts to provide the user with an immersive experience. Eye-tracking is a crucial technology to help AR glasses achieve a breakthrough through optimized display technology and gaze-based interaction concepts. Available eye-tracking technologies, such as Video Oculography (VOG), do not meet the requirements of AR glasses, especially regarding power consumption, robustness, and integrability. To further overcome these limitations and push mobile eye-tracking for AR glasses forward, novel laser-based eye-tracking sensor technologies are researched in this thesis. The thesis contributes to a significant scientific advancement towards energy-efficientmobile eye-tracking for AR glasses. In the first part of the thesis, novel scanned laser eye-tracking sensor technologies for AR glasses with retina projectors as display technology are researched. The goal is to solve the disadvantages of VOG systems and to enable robust eye-tracking and efficient ambient light and slippage through optimized sensing methods and algorithms. The second part of the thesis researches the use of static Laser Feedback Interferometry (LFI) sensors as low power always-on sensor modality for detection of user interaction by gaze gestures and context recognition through Human Activity Recognition (HAR) for AR glasses. The static LFI sensors can measure the distance to the eye and the eye’s surface velocity with an outstanding sampling rate. Furthermore, they offer high integrability regardless of the display technology. In the third part of the thesis, a model-based eye-tracking approach is researched based on the static LFI sensor technology. The approach leads to eye-tracking with an extremely high sampling rate by fusing multiple LFI sensors, which enables methods for display resolution enhancement such as foveated rendering for AR glasses and Virtual Reality (VR) systems. The scientific contributions of this work lead to a significant advance in the field of mobile eye-tracking for AR glasses through the introduction of novel sensor technologies that enable robust eye tracking in uncontrolled environments in particular. Furthermore, the scientific contributions of this work have been published in internationally renowned journals and conferences

A geometric basis for measurement of three-dimensional eye position using image processing

AbstractPolar cross correlation is commonly used for determination of ocular torsion from video images, but breaks down at eccentric positions if the spherical geometry of the eye is not considered. We have extended this method to allow three-dimensional eye position measurement over a range of ±20 deg by determining the correct projection of the eye onto the image plane of the camera. We also determine the orientation of the camera with respect to the eye, allowing eye position to be represented in appropriate head-fixed coordinates. These algorithms have been validated using both in vitro and in vivo measures of eye position

High-Accuracy Gaze Estimation for Interpolation-Based Eye-Tracking Methods

This study investigates the influence of the eye-camera location associated with the accuracy and precision of interpolation-based eye-tracking methods. Several factors can negatively influence gaze estimation methods when building a commercial or off-the-shelf eye tracker device, including the eye-camera location in uncalibrated setups. Our experiments show that the eye-camera location combined with the non-coplanarity of the eye plane deforms the eye feature distribution when the eye-camera is far from the eye’s optical axis. This paper proposes geometric transformation methods to reshape the eye feature distribution based on the virtual alignment of the eye-camera in the center of the eye’s optical axis. The data analysis uses eye-tracking data from a simulated environment and an experiment with 83 volunteer participants (55 males and 28 females). We evaluate the improvements achieved with the proposed methods using Gaussian analysis, which defines a range for high-accuracy gaze estimation between −0.5∘ and 0.5∘. Compared to traditional polynomial-based and homography-based gaze estimation methods, the proposed methods increase the number of gaze estimations in the high-accuracy range

The use of polarized light for biomedical applications

Polarized light has the ability to increase the specificity of the investigation of biomedical samples and is finding greater utilization in the fields of medical diagnostics, sensing, and measurement. In particular, this dissertation focuses on the application of polarized light to address a major obstacle in the development of an optical based polarimetric non-invasive glucose detector that has the potential to improve the quality of life and prolong the life expectancy of the millions of people afflicted with the disease diabetes mellitus. By achieving the mapping of the relative variations in rabbit corneal birefringence, it is hoped that the understanding of the results contained herein will facilitate the development of techniques to eliminate the effects of changing corneal birefringence on polarimetric glucose measurement through the aqueous humor of the eye. This dissertation also focuses on the application of polarized light to address a major downside of cardiovascular biomechanics research, which is the utilization of toxic chemicals to prepare samples for histological examination. To this end, a polarization microscopy image processing technique is applied to non-stained cardiovascular samples as a means to eliminate, for certain cardiac samples, the necessity for staining using toxic chemicals. The results from this work have the potential to encourage more investigators to join the field of cardiac biomechanics, which studies the remodeling processes responsible for cardiovascular diseases such as myocardial infarct (heart attacks) and congestive heart failure. Cardiovascular disease is epidemic, particularly amongst the population group older than 65 years, and the number of people affected by this disease is expected to increase appreciably as the baby boomer generation transitions into this older, high risk population group. A better understanding of the responsible mechanisms for cardiac tissue remodeling will facilitate the development of better prevention and treatment regimens by improving the early detection and diagnosis of this disease

Deep into the Eyes: Applying Machine Learning to improve Eye-Tracking

Eye-tracking has been an active research area with applications in personal and behav- ioral studies, medical diagnosis, virtual reality, and mixed reality applications. Improving the robustness, generalizability, accuracy, and precision of eye-trackers while maintaining privacy is crucial. Unfortunately, many existing low-cost portable commercial eye trackers suffer from signal artifacts and a low signal-to-noise ratio. These trackers are highly depen- dent on low-level features such as pupil edges or diffused bright spots in order to precisely localize the pupil and corneal reflection. As a result, they are not reliable for studying eye movements that require high precision, such as microsaccades, smooth pursuit, and ver- gence. Additionally, these methods suffer from reflective artifacts, occlusion of the pupil boundary by the eyelid and often require a manual update of person-dependent parame- ters to identify the pupil region. In this dissertation, I demonstrate (I) a new method to improve precision while maintaining the accuracy of head-fixed eye trackers by combin- ing velocity information from iris textures across frames with position information, (II) a generalized semantic segmentation framework for identifying eye regions with a further extension to identify ellipse fits on the pupil and iris, (III) a data-driven rendering pipeline to generate a temporally contiguous synthetic dataset for use in many eye-tracking ap- plications, and (IV) a novel strategy to preserve privacy in eye videos captured as part of the eye-tracking process. My work also provides the foundation for future research by addressing critical questions like the suitability of using synthetic datasets to improve eye-tracking performance in real-world applications, and ways to improve the precision of future commercial eye trackers with improved camera specifications

Eye alignment:a novel approach

This thesis describes the design and development of an eye alignment/tracking system which allows self alignment of the eye’s optical axis with a measurement axis. Eye alignment is an area of research largely over-looked, yet it is a fundamental requirement in the acquisition of clinical data from the eye. New trends in the ophthalmic market, desiring portable hand-held apparatus, and the application of ophthalmic measurements in areas other than vision care have brought eye alignment under new scrutiny. Ophthalmic measurements taken in hand-held devices with out an clinician present requires alignment in an entirely new set of circumstances, requiring a novel solution. In order to solve this problem, the research has drawn upon eye tracking technology to monitor the eye, and a principle of self alignment to perform alignment correction. A handheld device naturally lends itself to the patient performing alignment, thus a technique has been designed to communicate raw eye tracking data to the user in a manner which allows the user to make the necessary corrections. The proposed technique is a novel methodology in which misalignment to the eye’s optical axis can be quantified, corrected and evaluated. The technique uses Purkinje Image tracking to monitor the eye’s movement as well as the orientation of the optical axis. The use of two sets of Purkinje Images allows quantification of the eye’s physical parameters needed for accurate Purkinje Image tracking, negating the need for prior anatomical data. An instrument employing the methodology was subsequently prototyped and validated, allowing a sample group to achieve self alignment of their optical axis with an imaging axis within 16.5-40.8 s, and with a rotational precision of 0.03-0.043°(95% confidence intervals). By encompassing all these factors the technique facilitates self alignment from an unaligned position on the visual axis to an aligned position on the optical axis. The consequence of this is that ophthalmic measurements, specifically pachymetric measurements, can be made in the absence of an optician, allowing the use of ophthalmic instrumentation and measurements in health professions other than vision care

An investigation into the use of Laser Speckle Interferometry for the analysis of corneal deformation with relation to biomechanics

There has been widespread interest in corneal biomechanics over recent years, driven largely by the advancements in, and the popularity of refractive surgery techniques and subsequent concerns over their safety. Lately there has been interest into whether crosslinking, which is currently used for the treatment of keratoconus, could be developed as a minimally invasive technique to change the refractive power of the cornea by selectively changing the corneal biomechanics in specific regions to induce a shape change. Successful application of this technique requires a detailed understanding of corneal biomechanics and so far, little is known about the biomechanics of this complex tissue. The current lack of understanding can be mostly attributed to the absence of a suitable measurement technique capable of examining the dynamic behaviour of the cornea under physiological loading conditions. This thesis describes the development of a novel full-field, ex vivo, measurement method incorporating speckle interferometric techniques, to examine the biomechanics of the cornea before and after crosslinking in response to hydrostatic pressure fluctuations representative of those that occur in vivo during the cardiac cycle. The eventual measurement system used for the experiments detailed in this thesis incorporated; an Electronic Speckle Pattern Interferometer (ESPI), a Lateral Shearing Interferometer (LSI) and a fringe projection shape measurement system. The combination of these systems enabled the 3-dimensional components of surface displacement and the 1st derivative of surface displacement to be determined in response to small pressure fluctuations up to 1 mmHg in magnitude. The use of both ESPI and LSI together also enabled the applicability of LSI for measurement of non-flat surfaces to be assessed, and limitations and error sources to be identified throughout this work. To enable the measurement of corneal biomechanics, part of this thesis was concerned with the design of a bespoke loading rig. A chamber was designed that could accommodate tissue of both porcine and human origin. This chamber was linked to a hydraulic loading rig, whereby the cornea could be held at a baseline pressure representative of normal intraocular pressure and small pressure variations could be introduced by the automated vertical movement of the reservoir supplying the chamber. Experiments were conducted on a range of non-biological samples with both flat and curved surface topography, and both uniform and non-uniform mechanical properties, to determine if the measurement configuration was giving the expected measurement data and the loading rig was stable and repeatable. Following experiments on non-biological samples, a range of experiments were conducted on porcine corneas to develop a suitable testing methodology and address some of the challenges associated with corneal measurement, including transparency and hydration instability. During these investigations, a suitable surface coating was identified to generate an adequate return signal from the corneal surface, while not interfering with the response. Alongside this, the natural variation in the response of the cornea was investigated over the total experimental time, and a range of data was presented on corneas before and after crosslinking, which confirmed the suitability of the measurement methods for the assessment of crosslinking. Ultimately, a small sample size of six human corneas were investigated before and after crosslinking in specific topographic locations. From the experiments on human and porcine corneas, full-field maps of surface deformation have been presented, and a compliant region incorporating the peripheral and limbal areas has been identified as being fundamental to the response of the cornea to small pressure fluctuations. In addition to this, the regional effects of crosslinking in four different topographic locations on corneal biomechanics have been evaluated. From this, it has been demonstrated that crosslinking in specific regions in isolation can influence the way the cornea deforms to physiological-scale fluctuations in hydrostatic pressure and this could have implications for refractive correction