LIMBUSTRACK: STABLE EYE-TRACKING IN IMPERFECT LIGHT CONDITIONS

We are aware of only one serious effort at development of a cheap, accurate, wearable eye tracker: the open source openEyes project. However, its method of ocular feature detection is such that it is prone to failure in variable lighting conditions. To address this deficiency, we have developed a cheap wearable eye tracker. At the heart of our development are novel techniques that allow operation under variable illumination

High-Density Diffuse Optical Tomography During Passive Movie Viewing: A Platform for Naturalistic Functional Brain Mapping

Human neuroimaging techniques enable researchers and clinicians to non-invasively study brain function across the lifespan in both healthy and clinical populations. However, functional brain imaging methods such as functional magnetic resonance imaging (fMRI) are expensive, resource-intensive, and require dedicated facilities, making these powerful imaging tools generally unavailable for assessing brain function in settings demanding open, unconstrained, and portable neuroimaging assessments. Tools such as functional near-infrared spectroscopy (fNIRS) afford greater portability and wearability, but at the expense of cortical field-of-view and spatial resolution. High-Density Diffuse Optical Tomography (HD-DOT) is an optical neuroimaging modality directly addresses the image quality limitations associated with traditional fNIRS techniques through densely overlapping optical measurements. This thesis aims to establish the feasibility of using HD-DOT in a novel application demanding exceptional portability and flexibility: mapping disrupted cortical activity in chronically malnourished children. I first motivate the need for dense optical measurements of brain tissue to achieve fMRI-comparable localization of brain function (Chapter 2). Then, I present imaging work completed in Cali, Colombia, where a cohort of chronically malnourished children were imaged using a custom HD-DOT instrument to establish feasibility of performing field-based neuroimaging in this population (Chapter 3). Finally, in order to meet the need for age appropriate imaging paradigms in this population, I develop passive movie viewing paradigms for use in optical neuroimaging, a flexible and rich stimulation paradigm that is suitable for both adults and children (Chapter 4)

How to Build an Embodiment Lab: Achieving Body Representation Illusions in Virtual Reality

Advances in computer graphics algorithms and virtual reality (VR) systems, together with the reduction in cost of associated equipment, have led scientists to consider VR as a useful tool for conducting experimental studies in fields such as neuroscience and experimental psychology. In particular virtual body ownership, where the feeling of ownership over a virtual body is elicited in the participant, has become a useful tool in the study of body representation, in cognitive neuroscience and psychology, concerned with how the brain represents the body. Although VR has been shown to be a useful tool for exploring body ownership illusions, integrating the various technologies necessary for such a system can be daunting. In this paper we discuss the technical infrastructure necessary to achieve virtual embodiment. We describe a basic VR system and how it may be used for this purpose, and then extend this system with the introduction of real-time motion capture, a simple haptics system and the integration of physiological and brain electrical activity recordings

An informatics system for exploring eye movements in reading

Eye tracking techniques have been widely used in many research areas including cognitive science, psychology, human-computer interaction, marketing research, medical research etc. Many computer programs have emerged to help these researchers to design experiments, present visual stimuli and process the large quantity of numerical data produced by the eye tracker. However, most applications, especially commercial products, are designed for a particular tracking device and tend to be general purpose. Few of them are designed specifically for reading research. This can be inconvenient when dealing with complex experimental design, multi-source data collection, and text based data analysis, including almost every aspect of a reading study lifecycle. A flexible and powerful system that manages the lifecycle of different reading studies is required to fulfill these demands. Therefore, we created an informatics system with two major software suites: Experiment Executor and EyeMap. It is a system designed specifically for reading research. Experiment Executor helps reading researchers build complex experimental environments, which can rapidly present display changes and support the co-registration of eye tracking information with other data collection devices such as EEG (electroencephalography) amplifiers. The EyeMap component helps researchers visualize and analysis a wide range of writing systems including spaced and unspaced scripts, which can be presented in proportional or non-proportional font types. The aim of the system is to accelerate the life cycle of a reading experiment from design through analysis. Several experiments were conducted on this system. These experiments confirmed the effectiveness and the capability of the system with several new reading research findings from the visual information processing stages of reading

SPATIO-TEMPORAL REGISTRATION IN AUGMENTED REALITY

The overarching goal of Augmented Reality (AR) is to provide users with the illusion that virtual and real objects coexist indistinguishably in the same space. An effective persistent illusion requires accurate registration between the real and the virtual objects, registration that is spatially and temporally coherent. However, visible misregistration can be caused by many inherent error sources, such as errors in calibration, tracking, and modeling, and system delay. This dissertation focuses on new methods that could be considered part of "the last mile" of spatio-temporal registration in AR: closed-loop spatial registration and low-latency temporal registration: 1. For spatial registration, the primary insight is that calibration, tracking and modeling are means to an end---the ultimate goal is registration. In this spirit I present a novel pixel-wise closed-loop registration approach that can automatically minimize registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are minimized in both global world space via camera pose refinement, and local screen space via pixel-wise adjustments. This approach is presented in the context of Video See-Through AR (VST-AR) and projector-based Spatial AR (SAR), where registration results are measurable using a commodity color camera. 2. For temporal registration, the primary insight is that the real-virtual relationships are evolving throughout the tracking, rendering, scanout, and display steps, and registration can be improved by leveraging fine-grained processing and display mechanisms. In this spirit I introduce a general end-to-end system pipeline with low latency, and propose an algorithm for minimizing latency in displays (DLP DMD projectors in particular). This approach is presented in the context of Optical See-Through AR (OST-AR), where system delay is the most detrimental source of error. I also discuss future steps that may further improve spatio-temporal registration. Particularly, I discuss possibilities for using custom virtual or physical-virtual fiducials for closed-loop registration in SAR. The custom fiducials can be designed to elicit desirable optical signals that directly indicate any error in the relative pose between the physical and projected virtual objects.Doctor of Philosoph

Final Report to NSF of the Standards for Facial Animation Workshop

The human face is an important and complex communication channel. It is a very familiar and sensitive object of human perception. The facial animation field has increased greatly in the past few years as fast computer graphics workstations have made the modeling and real-time animation of hundreds of thousands of polygons affordable and almost commonplace. Many applications have been developed such as teleconferencing, surgery, information assistance systems, games, and entertainment. To solve these different problems, different approaches for both animation control and modeling have been developed

The Motion-Lab: A Virtual Reality Laboratory for Spatial Updating Experiments

The main question addressed in the Motion-Lab is: How do we know where we are? Normally, humans know where they are with respect to the immediate surround. The overall perception of this environment results from the integration of multiple sensory modalities. Here we use Virtual Reality to study the interaction of visual, vestibular, and proprioceptive senses and explore the way these senses might be integrated into a coherent perception of spatial orientation and location. This Technical Report describes a Virtual Reality laboratory, its technical implementation as a distributed network of computers and discusses its usability for experiments designed to investigate questions of spatial orientation

Estimating and understanding motion : from diagnostic to robotic surgery

Estimating and understanding motion from an image sequence is a central topic in computer vision. The high interest in this topic is because we are living in a world where many events that occur in the environment are dynamic. This makes motion estimation and understanding a natural component and a key factor in a widespread of applications including object recognition , 3D shape reconstruction, autonomous navigation and medica! diagnosis. Particularly, we focus on the medical domain in which understanding the human body for clinical purposes requires retrieving the organs' complex motion patterns, which is in general a hard problem when using only image data. In this thesis, we cope with this problem by posing the question - How to achieve a realistic motion estimation to offer a better clinical understanding? We focus this thesis on answering this question by using a variational formulation as a basis to understand one of the most complex motions in the human's body, the heart motion, through three different applications: (i) cardiac motion estimation for diagnostic, (ii) force estimation and (iii) motion prediction, both for robotic surgery. Firstly, we focus on a central topic in cardiac imaging that is the estimation of the cardiac motion. The main aim is to offer objective and understandable measures to physicians for helping them in the diagnostic of cardiovascular diseases. We employ ultrafast ultrasound data and tools for imaging motion drawn from diverse areas such as low-rank analysis and variational deformation to perform a realistic cardiac motion estimation. The significance is that by taking low-rank data with carefully chosen penalization, synergies in this complex variational problem can be created. We demonstrate how our proposed solution deals with complex deformations through careful numerical experiments using realistic and simulated data. We then move from diagnostic to robotic surgeries where surgeons perform delicate procedures remotely through robotic manipulators without directly interacting with the patients. As a result, they lack force feedback, which is an important primary sense for increasing surgeon-patient transparency and avoiding injuries and high mental workload. To solve this problem, we follow the conservation principies of continuum mechanics in which it is clear that the change in shape of an elastic object is directly proportional to the force applied. Thus, we create a variational framework to acquire the deformation that the tissues undergo due to an applied force. Then, this information is used in a learning system to find the nonlinear relationship between the given data and the applied force. We carried out experiments with in-vivo and ex-vivo data and combined statistical, graphical and perceptual analyses to demonstrate the strength of our solution. Finally, we explore robotic cardiac surgery, which allows carrying out complex procedures including Off-Pump Coronary Artery Bypass Grafting (OPCABG). This procedure avoids the associated complications of using Cardiopulmonary Bypass (CPB) since the heart is not arrested while performing the surgery on a beating heart. Thus, surgeons have to deal with a dynamic target that compromisetheir dexterity and the surgery's precision. To compensate the heart motion, we propase a solution composed of three elements: an energy function to estimate the 3D heart motion, a specular highlight detection strategy and a prediction approach for increasing the robustness of the solution. We conduct evaluation of our solution using phantom and realistic datasets. We conclude the thesis by reporting our findings on these three applications and highlight the dependency between motion estimation and motion understanding at any dynamic event, particularly in clinical scenarios.L’estimació i comprensió del moviment dins d’una seqüència d’imatges és un tema central en la visió per ordinador, el que genera un gran interès perquè vivim en un entorn ple d’esdeveniments dinàmics. Per aquest motiu és considerat com un component natural i factor clau dins d’un ampli ventall d’aplicacions, el qual inclou el reconeixement d’objectes, la reconstrucció de formes tridimensionals, la navegació autònoma i el diagnòstic de malalties. En particular, ens situem en l’àmbit mèdic en el qual la comprensió del cos humà, amb finalitats clíniques, requereix l’obtenció de patrons complexos de moviment dels òrgans. Aquesta és, en general, una tasca difícil quan s’utilitzen només dades de tipus visual. En aquesta tesi afrontem el problema plantejant-nos la pregunta - Com es pot aconseguir una estimació realista del moviment amb l’objectiu d’oferir una millor comprensió clínica? La tesi se centra en la resposta mitjançant l’ús d’una formulació variacional com a base per entendre un dels moviments més complexos del cos humà, el del cor, a través de tres aplicacions: (i) estimació del moviment cardíac per al diagnòstic, (ii) estimació de forces i (iii) predicció del moviment, orientant-se les dues últimes en cirurgia robòtica. En primer lloc, ens centrem en un tema principal en la imatge cardíaca, que és l’estimació del moviment cardíac. L’objectiu principal és oferir als metges mesures objectives i comprensibles per ajudar-los en el diagnòstic de les malalties cardiovasculars. Fem servir dades d’ultrasons ultraràpids i eines per al moviment d’imatges procedents de diverses àrees, com ara l’anàlisi de baix rang i la deformació variacional, per fer una estimació realista del moviment cardíac. La importància rau en que, en prendre les dades de baix rang amb una penalització acurada, es poden crear sinergies en aquest problema variacional complex. Mitjançant acurats experiments numèrics, amb dades realístiques i simulades, hem demostrat com les nostres propostes solucionen deformacions complexes. Després passem del diagnòstic a la cirurgia robòtica, on els cirurgians realitzen procediments delicats remotament, a través de manipuladors robòtics, sense interactuar directament amb els pacients. Com a conseqüència, no tenen la percepció de la força com a resposta, que és un sentit primari important per augmentar la transparència entre el cirurgià i el pacient, per evitar lesions i per reduir la càrrega de treball mental. Resolem aquest problema seguint els principis de conservació de la mecànica del medi continu, en els quals està clar que el canvi en la forma d’un objecte elàstic és directament proporcional a la força aplicada. Per això hem creat un marc variacional que adquireix la deformació que pateixen els teixits per l’aplicació d’una força. Aquesta informació s’utilitza en un sistema d’aprenentatge, per trobar la relació no lineal entre les dades donades i la força aplicada. Hem dut a terme experiments amb dades in-vivo i ex-vivo i hem combinat l’anàlisi estadístic, gràfic i de percepció que demostren la robustesa de la nostra solució. Finalment, explorem la cirurgia cardíaca robòtica, la qual cosa permet realitzar procediments complexos, incloent la cirurgia coronària sense bomba (off-pump coronary artery bypass grafting o OPCAB). Aquest procediment evita les complicacions associades a l’ús de circulació extracorpòria (Cardiopulmonary Bypass o CPB), ja que el cor no s’atura mentre es realitza la cirurgia. Això comporta que els cirurgians han de tractar amb un objectiu dinàmic que compromet la seva destresa i la precisió de la cirurgia. Per compensar el moviment del cor, proposem una solució composta de tres elements: un funcional d’energia per estimar el moviment tridimensional del cor, una estratègia de detecció de les reflexions especulars i una aproximació basada en mètodes de predicció, per tal d’augmentar la robustesa de la solució. L’avaluació de la nostra solució s’ha dut a terme mitjançant conjunts de dades sintètiques i realistes. La tesi conclou informant dels nostres resultats en aquestes tres aplicacions i posant de relleu la dependència entre l’estimació i la comprensió del moviment en qualsevol esdeveniment dinàmic, especialment en escenaris clínics.Postprint (published version

High-Density Diffuse Optical Tomography During Passive Movie Viewing: A Platform for Naturalistic Functional Brain Mapping

Human neuroimaging techniques enable researchers and clinicians to non-invasively study brain function across the lifespan in both healthy and clinical populations. However, functional brain imaging methods such as functional magnetic resonance imaging (fMRI) are expensive, resource-intensive, and require dedicated facilities, making these powerful imaging tools generally unavailable for assessing brain function in settings demanding open, unconstrained, and portable neuroimaging assessments. Tools such as functional near-infrared spectroscopy (fNIRS) afford greater portability and wearability, but at the expense of cortical field-of-view and spatial resolution. High-Density Diffuse Optical Tomography (HD-DOT) is an optical neuroimaging modality directly addresses the image quality limitations associated with traditional fNIRS techniques through densely overlapping optical measurements. This thesis aims to establish the feasibility of using HD-DOT in a novel application demanding exceptional portability and flexibility: mapping disrupted cortical activity in chronically malnourished children. I first motivate the need for dense optical measurements of brain tissue to achieve fMRI-comparable localization of brain function (Chapter 2). Then, I present imaging work completed in Cali, Colombia, where a cohort of chronically malnourished children were imaged using a custom HD-DOT instrument to establish feasibility of performing field-based neuroimaging in this population (Chapter 3). Finally, in order to meet the need for age appropriate imaging paradigms in this population, I develop passive movie viewing paradigms for use in optical neuroimaging, a flexible and rich stimulation paradigm that is suitable for both adults and children (Chapter 4)