Measuring gaze and pupil in the real world: object-based attention,3D eye tracking and applications

This dissertation contains studies on visual attention, as measured by gaze orientation, and the use of mobile eye-tracking and pupillometry in applications. It combines the development of methods for mobile eye-tracking (studies II and III) with experimental studies on gaze guidance and pupillary responses in patients (studies IV and VI) and healthy observers (studies I and V). Object based attention / Study I What is the main factor of fixation guidance in natural scenes? Low-level features or objects? We developed a fixation-predicting model, which regards preferred viewing locations (PVL) per object and combines these distributions over the entirety of existing objects in the scene. Object-based fixation predictions for natural scene viewing perform at par with the best early salience model, that are based on low-level features. However, when stimuli are manipulated so that low-level features and objects are dissociated, the greater prediction power of saliency models diminishes. Thus, we dare to claim, that highly developed saliency models implicitly obtain object-hood and that fixation selection is mainly influenced by objects and much less by low-level features. Consequently, attention guidance in natural scenes is object-based. 3D tracking / Study II The second study focussed on improving calibration procedures for eye-in-head positions with a mobile eye-tracker.We used a mobile eye-tracker prototype, the EyeSeeCam with a high video-oculography (VOG) sampling rate and the technical gadget to follow the users gaze direction instantaneously with a rotatable camera. For a better accuracy in eye-positioning, we explored a refinement in the implementation of the eye-in-head calibration that yields a measure for fixation distance, which led to a mobile eye-tracker 3D calibration. Additionally, by developing the analytical mechanics for parametrically reorienting the gaze-centred camera, the 3D calibration could be applied to reliably record gaze-centred videos. Such videos are suitable as stimuli for investigating gaze-behaviour during object manipulation or object recognition in real worlds point-of-view (PoV) perspective. In fact, the 3D calibration produces a higher accuracy in positioning the gaze-centred camera over the whole 3D visual range. Study III, eye-tracking methods With a further development on the EyeSeeCam we achieved to record gaze-in-world data, by superposing eye-in-head and head-in-world coordinates. This novel approach uses a combination of few absolute head-positions extracted manually from the PoV video and of relative head-shifts integrated over angular velocities and translational accelerations, both given by an inertia measurement unit (IMU) synchronized to the VOG data. Gaze-in-world data consist of room-referenced gaze directions and their origins within the environment. They easily allow to assign fixation targets by using a 3D model of the measuring environment – a strong rationalisation regarding fixation analysis. Applications Study III Daylight is an important perceptual factor for visual comfort, but can also create discomfort glare situations during office work, so we developed to measure its behavioural influences. We achieve to compare luminance distributions and fixations in a real-world setting, by also recording indoor luminance variations time-resolved using luminance maps of a scenery spanning over a 3pi sr. Luminance evaluations in the workplace environment yield a well controlled categorisation of different lighting conditions and a localisation as well as a brightness measure of glare sources.We used common tasks like reading, typing on a computer, a phone call and thinking about a subject. The 3D model gives the possibility to test for gaze distribution shifts in the presence of glare patches and for variations between lighting conditions. Here, a low contrast lighting condition with no sun inside and a high contrast lighting condition with direct sunlight inside were compared. When the participants are not engaged in any visually focused task and the presence of the task support is minimal, the dominant view directions are inclined towards the view outside the window under the low contrast lighting conditions, but this tendency is less apparent and sways more towards the inside of the room under the high contrast lighting condition. This result implicates an avoidance of glare sources in gaze behaviour. In a second more extensive series of experiments, the participants’ subjective assessments of the lighting conditions will be included. Thus, the influence of glare can be analysed in more detail and tested whether visual discomfort judgements are correlated in differences in gaze-behaviour. Study IV The advanced eye-tracker calibration found application in several following projects and included in this dissertation is an investigation with patients suffering either from idiopathic Parkinson’s disease or from progressive supranuclear palsy (PSP) syndrome. PSP’s key symptom is the decreased ability to carry out vertical saccades and thus the main diagnostic feature for differentiating between the two forms of Parkinson’s syndrome. By measuring ocular movements during a rapid (< 20s) procedure with a standardized fixation protocol, we could successfully differentiate pre-diagnosed patients between idiopathic Parkinson’s disease and PSP, thus between PSP patients and HCs too. In PSP patients, the EyeSeeCam detected prominent impairment of both saccade velocity and amplitude. Furthermore, we show the benefits of a mobile eye-tracking device for application in clinical practice. Study V Decision-making is one of the basic cognitive processes of human behaviours and thus, also evokes a pupil dilation. Since this dilation reflects a marker for the temporal occurrence of the decision, we wondered whether individuals can read decisions from another’s pupil and thus become a mentalist. For this purpose, a modified version of the rock-paper-scissors childhood game was played with 3 prototypical opponents, while their eyes were video taped. These videos served as stimuli for further persons, who competed in rock-paper-scissors. Our results show, that reading decisions from a competitor’s pupil can be achieved and players can raise their winning probability significantly above chance. This ability does not require training but the instruction, that the time of maximum pupil dilation was indicative of the opponent’s choice. Therefore we conclude, that people could use the pupil to detect cognitive decisions in another individual, if they get explicit knowledge of the pupil’s utility. Study VI For patients with severe motor disabilities, a robust mean of communication is a crucial factor for well-being. Locked-in-Syndrome (LiS) patients suffer from quadriplegia and lack the ability of articulating their voice, though their consciousness is fully intact. While classic and incomplete LiS allows at least voluntary vertical eye movements or blinks to be used for communication, total LiS patients are not able to perform such movements. What remains, are involuntarily evoked muscle reactions, like it is the case with the pupillary response. The pupil dilation reflects enhanced cognitive or emotional processing, which we successfully observed in LiS patients. Furthermore, we created a communication system based on yes-no questions combined with the task of solving arithmetic problems during matching answer intervals, that yet invokes the most solid pupil dilation usable on a trial-by-trial basis for decoding yes or no as answers. Applied to HCs and patients with various severe motor disabilities, we provide the proof of principle that pupil responses allow communication for all tested HCs and 4/7 typical LiS patients. Résumé Together, the methods established within this thesis are promising advances in measuring visual attention allocation with 3D eye-tracking in real world and in the use of pupillometry as on-line measurement of cognitive processes. The two most outstanding findings are the possibility to communicate with complete LiS patients and further a conclusive evidence that objects are the primary unit of fixation selection in natural scenes

Simultaneous Wireless Information and Power Transfer in 5G communication

Green communication technology is expected to be widely adopted in future generation networks to improve energy efficiency and reliability of wireless communication network. Among the green communication technologies,simultaneous wireless information and power transfer (SWIPT) is adopted for its flexible energy harvesting technology through the radio frequency (RF) signa lthati sused for information transmission. Even though existing SWIPT techniques are flexible and adoptable for the wireless communication networks, the power and time resources of the signal need to be shared between infor- mation transmission and RF energy harvesting, and this compromises the quality of the signal. Therefore,SWIP Ttechniques need to be designed to allow an efficient resource allocation for communication and energy harvesting. The goal oft his thesisis to design SWIP Ttechniques that allow efficient,reliable and secure joint communications and power transference. A problem associated to SWIPT techniques combined with multi carrier signals is that the increased power requirements inherent to energy harvesting purposes can exacerbate nonlinear distortion effects at the transmitter. Therefore, we evaluate nonlinear distortion and present feasible solutions to mitigate the impact of nonlinear distortion effects on the performance.Another goal of the thesisis to take advantage of the energy harvesting signals in SWIP Ttechniques for channel estimation and security purposes.Theperformance of these SWIPT techniques is evaluated analytically, and those results are validated by simulations. It is shownthatthe proposed SWIPT schemes can have excellent performance, out performing conventional SWIPT schemes.Espera-se que aschamadas tecnologiasde green communications sejam amplamente ado- tadas em futuras redes de comunicação sem fios para melhorar a sua eficiência energética a fiabilidade.Entre estas,encontram-se as tecnologias SWIPT (Simultaneous Wireless Information and Power Transference), nas quais um sinal radio é usado para transferir simultaneamente potência e informações.Embora as técnicas SWIPT existentes sejam fle- xíveis e adequadas para as redes de comunicações sem fios, os recursos de energia e tempo do sinal precisam ser compartilhados entre a transmissão de informações e de energia, o que pode comprometer a qualidade do sinal. Deste modo,as técnicas SWIPT precisam ser projetadas para permitir uma alocação eficiente de recursos para comunicação e recolha de energia. O objetivo desta tese é desenvolver técnicas SWIPT que permitam transferência de energia e comunicações eficientes,fiáveis e seguras.Um problema associado às técnicas SWIPT combinadas com sinais multi-portadora são as dificuldades de amplificação ine- rentes à combinação de sinais de transmissão de energia com sinais de transferência de dados, que podem exacerbar os efeitos de distorção não-linear nos sinais transmitidos. Deste modo, um dos objectivos desta tese é avaliar o impacto da distorção não-linear em sinais SWIPT, e apresentar soluções viáveis para mitigar os efeitos da distorção não-linear no desempenho da transmissão de dados.Outro objetivo da tese é aproveitar as vantagens dos sinais de transferência de energia em técnicas SWIPT para efeitos de estimação de canal e segurança na comunicação.Os desempenhos dessas técnicas SWIPT são avaliados analiticamente,sendo os respectivos resultados validados por simulações.É mostrado que os esquemas SWIPT propostos podem ter excelente desempenho, superando esquemas SWIPT convencionais

Assessment of Utricular Nerve, Hair Cell and Mechanical Function, in vivo.

Vestibular research currently relies on single response measures such as ex vivo hair cell and in vivo single unit recordings. Although these methods allow detailed insight into the response properties of individual vestibular hair cells and neurons, they do not provide a holistic understanding of peripheral vestibular functioning and its relationship to vestibular pathology in a living system. For this to take place, in vivo recordings of peripheral vestibular nerve, hair cell and mechanical function are needed. The previous inability to record vestibular hair cell responses stemmed from a difficulty in accessing the vestibular end-organs and stimulating them in isolation of the cochlea. To circumvent this, we developed a ventral surgical approach, removing the cochlea, to provide full access to the basal surface of the utricular macula. This allowed functional and mechanical utricular hair cell recordings, alongside gross utricular nerve responses. Recordings were performed in anaesthetized guinea pigs using Bone Conducted Vibration (BCV) and Air Conducted Sound (ACS) stimuli, providing a clinical link to vestibular reflex testing. We have thus far performed experiments involving: 1) Selective manipulation of vestibular nerve function, using electrical stimulation of the central vestibular system. 2) Glass micropipette recordings from the basal surface of the macular epithelium, which provided a robust and localized measure of extracellular utricular hair cell function. 3) With the macular exposed, we have measured the dynamic motion of the macula using Laser Doppler Vibrometry, which was recorded alongside the hair cell and nerve response recordings. 4) We have used physiological and pharmacological experimental manipulations to selectively modulate utricular nerve, hair cell or mechanical function, demonstrating the ability to differentially diagnose the basis of peripheral vestibular disorders in the mammalian utricle. These tools allow for a more complete understanding of peripheral vestibular function and a first order perspective into clinical disorders effecting the otoliths

Resting-state functional MRI in multicenter studies on multiple sclerosis: a report on raw data quality and functional connectivity features from the Italian Neuroimaging Network Initiative

The Italian Neuroimaging Network Initiative (INNI) is an expanding repository of brain MRI data from multiple sclerosis (MS) patients recruited at four Italian MRI research sites. We describe the raw data quality of resting-state functional MRI (RS-fMRI) time-series in INNI and the inter-site variability in functional connectivity (FC) features after unified automated data preprocessing. MRI datasets from 489 MS patients and 246 healthy control (HC) subjects were retrieved from the INNI database. Raw data quality metrics included temporal signal-to-noise ratio (tSNR), spatial smoothness (FWHM), framewise displacement (FD), and differential variation in signals (DVARS). Automated preprocessing integrated white-matter lesion segmentation (SAMSEG) into a standard fMRI pipeline (fMRIPrep). FC features were calculated on pre-processed data and harmonized between sites (Combat) prior to assessing general MS-related alterations. Across centers (both groups), median tSNR and FWHM ranged from 47 to 84 and from 2.0 to 2.5, and median FD and DVARS ranged from 0.08 to 0.24 and from 1.06 to 1.22. After preprocessing, only global FC-related features were significantly correlated with FD or DVARS. Across large-scale networks, age/sex/FD-adjusted and harmonized FC features exhibited both inter-site and site-specific inter-group effects. Significant general reductions were obtained for somatomotor and limbic networks in MS patients (vs. HC). The implemented procedures provide technical information on raw data quality and outcome of fully automated preprocessing that might serve as reference in future RS-fMRI studies within INNI. The unified pipeline introduced little bias across sites and appears suitable for multisite FC analyses on harmonized network estimates

Analysis of Cell-Specific Heat Shock Transcriptomes in the Inner Ear

Ototoxic drug-induced death of sensory hair cells in the inner ear results in permanent hearing loss. There is currently no therapy available for the millions of patients who experience hearing loss after treatment with lifesaving drugs that have ototoxic side effects. An understanding of protective pathways in the inner ear is necessary for the development of therapies to mitigate ototoxic hearing loss. The induction of heat shock proteins (HSPs) protects hair cells from ototoxic death in vitro and in animal models. Furthermore, the protective effect of heat shock is non-cell autonomous, and the glial-like supporting cells in the inner ear may secrete the heat shock protein HSP70 to protect hair cells. This study elucidates the cell type-specific transcriptional responses in hair cells and supporting cells at steady-state and following heat shock. Two techniques, 1) massively-parallel RNA sequencing (RNA-Seq) of affinity-tagged, ribosomally-immunoprecipitated RNA (RiboTag), and 2) RNA-Seq single-cell profiling were used to accomplish this goal. Results using both techniques show effective isolation of cell type-specific transcriptomes, which were validated using orthogonal experimental techniques. Differential gene expression analysis revealed that the heat shock response is transcriptionally present in both cell types, and that hair cells may have reduced translational capacity to mount a heat shock response at the protein level relative to supporting cells. This study also utilized the transcriptional profile of heat shock in whole inner ear tissue (from RNA-Seq data) to bioinformatically predict compounds that would mimic the transcriptional response to heat chock in the inner ear. Predicted compounds from this matching process were screened in in a zebrafish model of ototoxicity and revealed three compounds that protected hair cells. Furthermore, one of those compounds, pifithrin-µ, promoted a heat shock-like transcriptional signature and protected cultured murine inner ear tissue from ototoxic drug-induced hair cell death. In summary, this study characterizes the transcriptional profiles of inner ear hair and supporting cells and provides insight into the nature of the protective heat shock response in these cell types. Secondly, this study translates the native heat shock response of the inner ear into a pharmacologically-induced protective effect against ototoxic hair cell death, which provides a valuable workflow for future studies aimed at preventing hair cell death and hearing loss

Early β-amyloid accumulation in the brain is associated with peripheral T cell alterations

INTRODUCTION Fast and minimally invasive approaches for early diagnosis of Alzheimer's disease (AD) are highly anticipated. Evidence of adaptive immune cells responding to cerebral β-amyloidosis has raised the question of whether immune markers could be used as proxies for β-amyloid accumulation in the brain. METHODS Here, we apply multidimensional mass-cytometry combined with unbiased machine-learning techniques to immunophenotype peripheral blood mononuclear cells from a total of 251 participants in cross-sectional and longitudinal studies. RESULTS We show that increases in antigen-experienced adaptive immune cells in the blood, particularly CD45RA-reactivated T effector memory (TEMRA) cells, are associated with early accumulation of brain β-amyloid and with changes in plasma AD biomarkers in still cognitively healthy subjects. DISCUSSION Our results suggest that preclinical AD pathology is linked to systemic alterations of the adaptive immune system. These immunophenotype changes may help identify and develop novel diagnostic tools for early AD assessment and better understand clinical outcomes