Exploring cognition in visual search and vigilance tasks with eye tracking and pupillometry

Recent findings in experimental psychology suggest that pupillometry, the measurement of pupil size, can provide insight into cognitive processes associated with effort and target detection in visual search tasks and monitoring performance in vigilance tasks. With the increasing availability, affordability and flexibility of video-based eye tracking hardware, these experimental findings point to lucrative practical applications such as real-time biobehavioural monitoring systems to assist with socially important tasks in operational settings. The aim of the current thesis was to explore this potential with further experimental work paying close attention to methodological issues which complicate cognitive interpretations of pupillary responses, such as physical stimulus confounds and eye movement-related measurement error in video-based systems. Six original experiments were designed to specifically explore the relationship between pupil size, cognition and behavioural performance in classic visual search and vigilance paradigms. Experiments 1-2 examined the pupillometric effects of effort and target detection in visual search with briefly presented stimuli. Pupil responses showed small variability with respect to manipulations of set size and target presence but were influenced substantially by the requirement for a motor response. Experiments 3-4 explored the cognitive pupil dynamics of free-viewing visual search with data-driven correction for eye movement artefacts. Group-level averages revealed small transient pupil dilations following fixations on targets but not distractors, an effect which was not contingent on a motor response or correction for gaze position artefacts. Experiments 5-6 looked at the relationship between pupil size and detection performance measures in two types of vigilance task. Changes in baseline and stimulus-evoked pupil responses loosely mirrored changes in performance, but the relationships were neither linear nor consistent. Overall, the thesis affirms the practical potential for using cognitive pupillometry in research and applied settings, but emphasises the constraints arising from methodological and theoretical limitations

Simultaneous pupillometry and functional Magnetic Resonance Imaging (fMRI) for the detection of stress-related endophenotypes

Mental diseases constitute a core health challenge of the 21st century. To date, diagnostics in psychiatry have been primarily based on subjective self-reports, largely bypassing the biological underpinnings and phenotypic heterogeneity of psychiatric disorders. As an effort to implement a more biologically valid classification of mental disorders, recent initiatives like the Research Domain Criteria (RDoC) project aim to identify endophenotypes that reflect transdiagnostic core mechanisms of psychiatric disorders. Stress is known to play a fundamental role in the development of mood and anxiety disorders. One key system involved in the physiological response to stress is the brainstem?s noradrenergic (NA) arousal center located in the locus coeruleus (LC), and previous studies indicate that pupil size provides an indirect index for activity of the LC-NA system. In order to investigate the relationship between spontaneous drifts in autonomic arousal and global brain activity in healthy human subjects, we first determined the fMRI correlates of spontaneous pupil fluctuations during the resting state. We found that pupil dilations are strongly coupled to activation of the dorsal anterior cingulate cortex (dACC) and bilateral insula (salience network [SN]). To assess whether this link between the pupil and the SN would also extend to emotional arousal, we next investigated the neural correlates of reward anticipation-induced pupil dilations in healthy subjects. Here, we could show that a cue signaling the possibility to receive a monetary reward evoked strong pupil dilations, the magnitude of which predicted response time to a target cue. Again, pupil dilations were strongly linked to SN activation. Furthermore, our results suggest that pupillometry is helpful to dissect different phases of reward anticipation and associated brain activity, disentangling reward prediction, arousal modulation and attentionrelated processes. These observations led us to the conclusion that the SN modulates arousal levels to optimize task performance, that is, to counteract drowsiness/ transitions to sleep during the resting state and to facilitate reward-directed behaviors in the reward anticipation task. Taken together, pupillometry appears to provide a reliable index for activity of the SN, a core network related to psychiatric disorders, making it a promising tool for the detection of stress-related endophenotypes

Pupillometry and the vigilance decrement: Task‐evoked but not baseline pupil measures reflect declining performance in visual vigilance tasks

Baseline and task-evoked pupil measures are known to reflect the activity of the nervous system's central arousal mechanisms. With the increasing availability, affordability and flexibility of video-based eye tracking hardware, these measures may one day find practical application in real-time biobehavioural monitoring systems to assess performance or fitness for duty in tasks requiring vigilant attention. But real-world vigilance tasks are predominantly visual in their nature and most research in this area has taken place in the auditory domain. Here, we explore the relationship between pupil size—both baseline and task-evoked—and behavioural performance measures in two novel vigilance tasks requiring visual target detection: (1) a traditional vigilance task involving prolonged, continuous and uninterrupted performance (n = 28) and (2) a psychomotor vigilance task (n = 25). In both tasks, behavioural performance and task-evoked pupil responses declined as time spent on task increased, corroborating previous reports in the literature of a vigilance decrement with a corresponding reduction in task-evoked pupil measures. Also in line with previous findings, baseline pupil size did not show a consistent relationship with performance measures. Our data offer novel insights into the complex interplay of brain systems involved in vigilant attention and question the validity of the assumption that baseline (prestimulus) pupil size and task-evoked (poststimulus) pupil measures reflect the tonic and phasic firing modes of the locus coeruleus

A closer look at the timecourse of mind wandering: Pupillary responses and behaviour

Mind wandering (MW) refers to the shift of attention away from a primary task and/or external environment towards thoughts unrelated to the task. Recent evidence has shown that pupillometry can be used as an objective marker of the onset and maintenance of externally-driven MW episodes. In the present study we aimed to further investigate pupillary changes associated with the onset and duration of self-reported MW episodes. We used a modified version of the joint behavioural-pupillometry paradigm we recently introduced. Participants were asked to perform a monotonous vigilance task which was intermixed with task-irrelevant cue-phrases (visually presented verbal cues); they were instructed to interrupt the task whenever a thought came to mind (self-caught method) and to indicate the trigger of their thought, if any. We found systematic pupil dilation after the presentation of verbal cues reported to have triggered MW, compared with other verbal cues presented during a supposedly on-task period (i.e., the period immediately following the resuming of the task after a self-caught interruption and MW report). These results confirm that pupil diameter is sensitive to the changes associated with the onset of MW and its unfolding over time. Moreover, by computing the latency between the trigger presentation and the task interruption (self-catch), we could also estimate the duration of MW episodes triggered by verbal cues. However, a high variability was found, implying very large inter-event variability, which could not be explained by any of the MW properties we acquired (including: temporal focus, specificity, emotional valence). Our behavioural and pupillometry findings stress the need for objective measures about the temporal unfolding of MW (while most studies focus on arbitrary time-window preceding self-reports of MW)

Physiological Measurements for Real-time Fatigue Monitoring in Train Drivers: Review of the State of the Art and Reframing the Problem

The impact of fatigue on train drivers is one of the most important safety-critical issues in rail. It affects drivers’ performance, significantly contributing to railway incidents and accidents. To address the issue of real-time fatigue detection in drivers, most reliable and applicable psychophysiological indicators of fatigue need to be identified. Hence, this paper aims to examine and present the current state of the art in physiological measures for real-time fatigue monitoring that could be applied in the train driving context. Three groups of such measures are identified: EEG, eye-tracking and heart-rate measures. This is the first paper to provide the analysis and review of these measures together on a granular level, focusing on specific variables. Their potential application to monitoring train driver fatigue is discussed in respective sections. A summary of all variables, key findings and issues across these measures is provided. An alternative reconceptualization of the problem is proposed, shifting the focus from the concept of fatigue to that of attention. Several arguments are put forward in support of attention as a better-defined construct, more predictive of performance decrements than fatigue, with serious ramifications on human safety. Proposed reframing of the problem coupled with the detailed presentation of findings for specific relevant variables can serve as a guideline for future empirical research, which is needed in this field

Technologies for the Monitoring and Prevention of Driver Fatigue

A series of driving simulation pilot studies on various technologies for alertness monitoring (head position sensor, eye-gaze system), fitness-for-duty testing (two pupil-based systems), and alertness promotion (in-seat vibration system) has been conducted in Circadian Technologies’ Alertness Testbed. The results indicate that, all tested technologies show promise for monitoring/testing or preventing driver fatigue, respectively. However, particularly for fatigue monitoring, no single measure alone may be sensitive and reliable enough to quantify driver fatigue. Since alertness is a complex phenomenon, a multi-parametric approach needs to be used. Such a multi-sensor approach imposes challenges for online data interpretation. We suggest using a neural-fuzzy hybrid system for the automatic assessment of complex data streams for driver fatigue. The final system output can then be used to trigger the activation of alertness countermeasures

Portable equipment for simultaneous assays of psychomotor vigilance test and pupillometry

This paper describes a system with potential for identification of sleep deprivation, which, based on our bibliographical survey, has not yet been described in the literature. The system combines two methodologies, i. e., Psychomotor vigilance test (PVT) and pupillometry, which are among the leading methods for the study of sleep deprivation. However, due to peculiarities of both methodologies, some adaptations were made in their procedures to allow them to co-exist in the same system. Such integration may not only ensure the complementarity of indexes, making the identification of sleep deprivation more solid, but also set up the equalization of the subject's psycho-physiological state, which is not possible in tests performed with a time lag. In this study, the performance of measurements provided by the system was assessed in subjects on alert. However, some measurements present a displacement with respect to their average values, which, according to assessment, are determined by system's technical requirements. The results obtained in this assessment, combined with the increasing demand for large scale application tools, able to be used outside the limits of the laboratory environment for studies in sleep deprivation disorders, point to this system as a potential tool. However, the undertaking of a rigorous experiment is necessary to assess whether the indexes obtained by the system allow the robust identification of sleep deprivation.Neste artigo é descrito um sistema com potencial para identificar a privação do sono, que, com base no levantamento bibliográfico realizado, ainda não foi abordado na literatura. Este sistema integra simultaneamente duas metodologias, o teste de vigilância psicomotora (Psychomotor vigilance test, PVT) e a pupilometria, que se destacam no estudo da privação do sono. Entretanto, para atender às peculiaridades destas metodologias, permitindo que coexistam em um único sistema, algumas adaptações foram realizadas em seus procedimentos. Esta integração poderá garantir não só a complementariedade de indicadores que torna a identificação da privação do sono mais robusta, assim como estabelecer a equalização do estado psicofisiológico do sujeito, o que não é possível em testes realizados com defasagem temporal. Neste estudo, a validação das métricas do sistema foi realizada com sujeitos em estado de alerta. Os resultados mostraram-se coerentes com a literatura. Entretanto, algumas métricas apresentam um deslocamento em seus valores médios, que segundo as avaliações realizadas são determinadas pelas exigências técnicas do sistema. Os resultados obtidos nesta avaliação, somados à crescente demanda de ferramentas de aplicação em larga escala e que possam ser utilizadas além dos limites laboratoriais para estudos em distúrbios e privação do sono, apontam este sistema como uma potencial ferramenta. Entretanto, será necessário o estabelecimento de um experimento rigoroso, para avaliar se os indicadores oriundos das métricas do sistema permitem a identificação robusta da privação do sono.Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)Fundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG)Universidade Federal de Minas Gerais Engenharia ElétricaUniversidade Federal de São Paulo (UNIFESP) Departamento de PsicobiologiaUNIFESP, Depto. de PsicobiologiaSciEL

Using event-related fMRI to examine sustained attention processes and effects of APOE ε4 in young adults

In this study we investigated effects of the APOE ε4 allele (which confers an enhanced risk of poorer cognitive ageing, and Alzheimer’s Disease) on sustained attention (vigilance) performance in young adults using the Rapid Visual Information Processing (RVIP) task and event-related fMRI. Previous fMRI work with this task has used block designs: this study is the first to image an extended (6-minute) RVIP task. Participants were 26 carriers of the APOE ε4 allele, and 26 non carriers (aged 18–28). Pupil diameter was measured throughout, as an index of cognitive effort. We compared activity to RVIP task hits to hits on a control task (with similar visual parameters and response requirements but no working memory load): this contrast showed activity in medial frontal, inferior and superior parietal, temporal and visual cortices, consistent with previous work, demonstrating that meaningful neural data can be extracted from the RVIP task over an extended interval and using an event-related design. Behavioural performance was not affected by genotype; however, a genotype by condition (experimental task/control task) interaction on pupil diameter suggested that ε4 carriers deployed more effort to the experimental compared to the control task. fMRI results showed a condition by genotype interaction in the right hippocampal formation: only ε4 carriers showed downregulation of this region to experimental task hits versus control task hits. Experimental task beta values were correlated against hit rate: parietal correlations were seen in ε4 carriers only, frontal correlations in non-carriers only. The data indicate that, in the absence of behavioural differences, young adult ε4 carriers already show a different linkage between functional brain activity and behaviour, as well as aberrant hippocampal recruitment patterns. This may have relevance for genotype differences in cognitive ageing trajectories