To look and look again: Memory accumulation across sequential eye movements and related brain activity

When we visually explore the world, our eyes are constantly moving in order to sample the information available in front of us. The timing of these movements, as well as where our gaze is directed at, can tell us a great deal about how this information is acquired. Even more can be learned from combining the study of eye movement with that of brain activity. In this dissertation, I co-registered eye movements and electrical brain activity (EEG) in free viewing behavior. Co-registration is instrumental for studying how visual memory accumulates information during free viewing. To address this issue, I developed a task, consisting of multiple target visual search followed by change detection. In a first study, I assessed eye movement parameters as possible indicators of memory load. Pupil size is known as a reliable measure of memory load during fixed-gaze viewing conditions. Hence, pupil size was a primary candidate for free viewing conditions as well. Pupil size reflected memory load only when it exceeded working memory capacity. Exhibiting a slow physiological response, pupil size also was unable to reveal instantaneous effects of memory load based on the task-relevance of the fixated item. I, therefore, also tested the effect of working memory load on fixation duration. Fixation duration increased with memory load for fixations on task-relevant (target) but not on task-irrelevant (distractor) items. Since effects of memory accumulation were observed in eye movement, an effect on brain activity could be expected. With co-registered EEG and eye movements, fixations and saccades become markers to segment EEG activity. However, a solution is needed to the problem that, in free viewing, EEG related to cognitive processes is confounded by activity related to sequential eye movements. I explored two approaches to solving this problem. The first involves selecting EEG epochs across experimental conditions while matching on eye movement parameters such as saccade size and fixation duration. When eye movement parameters are matched, effects on EEG related to eye movements are balanced across conditions, allowing us to study the effect of the cognitive process of interest. I developed a procedure for matching conditions across eye movement parameters using Mahalanobis distance. The second approach involves generalized additive mixed models (GAMM), which estimates nonlinearities in EEG in terms of smoothing splines. To obtain the estimates, eye movement parameters were used as predictors along with experimental conditions. The partial effects of the experimental condition on EEG could then be separately investigated. Using the first approach, I matched eye movement parameters across memory load conditions. Subsequent target fixations were used as criterion to distinguish increases in memory load. I compared EEG in the intervals pre-saccadic and post-saccadic to subsequent target fixations. However, no reliable EEG correlate of visual working memory load could be found. I concluded that effects of single items in working memory could have been overshadowed by more complex memory processes and memorization strategies. To elucidate these memory-encoding strategies, I analyzed refixation behavior in free viewing. To do this, two approaches were taken. In the first, traditional approach, refixations on targets and distractors were counted. Target refixations increased with memory load, whereas, distractor refixations decreased with memory load. There were more target fixations during correct responses than incorrect responses, while the opposite was seen for distractors. In the second approach, eye movement patterns were investigated using recurrence quantification analysis. Repetition of eye movement sequences increased with memory load. This result implies that along with item information, eye movements are also encoded into working memory, in agreement with scanpath theory. Overall, the study of refixation behavior revealed that oculomotor strategies are flexibly adapted to the requirements of the task. Finally, I investigated the neural mechanisms of refixation behavior. The dependence of saccade planning on task-relevance was investigated in the presaccadic potentials of first visits and refixations of items. Refixations differed from first visits to targets in their effect on planning of the subsequent saccade. No such difference was seen for distractors. These presaccadic potentials showed that task-relevance affected memory processes and consequently saccade planning. In sum, my studies have contributed to the understanding of accumulation of visual information during free viewing. This process was seen to involve complex memory strategies that are continually adapted during the task, not only affecting eye movements but also employing them for memorization.status: publishe

Combining EEG and eye movement recording in free viewing: pitfalls and possibilities

Co-registration of EEG and eye movement has promise for investigating perceptual processes in free viewing conditions, provided certain methodological challenges can be addressed. Most of these arise from the self-paced character of eye movements in free viewing conditions. Successive eye movements occur within short time intervals. Their evoked activity is likely to distort the EEG signal during fixation. Due to the non-uniform distribution of fixation durations, these distortions are systematic, survive across-trials averaging, and can become a source of confounding. We illustrate this problem with effects of sequential eye movements on the evoked potentials and time-frequency components of EEG and propose a solution based on matching of eye movement characteristics between experimental conditions. The proposal leads to a discussion of which eye movement characteristics are to be matched, depending on the EEG activity of interest. We also compare segmentation of EEG into saccade-related epochs relative to saccade and fixation onsets and discuss the problem of baseline selection and its solution. Further recommendations are given for implementing EEG-eye movement co-registration in free viewing conditions. By resolving some of the methodological problems involved, we aim to facilitate the transition from the traditional stimulus-response paradigm to the study of visual perception in more naturalistic conditions.publisher: Elsevier articletitle: Combining EEG and eye movement recording in free viewing: Pitfalls and possibilities journaltitle: Brain and Cognition articlelink: http://dx.doi.org/10.1016/j.bandc.2016.06.004 content_type: article copyright: © 2016 Elsevier Inc. All rights reserved.status: publishe

Refixation patterns reveal memory-encoding strategies in free viewing

We investigated visual working memory encoding across saccadic eye movements, focusing our analysis on refixation behavior. Over 10-s periods, participants performed a visual search for three, four, or five targets and remembered their orientations for a subsequent change-detection task. In 50% of the trials, one of the targets had its orientation changed. From the visual search period, we scored three types of refixations and applied measures for quantifying eye-fixation recurrence patterns. Repeated fixations on the same regions as well as repeated fixation patterns increased with memory load. Correct change detection was associated with more refixations on targets and less on distractors, with increased frequency of recurrence, and with longer intervals between refixations. The results are in accordance with the view that patterns of eye movement are an integral part of visual working memory representation.status: publishe

Refixation control in free viewing: a specialized mechanism divulged by eye-movement-related brain activity

In free viewing, the eyes return to previously visited locations rather frequently, even though the attentional and memory-related processes controlling eye-movement show a strong antirefixation bias. To overcome this bias, a special refixation triggering mechanism may have to be recruited. We probed the neural evidence for such a mechanism by combining eye tracking with EEG recording. A distinctive signal associated with refixation planning was observed in the EEG during the presaccadic interval: the presaccadic potential was reduced in amplitude before a refixation compared with normal fixations. The result offers direct evidence for a special refixation mechanism that operates in the saccade planning stage of eye movement control. Once the eyes have landed on the revisited location, acquisition of visual information proceeds indistinguishably from ordinary fixations. NEW & NOTEWORTHY A substantial proportion of eye fixations in human natural viewing behavior are revisits of recently visited locations, i.e., refixations. Our recently developed methods enabled us to study refixations in a free viewing visual search task, using combined eye movement and EEG recording. We identified in the EEG a distinctive refixation-related signal, signifying a control mechanism specific to refixations as opposed to ordinary eye fixations.status: publishe

Fixation duration surpasses pupil size as a measure of memory load in free viewing

Oculomotor behavior reveals, not only the acquisition of visual information at fixation, but also the accumulation of information in memory across subsequent fixations. Two candidate measures were considered as indicators of such dynamic visual memory load: fixation duration and pupil size. While recording these measures, we displayed an arrangement of 3, 4 or 5 targets among distractors. Both occurred in various orientations. Participants searched for targets and reported whether in a subsequent display one of them had changed orientation. We determined to what extent fixation duration and pupil size indicated dynamic memory load, as a function of the number of targets fixated during the search. We found that fixation duration reflects the number of targets, both when this number is within and above the limit of working memory capacity. Pupil size reflects the number of targets only when it exceeds the capacity limit. Moreover, the duration of fixations on successive targets but not on distractors increases whereas pupil size does not. The increase in fixation duration with number of targets both within and above working memory capacity suggests that in free viewing fixation duration is sensitive to actual memory load as well as to processing load, whereas pupil size is indicative of processing load only. Two alternative models relating visual attention and working memory are considered relevant to these results. We discuss the results as supportive of a model which involves a temporary buffer in the interaction of attention and working memory.status: publishe

Neural correlates of task-related refixation behavior

Eye movement research has shown that attention shifts from the currently fixated location to the next before a saccade is executed. We investigated whether the cost of the attention shift depends on higher-order processing at the time of fixation, in particular on visual working memory load differences between fixations and refixations on task-relevant items. The attention shift is reflected in EEG activity in the saccade-related potential (SRP). In a free viewing task involving visual search and memorization of multiple targets amongst distractors, we compared the SRP in first fixations versus refixations on targets and distractors. The task-relevance of targets implies that more information will be loaded in memory (e.g. both identity and location) than for distractors (e.g. location only). First fixations will involve greater memory load than refixations, since first fixations involve loading of new items, while refixations involve rehearsal of previously visited items. The SRP in the interval preceding the saccade away from a target or distractor revealed that saccade preparation is affected by task-relevance and refixation behavior. For task-relevant items only, we found longer fixation duration and higher SRP amplitudes for first fixations than for refixations over the occipital region and the opposite effect over the frontal region. Our findings provide first neurophysiological evidence that working memory loading of task-relevant information at fixation affects saccade planning.status: publishe

Neural correlates of task-related refixation behavior

Eye movement research has shown that attention shifts from the currently fixated location to the next before a saccade is executed. We investigated whether the cost of the attention shift depends on higher-order processing at the time of fixation, in particular on visual working memory load differences between fixations and refixations on task-relevant items. The attention shift is reflected in EEG activity in the saccade-related potential (SRP). In a free viewing task involving visual search and memorization of multiple targets amongst distractors, we compared the SRP in first fixations versus refixations on targets and distractors. The task-relevance of targets implies that more information will be loaded in memory (e.g. both identity and location) than for distractors (e.g. location only). First fixations will involve greater memory load than refixations, since first fixations involve loading of new items, while refixations involve rehearsal of previously visited items. The SRP in the interval preceding the saccade away from a target or distractor revealed that saccade preparation is affected by task-relevance and refixation behavior. For task-relevant items only, we found longer fixation duration and higher SRP amplitudes for first fixations than for refixations over the occipital region and the opposite effect over the frontal region. Our findings provide first neurophysiological evidence that working memory loading of task-relevant information at fixation affects saccade planning

Planning to revisit: Neural activity in refixation precursors

Eye tracking studies suggest that refixations-fixations to locations previously visited-serve to recover information lost or missed during earlier exploration of a visual scene. These studies have largely ignored the role of precursor fixations-previous fixations on locations the eyes return to later. We consider the possibility that preparations to return later are already made during precursor fixations. This process would mark precursor fixations as a special category of fixations, that is, distinct in neural activity from other fixation categories such as refixations and fixations to locations visited only once. To capture the neural signals associated with fixation categories, we analyzed electroencephalograms (EEGs) and eye movements recorded simultaneously in a free-viewing contour search task. We developed a methodological pipeline involving regression-based deconvolution modeling, allowing our analyses to account for overlapping EEG responses owing to the saccade sequence and other oculomotor covariates. We found that precursor fixations were preceded by the largest saccades among the fixation categories. Independent of the effect of saccade length, EEG amplitude was enhanced in precursor fixations compared with the other fixation categories 200 to 400 ms after fixation onsets, most noticeably over the occipital areas. We concluded that precursor fixations play a pivotal role in visual perception, marking the continuous occurrence of transitions between exploratory and exploitative modes of eye movement in natural viewing behavior

Presaccadic EEG activity predicts visual saliency in free-viewing contour integration

While viewing a scene, the eyes are attracted to salient stimuli. We set out to identify the brain signals controlling this process. In a contour integration task, in which participants searched for a collinear contour in a field of randomly oriented Gabor elements, a previously established model was applied to calculate a visual saliency value for each fixation location. We studied brain activity related to the modeled saliency values, using coregistered eye tracking and EEG. To disentangle EEG signals reflecting salience in free viewing from overlapping EEG responses to sequential eye movements, we adopted generalized additive mixed modeling (GAMM) to single epochs of saccade-related EEG. We found that, when saliency at the next fixation location was high, amplitude of the presaccadic EEG activity was low. Since presaccadic activity reflects covert attention to the saccade target, our results indicate that larger attentional effort is needed for selecting less salient saccade targets than more salient ones. This effect was prominent in contour-present conditions (half of the trials), but ambiguous in the contour-absent condition. Presaccadic EEG activity may thus be indicative of bottom-up factors in saccade guidance. The results underscore the utility of GAMM for EEG-eye movement coregistration research.status: publishe

Spatial Sound in a 3D Virtual Environment: All Bark and No Bite?

Although the focus of Virtual Reality (VR) lies predominantly on the visual world, acoustic components enhance the functionality of a 3D environment. To study the interaction between visual and auditory modalities in a 3D environment, we investigated the effect of auditory cues on visual searches in 3D virtual environments with both visual and auditory noise. In an experiment, we asked participants to detect visual targets in a 360° video in conditions with and without environmental noise. Auditory cues indicating the target location were either absent or one of simple stereo or binaural audio, both of which assisted sound localization. To investigate the efficacy of these cues in distracting environments, we measured participant performance using a VR headset with an eye tracker. We found that the binaural cue outperformed both stereo and no auditory cues in terms of target detection irrespective of the environmental noise. We used two eye movement measures and two physiological measures to evaluate task dynamics and mental effort. We found that the absence of a cue increased target search duration and target search path, measured as time to fixation and gaze trajectory lengths, respectively. Our physiological measures of blink rate and pupil size showed no difference between the different stadium and cue conditions. Overall, our study provides evidence for the utility of binaural audio in a realistic, noisy and virtual environment for performing a target detection task, which is a crucial part of everyday behaviour—finding someone in a crowd