Prospects and pitfalls in combining eye tracking data and verbal reports

It is intuitively appealing to try to combine eye-tracking data and verbal reports when investigating medical image interpretation. However, before collecting such data, important decisions have to be made including exactly when and how to collect the verbal reports. The purpose of this methodological article is to reflect upon the pros and cons of different solutions and to offer some guidelines to investigators. We start by exploring the ontology of vision and speech production and the epistemology of eye movements to grasp what fixations and verbal reports actually reflect. We are also interested in the major constraints of the two systems. Second, we elaborate on two dominant investigational approaches to verbal accounts, namely concurrent think-aloud and Chi’s explanations, and move on to other approaches. Third, we present and critically evaluate studies from the literature on medical image interpretation that have sought to contrast or integrate eye movement data and verbal reports. Fourth, we conclude with some practical guidelines and suggestions for further research.              

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Exploring And Training Spatial Reasoning Via Eye Movements: Implications On Performance

This dissertation sought to determine if eye movements could serve as an indicator of success in spatial reasoning, and if eye movements associated with successful completion could be applied to strategically improve spatial reasoning. Using the line images of Shepard and Metzler, an electronic test of mental rotations ability (EMRT) was designed. Two versions of the test were created, allowing for both a timed (6 seconds per question) and untimed testing environment. Four experiments were designed and completed to relate mental rotation ability (MRA) scores from the EMRT, to patterns in chrononumeric and visual salience data. In each experiment, participants completed the EMRT under a different protocol. These protocols included an untimed EMRT, a timed EMRT, a within-participant crossover study where participants completed both the timed, and untimed EMRT in series, and a training crossover study where low MRA participants completed the timed EMRT in both a guided and unguided environment. In the untimed experiment, individuals of high and low MRA were asked to complete the EMRT while their eye movements were observed. As no time limit was imposed, the results allowed for observations based on MRA alone, and served to demonstrate and how individuals of different skill level differ in terms of eye movement. In the following experiment, the addition of a time limit to the EMRT revealed how individuals of high and low MRA perform when under a time restriction. The results of the Timed experiment confirmed differences between the high and low MRA group in terms of eye movements, and attention to salient regions of test images. In the third experiment, the addition of a time limit was further explored through a crossover design. By adding a time limit to an MRT, the ability of individuals to solve spatial problems is impaired, and is manifest in eye movements. Data derived from the Crossover Experiment suggested that salience-based metrics might serve to distinguish between groups of MRA, and that time restrictions may influence both participant accuracy, and identification of visually salient elements. The results from the first three experiments were then applied in the Guidance Experiment to confirm the role that visual salience plays in the context of spatial problem solving. By mapping the apprehension patterns of successful high MRA individuals onto the EMRT, low MRA individuals could be guided to salient areas on the timed EMRT. The results revealed that the application of visual guidance is an effective mechanism for MRA training. This research attends to a previously unaddressed niche in eye-movement and spatial ability training literature. As a result, it may serve as a foundation to cultivate methods of honing and improving spatial skills in the general population

Toward a second-person neuroscience

LS & BT : equal contributions (shared first-authorship)Peer reviewedPreprin

Evoking a Simulated Past: technology and manipulation of human memory.

A memória é uma das características fundamentais do ser humano, ela molda a nossa personalidade e permite-nos reagir às experiências da vida. Ao longo da História, os seres humanos têm vindo a procurar formas de melhorar as suas capacidades cognitivas, contando com mecanismos e dispositivos externos, como a escrita e a produção de imagens. No século vinte e um, a memória humana não pode ser separada da tecnologia digital, já que armazenamos constantemente registos das nossas experiências em smartphones e computadores. Hoje, temos acesso a quase toda a informação do mundo nos nossos bolsos, o que nos leva a confiar demasiado na tecnologia para armazenar as nossas memórias. Mas esse acesso a toda essa informação está a transformar a maneira como formamos, armazenamos e reconstruímos memórias. Apesar de termos registos de várias memórias do dia-a-dia, a capacidade do nosso cérebro de formar novas memórias está a mudar, tornando-nos mais esquecidos. Partindo destas ideias, desenvolvemos uma instalação multimédia que explora processos de reconstrução e associação de memórias, e um ensaio crítico que potencie o questionamento sobre a tecnologia e a manipulação da memória humana. Os nossos objectivos principais são pesquisar sobre o cérebro humano, a memória humana e a sua relação com as emoções; pesquisar sobre práticas arquivísticas; e pesquisar sobre o impacto das tecnologias da memória; a fim de potenciar o questionamento e gerar discussão sobre o impacto da tecnologia na memória humana individual e os seus possíveis efeitos. A instalação, denominada Evoking a Simulated Past, é descrita nesta dissertação, bem como o processo criativo e as contribuições da pesquisa sobre a memória humana e as práticas arquivísticas para a implementação do projecto.Memory is one of the fundamental characteristics of human beings, it shapes our personality and allows us to react to life experiences. Throughout History human beings have been looking for ways to enhance their cognitive abilities by relying on mechanisms and external devices, such as writing and picture-making. In the 21st century human memory cannot be separated from digital technology, as we constantly store records of our daily experiences in smartphones and computers. Today, we have access to almost all the information of the world in our pockets which leads us to trust too much on technology to store our memories. But the access to all that information is changing how we form, store and reconstruct memories. While we have recordings of our daily memories in our smartphones and computers, the ability of our brain to form new memories is changing, making us more forgetful. Regarding these ideas, we developed a multimedia installation that explores processes of memory reconstruction and association and a critical essay that fosters the questioning about technology and manipulation of human memory. Our main goals are to research on the human brain; the human memory and its relationship with emotions; research on archival practices; research on the impact of technologies on human memory; in order to foster the questioning and generate discussion about the impact of technology on the individual memory and its possible outcomes. The installation, named Evoking a Simulated Past, is described in this dissertation as well as the creative process and how the research on human memory and archival practices contributed to its implementation

Prospects and pitfalls in combining eye tracking data and verbal reports

It is intuitively appealing to try to combine eye-tracking data and verbal reports when investigating medical image interpretation. However, before collecting such data, important decisions have to be made including exactly when and how to collect the verbal reports. The purpose of this methodological article is to reflect upon the pros and cons of different solutions and to offer some guidelines to investigators. We start by exploring the ontology of vision and speech production and the epistemology of eye movements to grasp what fixations and verbal reports actually reflect. We are also interested in the major constraints of the two systems. Second, we elaborate on two dominant investigational approaches to verbal accounts, namely concurrent think-aloud and Chi’s explanations, and move on to other approaches. Third, we present and critically evaluate studies from the literature on medical image interpretation that have sought to contrast or integrate eye movement data and verbal reports. Fourth, we conclude with some practical guidelines and suggestions for further research</p

Eye movements as a window to cognitive processes

Eye movement research is a highly active and productive research field. Here we focus on how the embodied nature of eye movements can act as a window to the brain and the mind. In particular, we discuss how conscious perception depends on the trajectory of fixated locations and consequently address how fixation locations are selected. Specifically, we argue that the selection of fixation points during visual exploration can be understood to a large degree based on retinotopically structured models. Yet, these models largely ignore spatiotemporal structure in eye-movement sequences. Explaining spatiotemporal structure in eye-movement trajectories requires an understanding of spatiotemporal properties of the visual sampling process. With this in mind, we discuss the availability of external information to internal inference about causes in the world. We demonstrate that visual foraging is a dynamic process that can be systematically modulated either towards exploration or exploitation. For an analysis at high temporal resolution, we suggest a new method: The renewal density allows the investigation of precise temporal relation of eye movements and other actions like a button press. We conclude with an outlook and propose that eye movement research has reached an appropriate stage and can easily be combined with other research methods to utilize this window to the brain and mind to its fullest

How the brain grasps tools: fMRI & motion-capture investigations

Humans’ ability to learn about and use tools is considered a defining feature of our species, with most related neuroimaging investigations involving proxy 2D picture viewing tasks. Using a novel tool grasping paradigm across three experiments, participants grasped 3D-printed tools (e.g., a knife) in ways that were considered to be typical (i.e., by the handle) or atypical (i.e., by the blade) for subsequent use. As a control, participants also performed grasps in corresponding directions on a series of 3D-printed non-tool objects, matched for properties including elongation and object size. Project 1 paired a powerful fMRI block-design with visual localiser Region of Interest (ROI) and searchlight Multivoxel Pattern Analysis (MVPA) approaches. Most remarkably, ROI MVPA revealed that hand-selective, but not anatomically overlapping tool-selective, areas of the left Lateral Occipital Temporal Cortex and Intraparietal Sulcus represented the typicality of tool grasping. Searchlight MVPA found similar evidence within left anterior temporal cortex as well as right parietal and temporal areas. Project 2 measured hand kinematics using motion-capture during a highly similar procedure, finding hallmark grip scaling effects despite the unnatural task demands. Further, slower movements were observed when grasping tools, relative to non-tools, with grip scaling also being poorer for atypical tool, compared to non-tool, grasping. Project 3 used a slow-event related fMRI design to investigate whether representations of typicality were detectable during motor planning, but MVPA was largely unsuccessful, presumably due to a lack of statistical power. Taken together, the representations of typicality identified within areas of the ventral and dorsal, but not ventro-dorsal, pathways have implications for specific predictions made by leading theories about the neural regions supporting human tool-use, including dual visual stream theory and the two-action systems model