Binocular fusion and invariant category learning due to predictive remapping during scanning of a depthful scene with eye movements

How does the brain maintain stable fusion of 3D scenes when the eyes move? Every eye movement causes each retinal position to process a different set of scenic features, and thus the brain needs to binocularly fuse new combinations of features at each position after an eye movement. Despite these breaks in retinotopic fusion due to each movement, previously fused representations of a scene in depth often appear stable. The 3D ARTSCAN neural model proposes how the brain does this by unifying concepts about how multiple cortical areas in the What and Where cortical streams interact to coordinate processes of 3D boundary and surface perception, spatial attention, invariant object category learning, predictive remapping, eye movement control, and learned coordinate transformations. The model explains data from single neuron and psychophysical studies of covert visual attention shifts prior to eye movements. The model further clarifies how perceptual, attentional, and cognitive interactions among multiple brain regions (LGN, V1, V2, V3A, V4, MT, MST, PPC, LIP, ITp, ITa, SC) may accomplish predictive remapping as part of the process whereby view-invariant object categories are learned. These results build upon earlier neural models of 3D vision and figure-ground separation and the learning of invariant object categories as the eyes freely scan a scene. A key process concerns how an object's surface representation generates a form-fitting distribution of spatial attention, or attentional shroud, in parietal cortex that helps maintain the stability of multiple perceptual and cognitive processes. Predictive eye movement signals maintain the stability of the shroud, as well as of binocularly fused perceptual boundaries and surface representations.Published versio

Timing of visual stimuli in V1 and V5/MT: fMRI and TMS

Time perception is used in our day-to-day activities. While we understand quite well how our brain processes vision, touch or taste, brain mechanisms subserving time perception remain largely understudied. In this study, we extended an experiment of previous master thesis run by Tatiana Kenel-Pierre. We focused on time perception in the range of milliseconds. Previous studies have demonstrated the involvement of visual areas V1 and V5/MT in the encoding of temporal information of visual stimuli. Based on these previous findings the aim of the present study was to understand if temporal information was encoded in V1 and extrastriate area V5/MT in different spatial frames i.e., head- centered versus eye-centered. To this purpose we asked eleven healthy volunteers to perform a temporal discrimination task of visual stimuli. Stimuli were presented at 4 different spatial positions (i.e., different combinations of retinotopic and spatiotopic position). While participants were engaged in this task we interfered with the activity of the right dorsal V1 and the right V5/MT with transcranial magnetic stimulation (TMS). Our preliminary results showed that TMS over both V1 and V5/MT impaired temporal discrimination of visual stimuli presented at specific spatial coordinates. But whereas TMS over V1 impaired temporal discrimination of stimuli presented in the lower left quadrant, TMS over V5/MT affected temporal discrimination of stimuli presented at the top left quadrant. Although it is always difficult to draw conclusions from preliminary results, we could tentatively say that our data seem to suggest that both V1 and V5/MT encode visual temporal information in specific spatial frames

Mechanisms of top-down visual spatial attention: computational and behavioral investigations

This thesis examines the mechanisms underlying visual spatial attention. In particular I focused on top-­‐down or voluntary attention, namely the ability to select relevant information and discard the irrelevant according to our goals. Given the limited processing resources of the human brain, which does not allow to process all the available information to the same degree, the ability to correctly allocate processing resources is fundamental for the accomplishment of most everyday tasks. The cost of misoriented attention is that we could miss some relevant information, with potentially serious consequences. In the first study (chapter 2) I will address the issue of the neural substrates of visual spatial attention: what are the neural mechanisms that allow the deployment of visual spatial attention? According to the premotor theory orienting attention to a location in space is equivalent to planning an eye movement to the same location, an idea strongly supported by neuroimaging and neurophysiological evidence. Accordingly, in this study I will present a model that can account for several attentional effects without requiring additional mechanisms separate from the circuits that perform sensorimotor transformations for eye movements. Moreover, it includes a mechanism that allows, within the framework of the premotor theory, to explain dissociations between attention and eye movements that may be invoked to disprove it. In the second model presented (chapter 3) I will further investigate the computational mechanisms underlying sensorimotor transformations. Specifically I will show that a representation in which the amplitude of visual responses is modulated by postural signal is both efficient and plausible, emerging also in a neural network model trained through unsupervised learning (i.e., using only signals locally available at the neuron level). Ultimately this result gives additional support to the approach adopted in the first model. Next, I will present a series of behavioral studies: in the first (chapter 4) I will show that spatial constancy of attention (i.e., the ability to sustain attention at a spatial location across eye movements) is dependent on some properties of the image, namely the presence of continuous visual landmarks at the attended locations. Importantly, this finding helps resolve contrasts between several recent results. In the second behavioral study (chapter 5), I will investigate an often neglected aspect of spatial cueing paradigms, probably the most widely used technique in studies of covert attention: the role of cue predictivity (i.e. the extent to which the spatial cue correctly indicates the location where the target stimulus will appear). Results show that, independently of participant’s awareness, changes  in predictivity result in changes in spatial validity effects, and that reliable shifts of attention can take place also in the absence of a predictive cue. In sum the results question the appropriateness of using predictive cues for delineating pure voluntary shifts of spatial attention. Finally, in the last study I will use a psychophysiological measure, the diameter of the eye’s pupil, to investigate intensive aspects of attention. Event-­‐related pupil dilations accurately mirrored changes in visuospatial awareness induced by a dual-­‐task manipulation that consumed attentional resources. Moreover, results of the primary spatial monitoring task revealed a significant rightward bias, indicated by a greater proportion of missed targets in the left hemifield. Interestingly this result mimics the extinction to double simultaneous stimulation (i.e., the failure to respond to a stimulus when it is presented simultaneously with another stimulus) which is often found in patients with unilateral brain damage. Overall, these studies present an emerging picture of attention as a complex mechanism that even in its volitional aspects is modulated by other non-­‐volitional factors, both external and internal to the individua

Neural dynamics of invariant object recognition: relative disparity, binocular fusion, and predictive eye movements

How does the visual cortex learn invariant object categories as an observer scans a depthful scene? Two neural processes that contribute to this ability are modeled in this thesis. The first model clarifies how an object is represented in depth. Cortical area V1 computes absolute disparity, which is the horizontal difference in retinal location of an image in the left and right foveas. Many cells in cortical area V2 compute relative disparity, which is the difference in absolute disparity of two visible features. Relative, but not absolute, disparity is unaffected by the distance of visual stimuli from an observer, and by vergence eye movements. A laminar cortical model of V2 that includes shunting lateral inhibition of disparity-sensitive layer 4 cells causes a peak shift in cell responses that transforms absolute disparity from V1 into relative disparity in V2. The second model simulates how the brain maintains stable percepts of a 3D scene during binocular movements. The visual cortex initiates the formation of a 3D boundary and surface representation by binocularly fusing corresponding features from the left and right retinotopic images. However, after each saccadic eye movement, every scenic feature projects to a different combination of retinal positions than before the saccade. Yet the 3D representation, resulting from the prior fusion, is stable through the post-saccadic re-fusion. One key to stability is predictive remapping: the system anticipates the new retinal positions of features entailed by eye movements by using gain fields that are updated by eye movement commands. The 3D ARTSCAN model developed here simulates how perceptual, attentional, and cognitive interactions across different brain regions within the What and Where visual processing streams interact to coordinate predictive remapping, stable 3D boundary and surface perception, spatial attention, and the learning of object categories that are invariant to changes in an object's retinal projections. Such invariant learning helps the system to avoid treating each new view of the same object as a distinct object to be learned. The thesis hereby shows how a process that enables invariant object category learning can be extended to also enable stable 3D scene perception

Cortical Dynamics of Contextually-Cued Attentive Visual Learning and Search: Spatial and Object Evidence Accumulation

How do humans use predictive contextual information to facilitate visual search? How are consistently paired scenic objects and positions learned and used to more efficiently guide search in familiar scenes? For example, a certain combination of objects can define a context for a kitchen and trigger a more efficient search for a typical object, such as a sink, in that context. A neural model, ARTSCENE Search, is developed to illustrate the neural mechanisms of such memory-based contextual learning and guidance, and to explain challenging behavioral data on positive/negative, spatial/object, and local/distant global cueing effects during visual search. The model proposes how global scene layout at a first glance rapidly forms a hypothesis about the target location. This hypothesis is then incrementally refined by enhancing target-like objects in space as a scene is scanned with saccadic eye movements. The model clarifies the functional roles of neuroanatomical, neurophysiological, and neuroimaging data in visual search for a desired goal object. In particular, the model simulates the interactive dynamics of spatial and object contextual cueing in the cortical What and Where streams starting from early visual areas through medial temporal lobe to prefrontal cortex. After learning, model dorsolateral prefrontal cortical cells (area 46) prime possible target locations in posterior parietal cortex based on goalmodulated percepts of spatial scene gist represented in parahippocampal cortex, whereas model ventral prefrontal cortical cells (area 47/12) prime possible target object representations in inferior temporal cortex based on the history of viewed objects represented in perirhinal cortex. The model hereby predicts how the cortical What and Where streams cooperate during scene perception, learning, and memory to accumulate evidence over time to drive efficient visual search of familiar scenes.CELEST, an NSF Science of Learning Center (SBE-0354378); SyNAPSE program of Defense Advanced Research Projects Agency (HR0011-09-3-0001, HR0011-09-C-0011

Updating spatial working memory in a dynamic visual environment

The present review describes recent developments regarding the role of the eye movement system in representing spatial information and keeping track of locations of relevant objects. First, we discuss the active vision perspective and why eye movements are considered crucial for perception and attention. The second part focuses on the question of how the oculomotor system is used to represent spatial attentional priority, and the role of the oculomotor system in maintenance of this spatial information. Lastly, we discuss recent findings demonstrating rapid updating of information across saccadic eye movements. We argue that the eye movement system plays a key role in maintaining and rapidly updating spatial information. Furthermore, we suggest that rapid updating emerges primarily to make sure actions are minimally affected by intervening eye movements, allowing us to efficiently interact with the world around us

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

A Unified Cognitive Model of Visual Filling-In Based on an Emergic Network Architecture

The Emergic Cognitive Model (ECM) is a unified computational model of visual filling-in based on the Emergic Network architecture. The Emergic Network was designed to help realize systems undergoing continuous change. In this thesis, eight different filling-in phenomena are demonstrated under a regime of continuous eye movement (and under static eye conditions as well). ECM indirectly demonstrates the power of unification inherent with Emergic Networks when cognition is decomposed according to finer-grained functions supporting change. These can interact to raise additional emergent behaviours via cognitive re-use, hence the Emergic prefix throughout. Nevertheless, the model is robust and parameter free. Differential re-use occurs in the nature of model interaction with a particular testing paradigm. ECM has a novel decomposition due to the requirements of handling motion and of supporting unified modelling via finer functional grains. The breadth of phenomenal behaviour covered is largely to lend credence to our novel decomposition. The Emergic Network architecture is a hybrid between classical connectionism and classical computationalism that facilitates the construction of unified cognitive models. It helps cutting up of functionalism into finer-grains distributed over space (by harnessing massive recurrence) and over time (by harnessing continuous change), yet simplifies by using standard computer code to focus on the interaction of information flows. Thus while the structure of the network looks neurocentric, the dynamics are best understood in flowcentric terms. Surprisingly, dynamic system analysis (as usually understood) is not involved. An Emergic Network is engineered much like straightforward software or hardware systems that deal with continuously varying inputs. Ultimately, this thesis addresses the problem of reduction and induction over complex systems, and the Emergic Network architecture is merely a tool to assist in this epistemic endeavour. ECM is strictly a sensory model and apart from perception, yet it is informed by phenomenology. It addresses the attribution problem of how much of a phenomenon is best explained at a sensory level of analysis, rather than at a perceptual one. As the causal information flows are stable under eye movement, we hypothesize that they are the locus of consciousness, howsoever it is ultimately realized