A computer vision model for visual-object-based attention and eye movements

This is the post-print version of the final paper published in Computer Vision and Image Understanding. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2008 Elsevier B.V.This paper presents a new computational framework for modelling visual-object-based attention and attention-driven eye movements within an integrated system in a biologically inspired approach. Attention operates at multiple levels of visual selection by space, feature, object and group depending on the nature of targets and visual tasks. Attentional shifts and gaze shifts are constructed upon their common process circuits and control mechanisms but also separated from their different function roles, working together to fulfil flexible visual selection tasks in complicated visual environments. The framework integrates the important aspects of human visual attention and eye movements resulting in sophisticated performance in complicated natural scenes. The proposed approach aims at exploring a useful visual selection system for computer vision, especially for usage in cluttered natural visual environments.National Natural Science of Founda- tion of Chin

Saccade learning with concurrent cortical and subcortical basal ganglia loops

The Basal Ganglia is a central structure involved in multiple cortical and subcortical loops. Some of these loops are believed to be responsible for saccade target selection. We study here how the very specific structural relationships of these saccadic loops can affect the ability of learning spatial and feature-based tasks. We propose a model of saccade generation with reinforcement learning capabilities based on our previous basal ganglia and superior colliculus models. It is structured around the interactions of two parallel cortico-basal loops and one tecto-basal loop. The two cortical loops separately deal with spatial and non-spatial information to select targets in a concurrent way. The subcortical loop is used to make the final target selection leading to the production of the saccade. These different loops may work in concert or disturb each other regarding reward maximization. Interactions between these loops and their learning capabilities are tested on different saccade tasks. The results show the ability of this model to correctly learn basic target selection based on different criteria (spatial or not). Moreover the model reproduces and explains training dependent express saccades toward targets based on a spatial criterion. Finally, the model predicts that in absence of prefrontal control, the spatial loop should dominate

Aftereffects of Saccades Explored in a Dynamic Neural Field Model of the Superior Colliculus

When viewing a scene or searching for a target, an observer usually makes a series of saccades that quickly shift the orientation of the eyes. The present study explored how one saccade affects subsequent saccades within a dynamic neural field model of the superior colliculus (SC). The SC contains an oculocentric motor map that encodes the vector of saccades and remaps to the new fixation location after each saccade. Our simulations demonstrated that the observation that saccades which reverse their vectors are slower to initiate than those which repeat vectors can be explained by the afore-mentioned remapping process and the internal dynamics of the SC. How this finding connects to the study of inhibition of return is discussed and suggestions for future studies are presented

A Distal Model of Congenital Nystagmus as Nonlinear Adaptive Oscillations

Congenital nystagmus (CN) is an incurable pathological spontaneous oscillation of the eyes with an onset in the first few months of life. The pathophysiology of CN is mysterious. There is no consistent neurological abnormality, but the majority of patients have a wide range of unrelated congenital visual abnormalities affecting either the cornea, lens, retina or optic nerve. In this theoretical study, we show that these eye oscillations could develop as an adaptive response to maximize visual contrast with poor foveal function in the infant visuomotor system, at a time of peak neural plasticity. We argue that in a visual system with abnormally poor high spatial frequency sensitivity, image contrast is not only maintained by keeping the image on the fovea (or its remnant) but also by some degree of image motion. Using the calculus of variations, we show that the optimal trade-off between these conflicting goals is to generate oscillatory eye movements with increasing velocity waveforms, as seen in real CN. When we include a stochastic component to the start of each epoch (quick-phase inaccuracy) various observed waveforms (including pseudo-cycloid) emerge as optimal strategies. Using the delay embedding technique, we find a low fractional dimension as reported in real data. We further show that, if a velocity command-based pre-motor circuitry (neural integrator) is harnessed to generate these waveforms, the emergence of a null region is inevitable. We conclude that CN could emerge paradoxically as an ‘optimal’ adaptive response in the infant visual system during an early critical period. This can explain why CN does not emerge later in life and why CN is so refractory to treatment. It also implies that any therapeutic intervention would need to be very early in life

A Computational Predictor of Human Episodic Memory Based on a Theta Phase Precession Network

In the rodent hippocampus, a phase precession phenomena of place cell firing with the local field potential (LFP) theta is called “theta phase precession” and is considered to contribute to memory formation with spike time dependent plasticity (STDP). On the other hand, in the primate hippocampus, the existence of theta phase precession is unclear. Our computational studies have demonstrated that theta phase precession dynamics could contribute to primate–hippocampal dependent memory formation, such as object–place association memory. In this paper, we evaluate human theta phase precession by using a theory–experiment combined analysis. Human memory recall of object–place associations was analyzed by an individual hippocampal network simulated by theta phase precession dynamics of human eye movement and EEG data during memory encoding. It was found that the computational recall of the resultant network is significantly correlated with human memory recall performance, while other computational predictors without theta phase precession are not significantly correlated with subsequent memory recall. Moreover the correlation is larger than the correlation between human recall and traditional experimental predictors. These results indicate that theta phase precession dynamics are necessary for the better prediction of human recall performance with eye movement and EEG data. In this analysis, theta phase precession dynamics appear useful for the extraction of memory-dependent components from the spatio–temporal pattern of eye movement and EEG data as an associative network. Theta phase precession may be a common neural dynamic between rodents and humans for the formation of environmental memories

Visual and eye movement functions of the posterior parietal cortex

Lesions of the posterior parietal area in humans produce interesting spatial-perceptual and spatial-behavioral deficits. Among the more important deficits observed are loss of spatial memories, problems representing spatial relations in models or drawings, disturbances in the spatial distribution of attention, and the inability to localize visual targets. Posterior parietal lesions in nonhuman primates also produce visual spatial deficits not unlike those found in humans. Mountcastle and his colleagues were the first to explore this area, using single cell recording techniques in behaving monkeys over 13 years ago. Subsequent work by Mountcastle, Lynch and colleagues, Hyvarinen and colleagues, Robinson, Goldberg & Stanton, and Sakata and colleagues during the period of the late 1970s and early 1980s provided an informational and conceptual foundation for exploration of this fascinating area of the brain. Four new directions of research that are presently being explored from this foundation are reviewed in this article. 1. The anatomical and functional organization of the inferior parietal lobule is presently being investigated with neuroanatomical tracing and single cell recording techniques. This area is now known to be comprised of at least four separate cortical fields. 2. Neural mechanisms for spatial constancy are being explored. In area 7a information about eye position is found to be integrated with visual inputs to produce representations of visual space that are head-centered (the meaning of a head-centered coordinate system is explained on p. 13). 3. The role of the posterior parietal cortex, and the pathways projecting into this region, in processing information about motion in the visual world is under investigation. Visual areas within the posterior parietal cortex may play a role in extracting higher level motion information including the perception of structure-from-motion. 4. A previously unexplored area within the intraparietal sulcus has been found whose cells hold a representation in memory of planned eye movements. Special experimental protocols have shown that these cells code the direction and amplitude of intended movements in motor coordinates and suggest that this area plays a role in motor planning

Training Deep Spiking Neural Networks Using Backpropagation

Deep spiking neural networks (SNNs) hold the potential for improving the latency and energy efficiency of deep neural networks through data-driven event-based computation. However, training such networks is difficult due to the non-differentiable nature of spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are considered as noise. This enables an error backpropagation mechanism for deep SNNs that follows the same principles as in conventional deep networks, but works directly on spike signals and membrane potentials. Compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statistics of spikes more precisely. We evaluate the proposed framework on artificially generated events from the original MNIST handwritten digit benchmark, and also on the N-MNIST benchmark recorded with an event-based dynamic vision sensor, in which the proposed method reduces the error rate by a factor of more than three compared to the best previous SNN, and also achieves a higher accuracy than a conventional convolutional neural network (CNN) trained and tested on the same data. We demonstrate in the context of the MNIST task that thanks to their event-driven operation, deep SNNs (both fully connected and convolutional) trained with our method achieve accuracy equivalent with conventional neural networks. In the N-MNIST example, equivalent accuracy is achieved with about five times fewer computational operations