Top-down neural attention by excitation backprop

We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation scheme, called Excitation Backprop, to pass along top-down signals downwards in the network hierarchy via a probabilistic Winner-Take-All process. Furthermore, we introduce the concept of contrastive attention to make the top-down attention maps more discriminative. In experiments, we demonstrate the accuracy and generalizability of our method in weakly supervised localization tasks on the MS COCO, PASCAL VOC07 and ImageNet datasets. The usefulness of our method is further validated in the text-to-region association task. On the Flickr30k Entities dataset, we achieve promising performance in phrase localization by leveraging the top-down attention of a CNN model that has been trained on weakly labeled web images.https://arxiv.org/abs/1608.00507Accepted manuscrip

Selective Attention in Multi-Chip Address-Event Systems

Selective attention is the strategy used by biological systems to cope with the inherent limits in their available computational resources, in order to efficiently process sensory information. The same strategy can be used in artificial systems that have to process vast amounts of sensory data with limited resources. In this paper we present a neuromorphic VLSI device, the “Selective Attention Chip” (SAC), which can be used to implement these models in multi-chip address-event systems. We also describe a real-time sensory-motor system, which integrates the SAC with a dynamic vision sensor and a robotic actuator. We present experimental results from each component in the system, and demonstrate how the complete system implements a real-time stimulus-driven selective attention model

Sparse Coding with a Somato-Dendritic Rule

© 2020 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence http://creativecommons.org/licenses/by-nc-nd/4.0/.Cortical neurons are silent most of the time. This sparse activity is energy efficient, and the resulting neural code has favourable properties for associative learning. Most neural models of sparse coding use some form of homeostasis to ensure that each neuron fires infrequently. But homeostatic plasticity acting on a fast timescale may not be biologically plausible, and could lead to catastrophic forgetting in embodied agents that learn continuously. We set out to explore whether inhibitory plasticity could play that role instead, regulating both the population sparseness and the average firing rates. We put the idea to the test in a hybrid network where rate-based dendritic compartments integrate the feedforward input, while spiking somas compete through recurrent inhibition. A somato-dendritic learning rule allows somatic inhibition to modulate nonlinear Hebbian learning in the dendrites. Trained on MNIST digits and natural images, the network discovers independent components that form a sparse encoding of the input and support linear decoding. These findings con-firm that intrinsic plasticity is not strictly required for regulating sparseness: inhibitory plasticity can have the same effect, although that mechanism comes with its own stability-plasticity dilemma. Going beyond point neuron models, the network illustrates how a learning rule can make use of dendrites and compartmentalised inputs; it also suggests a functional interpretation for clustered somatic inhibition in cortical neurons.Peer reviewe

Sequential memory-guided saccades and target selection: a neural model of the frontal eye fields

AbstractWe present a neural model of the frontal eye fields. It consists of several retinotopic arrays of neuron-like units that are recurrently connected. The network is trained to make memory-guided saccades to sequentially flashed targets that appear at arbitrary locations. This task is interesting because the large number of possible sequences does not permit a pre-learned response. Instead locations and their priority must be maintained in active working memory. The network learns to perform the task. Surprisingly, after training it can also select targets in visual search tasks. When targets are shown in parallel it chooses them according to their salience. Its search behavior is comparable to that of humans. It exhibits saccadic averaging, increased reaction times with more distractors, latency vs accuracy trade-offs, and inhibition of return. Analysis of the network shows that it operates like a queue, storing the potential targets in sequence for later execution. A small number of unit types are sufficient to encode this information, but the manner of coding is non-obvious. Units respond to multiple targets similar to quasi-visual cells recently studied [Exp. Brain Res. 130 (2000) 433]. Predictions are made that can be experimentally tested

Information processing in a midbrain visual pathway

Visual information is processed in brain via the intricate interactions between neurons. We investigated a midbrain visual pathway: optic tectum and its isthmic nucleus) that is motion sensitive and is thought as part of attentional system. We determined the physiological properties of individual neurons as well as their synaptic connections with intracellular recordings. We reproduced the center-surround receptive field structure of tectal neurons in a dynamical recurrent feedback loop. We reveal in a computational model that the anti-topographic inhibitory feedback could mediate competitive stimulus selection in a complex visual scene. We also investigated the dynamics of the competitive selection in a rate model. The isthmotectal feedback loop gates the information transfer from tectum to thalamic rotundus. We discussed the role of a localized feedback projection in contributing to the gating mechanisms with both experimental and numerical approaches. We further discussed the dynamics of the isthmotectal system by considering the propagation delays between different components. We conclude that the isthmotectal system is involved in attention-like competitive stimulus selection and control the information coding in the motion sensitive SGC-I neurons by modulating the retino-tectal synaptic transmission

New architectures for very deep learning

Artificial Neural Networks are increasingly being used in complex real- world applications because many-layered (i.e., deep) architectures can now be trained on large quantities of data. However, training even deeper, and therefore more powerful networks, has hit a barrier due to fundamental limitations in the design of existing networks. This thesis develops new architectures that, for the first time, allow very deep networks to be optimized efficiently and reliably. Specifically, it addresses two key issues that hamper credit assignment in neural networks: cross-pattern interference and vanishing gradients. Cross- pattern interference leads to oscillations of the network’s weights that make training inefficient. The proposed Local Winner-Take-All networks reduce interference among computation units in the same layer through local competition. An in-depth analysis of locally competitive networks provides generalizable insights and reveals unifying properties that improve credit assignment. As network depth increases, vanishing gradients make a network’s outputs increasingly insensitive to the weights close to the inputs, causing the failure of gradient-based training. To overcome this limitation, the proposed Highway networks regulate information flow across layers through additional skip connections which are modulated by learned computation units. Their beneficial properties are extended to the sequential domain with Recurrent Highway Networks that gain from increased depth and learn complex sequential transitions without requiring more parameters

Temporal Dynamics of Binocular Display Processing with Corticogeniculate Interactions

A neural model of binocular vision is developed to simulate psychophysical and neurobiological data concerning the dynamics of binocular disparity processing. The model shows how feedforward and feedback interactions among LGN ON and OFF cells and cortical simple, complex, and hypercomplex cells can simulate binocular summation, the Pulfrich effect, and the fusion of delayed anticorrelated stereograms. Model retinal ON and OFF cells are linked by an opponent process capable of generating antagonistic rebounds from OFF cells after offset of an ON cell input. Spatially displaced ON and OFF cells excite simple cells. Opposite polarity simple cells compete before their half-wave rectified outputs excite complex cells. Complex cells binocularly match like-polarity simple cell outputs before pooling half-wave rectified signals frorn opposite polarities. Competitive feedback among complex cells leads to sharpening of disparity selectivity and normalizes cell activity. Slow inhibitory interneurons help to reset complex cells after input offset. The Pulfrich effect occurs because the delayed input from the one eye fuses with the present input from the other eye to create a disparity. Binocular summation occurs for stimuli of brief duration or of low contrast because competitive normalization takes time, and cannot occur for very brief or weak stimuli. At brief SOAs, anticorrelatecd stereograms can be fused because the rebound mechanism ensures that the present image to one eye can fuse with the afterimage from a previous image to the other eye. Corticogeniculate feedback embodies a matching process that enhances the speed and temporal accuracy of complex cell disparity tuning. Model mechanisms interact to control the stable development of sharp disparity tuning.Air Force Office of Scientific Research (F19620-92-J-0499, F49620-92-J-0334, F49620-92-J-0225); Office of Naval Research (N00014-95-1-0409, N00014-95-l-0657, N00014-92-J-1015, N00014-91-J-4100