1,447 research outputs found
Efficient Computation in Adaptive Artificial Spiking Neural Networks
Artificial Neural Networks (ANNs) are bio-inspired models of neural
computation that have proven highly effective. Still, ANNs lack a natural
notion of time, and neural units in ANNs exchange analog values in a
frame-based manner, a computationally and energetically inefficient form of
communication. This contrasts sharply with biological neurons that communicate
sparingly and efficiently using binary spikes. While artificial Spiking Neural
Networks (SNNs) can be constructed by replacing the units of an ANN with
spiking neurons, the current performance is far from that of deep ANNs on hard
benchmarks and these SNNs use much higher firing rates compared to their
biological counterparts, limiting their efficiency. Here we show how spiking
neurons that employ an efficient form of neural coding can be used to construct
SNNs that match high-performance ANNs and exceed state-of-the-art in SNNs on
important benchmarks, while requiring much lower average firing rates. For
this, we use spike-time coding based on the firing rate limiting adaptation
phenomenon observed in biological spiking neurons. This phenomenon can be
captured in adapting spiking neuron models, for which we derive the effective
transfer function. Neural units in ANNs trained with this transfer function can
be substituted directly with adaptive spiking neurons, and the resulting
Adaptive SNNs (AdSNNs) can carry out inference in deep neural networks using up
to an order of magnitude fewer spikes compared to previous SNNs. Adaptive
spike-time coding additionally allows for the dynamic control of neural coding
precision: we show how a simple model of arousal in AdSNNs further halves the
average required firing rate and this notion naturally extends to other forms
of attention. AdSNNs thus hold promise as a novel and efficient model for
neural computation that naturally fits to temporally continuous and
asynchronous applications
The effect of neural adaptation of population coding accuracy
Most neurons in the primary visual cortex initially respond vigorously when a
preferred stimulus is presented, but adapt as stimulation continues. The
functional consequences of adaptation are unclear. Typically a reduction of
firing rate would reduce single neuron accuracy as less spikes are available
for decoding, but it has been suggested that on the population level,
adaptation increases coding accuracy. This question requires careful analysis
as adaptation not only changes the firing rates of neurons, but also the neural
variability and correlations between neurons, which affect coding accuracy as
well. We calculate the coding accuracy using a computational model that
implements two forms of adaptation: spike frequency adaptation and synaptic
adaptation in the form of short-term synaptic plasticity. We find that the net
effect of adaptation is subtle and heterogeneous. Depending on adaptation
mechanism and test stimulus, adaptation can either increase or decrease coding
accuracy. We discuss the neurophysiological and psychophysical implications of
the findings and relate it to published experimental data.Comment: 35 pages, 8 figure
Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks
Biological neurons communicate with a sparing exchange of pulses - spikes. It
is an open question how real spiking neurons produce the kind of powerful
neural computation that is possible with deep artificial neural networks, using
only so very few spikes to communicate. Building on recent insights in
neuroscience, we present an Adapting Spiking Neural Network (ASNN) based on
adaptive spiking neurons. These spiking neurons efficiently encode information
in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding while
homeostatically optimizing their firing rate. In the proposed paradigm of
spiking neuron computation, neural adaptation is tightly coupled to synaptic
plasticity, to ensure that downstream neurons can correctly decode upstream
spiking neurons. We show that this type of network is inherently able to carry
out asynchronous and event-driven neural computation, while performing
identical to corresponding artificial neural networks (ANNs). In particular, we
show that these adaptive spiking neurons can be drop in replacements for ReLU
neurons in standard feedforward ANNs comprised of such units. We demonstrate
that this can also be successfully applied to a ReLU based deep convolutional
neural network for classifying the MNIST dataset. The ASNN thus outperforms
current Spiking Neural Networks (SNNs) implementations, while responding (up
to) an order of magnitude faster and using an order of magnitude fewer spikes.
Additionally, in a streaming setting where frames are continuously classified,
we show that the ASNN requires substantially fewer network updates as compared
to the corresponding ANN
Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks
Biological neurons communicate with a sparing exchange of pulses - spikes. It
is an open question how real spiking neurons produce the kind of powerful
neural computation that is possible with deep artificial neural networks, using
only so very few spikes to communicate. Building on recent insights in
neuroscience, we present an Adapting Spiking Neural Network (ASNN) based on
adaptive spiking neurons. These spiking neurons efficiently encode information
in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding while
homeostatically optimizing their firing rate. In the proposed paradigm of
spiking neuron computation, neural adaptation is tightly coupled to synaptic
plasticity, to ensure that downstream neurons can correctly decode upstream
spiking neurons. We show that this type of network is inherently able to carry
out asynchronous and event-driven neural computation, while performing
identical to corresponding artificial neural networks (ANNs). In particular, we
show that these adaptive spiking neurons can be drop in replacements for ReLU
neurons in standard feedforward ANNs comprised of such units. We demonstrate
that this can also be successfully applied to a ReLU based deep convolutional
neural network for classifying the MNIST dataset. The ASNN thus outperforms
current Spiking Neural Networks (SNNs) implementations, while responding (up
to) an order of magnitude faster and using an order of magnitude fewer spikes.
Additionally, in a streaming setting where frames are continuously classified,
we show that the ASNN requires substantially fewer network updates as compared
to the corresponding ANN
Network Plasticity as Bayesian Inference
General results from statistical learning theory suggest to understand not
only brain computations, but also brain plasticity as probabilistic inference.
But a model for that has been missing. We propose that inherently stochastic
features of synaptic plasticity and spine motility enable cortical networks of
neurons to carry out probabilistic inference by sampling from a posterior
distribution of network configurations. This model provides a viable
alternative to existing models that propose convergence of parameters to
maximum likelihood values. It explains how priors on weight distributions and
connection probabilities can be merged optimally with learned experience, how
cortical networks can generalize learned information so well to novel
experiences, and how they can compensate continuously for unforeseen
disturbances of the network. The resulting new theory of network plasticity
explains from a functional perspective a number of experimental data on
stochastic aspects of synaptic plasticity that previously appeared to be quite
puzzling.Comment: 33 pages, 5 figures, the supplement is available on the author's web
page http://www.igi.tugraz.at/kappe
Nonlinear Hebbian learning as a unifying principle in receptive field formation
The development of sensory receptive fields has been modeled in the past by a
variety of models including normative models such as sparse coding or
independent component analysis and bottom-up models such as spike-timing
dependent plasticity or the Bienenstock-Cooper-Munro model of synaptic
plasticity. Here we show that the above variety of approaches can all be
unified into a single common principle, namely Nonlinear Hebbian Learning. When
Nonlinear Hebbian Learning is applied to natural images, receptive field shapes
were strongly constrained by the input statistics and preprocessing, but
exhibited only modest variation across different choices of nonlinearities in
neuron models or synaptic plasticity rules. Neither overcompleteness nor sparse
network activity are necessary for the development of localized receptive
fields. The analysis of alternative sensory modalities such as auditory models
or V2 development lead to the same conclusions. In all examples, receptive
fields can be predicted a priori by reformulating an abstract model as
nonlinear Hebbian learning. Thus nonlinear Hebbian learning and natural
statistics can account for many aspects of receptive field formation across
models and sensory modalities
Continuous-time spike-based reinforcement learning for working memory tasks
As the brain purportedly employs on-policy reinforcement learning compatible with SARSA learning, and most interesting cognitive tasks require some form of memory while taking place in continuous-time, recent work has developed plausible reinforcement learning schemes that are compatible with these requirements. Lacking is a formulation of both computation and learning in terms of spiking neurons. Such a formulation creates both a closer mapping to biology, and also expresses such learning in terms of asynchronous and sparse neural computation. We present a spiking neural network with memory that learns cognitive tasks in continuous time. Learning is biologically plausibly implemented using the AuGMeNT framework, and we show how separate spiking forward and feedback networks suffice for learning the tasks just as fast the analog CT-AuGMeNT counterpart, while computing efficiently using very few spikes: 1–20 Hz on average
- …