16 research outputs found
Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks
Biological neurons communicate with a sparing exchange of pulses - spikes. It
is an open question how real spiking neurons produce the kind of powerful
neural computation that is possible with deep artificial neural networks, using
only so very few spikes to communicate. Building on recent insights in
neuroscience, we present an Adapting Spiking Neural Network (ASNN) based on
adaptive spiking neurons. These spiking neurons efficiently encode information
in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding while
homeostatically optimizing their firing rate. In the proposed paradigm of
spiking neuron computation, neural adaptation is tightly coupled to synaptic
plasticity, to ensure that downstream neurons can correctly decode upstream
spiking neurons. We show that this type of network is inherently able to carry
out asynchronous and event-driven neural computation, while performing
identical to corresponding artificial neural networks (ANNs). In particular, we
show that these adaptive spiking neurons can be drop in replacements for ReLU
neurons in standard feedforward ANNs comprised of such units. We demonstrate
that this can also be successfully applied to a ReLU based deep convolutional
neural network for classifying the MNIST dataset. The ASNN thus outperforms
current Spiking Neural Networks (SNNs) implementations, while responding (up
to) an order of magnitude faster and using an order of magnitude fewer spikes.
Additionally, in a streaming setting where frames are continuously classified,
we show that the ASNN requires substantially fewer network updates as compared
to the corresponding ANN
Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks
Biological neurons communicate with a sparing exchange of pulses - spikes. It
is an open question how real spiking neurons produce the kind of powerful
neural computation that is possible with deep artificial neural networks, using
only so very few spikes to communicate. Building on recent insights in
neuroscience, we present an Adapting Spiking Neural Network (ASNN) based on
adaptive spiking neurons. These spiking neurons efficiently encode information
in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding while
homeostatically optimizing their firing rate. In the proposed paradigm of
spiking neuron computation, neural adaptation is tightly coupled to synaptic
plasticity, to ensure that downstream neurons can correctly decode upstream
spiking neurons. We show that this type of network is inherently able to carry
out asynchronous and event-driven neural computation, while performing
identical to corresponding artificial neural networks (ANNs). In particular, we
show that these adaptive spiking neurons can be drop in replacements for ReLU
neurons in standard feedforward ANNs comprised of such units. We demonstrate
that this can also be successfully applied to a ReLU based deep convolutional
neural network for classifying the MNIST dataset. The ASNN thus outperforms
current Spiking Neural Networks (SNNs) implementations, while responding (up
to) an order of magnitude faster and using an order of magnitude fewer spikes.
Additionally, in a streaming setting where frames are continuously classified,
we show that the ASNN requires substantially fewer network updates as compared
to the corresponding ANN
An ultra-low-power sigma-delta neuron circuit
Neural processing systems typically represent data using leaky integrate and
fire (LIF) neuron models that generate spikes or pulse trains at a rate
proportional to their input amplitudes. This mechanism requires high firing
rates when encoding time-varying signals, leading to increased power
consumption. Neuromorphic systems that use adaptive LIF neuron models overcome
this problem by encoding signals in the relative timing of their output spikes
rather than their rate. In this paper, we analyze recent adaptive LIF neuron
circuit implementations and highlight the analogies and differences between
them and a first-order sigma-delta feedback loop. We propose a new sigma-delta
neuron circuit that addresses some of the limitations in existing
implementations and present simulation results that quantify the improvements.
We show that the new circuit, implemented in a 1.8 V, 180 nm CMOS process,
offers up to 42 dB signal-to-distortion ratio and consumes orders of magnitude
lower energy. Finally, we also demonstrate how the sigma-delta interpretation
enables mapping of real-valued recurrent neural network to the spiking
framework to emphasize the envisioned application of the proposed circuit.Comment: Submitted to TCAS-II Briefs. Reference code
online-https://github.com/manuvn/sigma-delta-neural-networks.gi
Asynchronous spiking neurons, the natural key to exploit temporal sparsity
Inference of Deep Neural Networks for stream signal (Video/Audio) processing in edge devices is still challenging. Unlike the most state of the art inference engines which are efficient for static signals, our brain is optimized for real-time dynamic signal processing. We believe one important feature of the brain (asynchronous state-full processing) is the key to its excellence in this domain. In this work, we show how asynchronous processing with state-full neurons allows exploitation of the existing sparsity in natural signals. This paper explains three different types of sparsity and proposes an inference algorithm which exploits all types of sparsities in the execution of already trained networks. Our experiments in three different applications (Handwritten digit recognition, Autonomous Steering and Hand-Gesture recognition) show that this model of inference reduces the number of required operations for sparse input data by a factor of one to two orders of magnitudes. Additionally, due to fully asynchronous processing this type of inference can be run on fully distributed and scalable neuromorphic hardware platforms
Dynamic Quantization using Spike Generation Mechanisms
This paper introduces a neuro-inspired co-ding/decoding mechanism of a constant real value by using a Spike Generation Mechanism (SGM) and a combination of two Spike Interpretation Mechanisms (SIM). One of the most efficient and widely used SGMs to encode a real value is the Leaky-Integrate and Fire (LIF) model which produces a spike train. The duration of the spike train is bounded by a given time constraint. Seeking for a simple solution of how to interpret the spike train and to reconstruct the input value, we combine two different kinds of SIMs, the time-SIM and the rate-SIM. The time-SIM allows a high quality interpretation of the neural code and the rate-SIM allows a simple decoding mechanism by couting the spikes. The resulting coding/decoding process, called the Dual-SIM Quantizer (Dual-SIMQ), is a non-uniform quantizer. It is shown that it coincides with a uniform scalar quantizer under certain assumptions. Finally, it is also shown that the time constraint can be used to control automatically the reconstruction accuracy of this time-dependent quantizer
Asynchronous Spiking Neurons, the Natural Key to Exploit Temporal Sparsity
Inference of Deep Neural Networks for stream
signal (Video/Audio) processing in edge devices is still challenging.
Unlike the most state of the art inference engines which are
efficient for static signals, our brain is optimized for real-time
dynamic signal processing. We believe one important feature of
the brain (asynchronous state-full processing) is the key to its
excellence in this domain. In this work, we show how asynchronous
processing with state-full neurons allows exploitation of the
existing sparsity in natural signals. This paper explains three
different types of sparsity and proposes an inference algorithm
which exploits all types of sparsities in the execution of already
trained networks. Our experiments in three different applications
(Handwritten digit recognition, Autonomous Steering and
Hand-Gesture recognition) show that this model of inference
reduces the number of required operations for sparse input data
by a factor of one to two orders of magnitudes. Additionally,
due to fully asynchronous processing this type of inference can
be run on fully distributed and scalable neuromorphic hardware
platforms.European Union's Horizon 2020 No 687299 NeuRAMEuropean Union's Horizon 2020 No 824164 HERMESMinisterio de EconomĂa y Competitividad TEC2015-63884-C2-1-
Efficient Computation in Adaptive Artificial Spiking Neural Networks
Artificial Neural Networks (ANNs) are bio-inspired models of neural
computation that have proven highly effective. Still, ANNs lack a natural
notion of time, and neural units in ANNs exchange analog values in a
frame-based manner, a computationally and energetically inefficient form of
communication. This contrasts sharply with biological neurons that communicate
sparingly and efficiently using binary spikes. While artificial Spiking Neural
Networks (SNNs) can be constructed by replacing the units of an ANN with
spiking neurons, the current performance is far from that of deep ANNs on hard
benchmarks and these SNNs use much higher firing rates compared to their
biological counterparts, limiting their efficiency. Here we show how spiking
neurons that employ an efficient form of neural coding can be used to construct
SNNs that match high-performance ANNs and exceed state-of-the-art in SNNs on
important benchmarks, while requiring much lower average firing rates. For
this, we use spike-time coding based on the firing rate limiting adaptation
phenomenon observed in biological spiking neurons. This phenomenon can be
captured in adapting spiking neuron models, for which we derive the effective
transfer function. Neural units in ANNs trained with this transfer function can
be substituted directly with adaptive spiking neurons, and the resulting
Adaptive SNNs (AdSNNs) can carry out inference in deep neural networks using up
to an order of magnitude fewer spikes compared to previous SNNs. Adaptive
spike-time coding additionally allows for the dynamic control of neural coding
precision: we show how a simple model of arousal in AdSNNs further halves the
average required firing rate and this notion naturally extends to other forms
of attention. AdSNNs thus hold promise as a novel and efficient model for
neural computation that naturally fits to temporally continuous and
asynchronous applications