927 research outputs found
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
The emergence of brain-inspired neuromorphic computing as a paradigm for edge
AI is motivating the search for high-performance and efficient spiking neural
networks to run on this hardware. However, compared to classical neural
networks in deep learning, current spiking neural networks lack competitive
performance in compelling areas. Here, for sequential and streaming tasks, we
demonstrate how a novel type of adaptive spiking recurrent neural network
(SRNN) is able to achieve state-of-the-art performance compared to other
spiking neural networks and almost reach or exceed the performance of classical
recurrent neural networks (RNNs) while exhibiting sparse activity. From this,
we calculate a 100x energy improvement for our SRNNs over classical RNNs on
the harder tasks. To achieve this, we model standard and adaptive
multiple-timescale spiking neurons as self-recurrent neural units, and leverage
surrogate gradients and auto-differentiation in the PyTorch Deep Learning
framework to efficiently implement backpropagation-through-time, including
learning of the important spiking neuron parameters to adapt our spiking
neurons to the tasks.Comment: 11 pages,5 figure
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a > 100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a >100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks
SuperSpike: Supervised learning in multi-layer spiking neural networks
A vast majority of computation in the brain is performed by spiking neural
networks. Despite the ubiquity of such spiking, we currently lack an
understanding of how biological spiking neural circuits learn and compute
in-vivo, as well as how we can instantiate such capabilities in artificial
spiking circuits in-silico. Here we revisit the problem of supervised learning
in temporally coding multi-layer spiking neural networks. First, by using a
surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based
three factor learning rule capable of training multi-layer networks of
deterministic integrate-and-fire neurons to perform nonlinear computations on
spatiotemporal spike patterns. Second, inspired by recent results on feedback
alignment, we compare the performance of our learning rule under different
credit assignment strategies for propagating output errors to hidden units.
Specifically, we test uniform, symmetric and random feedback, finding that
simpler tasks can be solved with any type of feedback, while more complex tasks
require symmetric feedback. In summary, our results open the door to obtaining
a better scientific understanding of learning and computation in spiking neural
networks by advancing our ability to train them to solve nonlinear problems
involving transformations between different spatiotemporal spike-time patterns
How single neuron properties shape chaotic dynamics and signal transmission in random neural networks
While most models of randomly connected networks assume nodes with simple
dynamics, nodes in realistic highly connected networks, such as neurons in the
brain, exhibit intrinsic dynamics over multiple timescales. We analyze how the
dynamical properties of nodes (such as single neurons) and recurrent
connections interact to shape the effective dynamics in large randomly
connected networks. A novel dynamical mean-field theory for strongly connected
networks of multi-dimensional rate units shows that the power spectrum of the
network activity in the chaotic phase emerges from a nonlinear sharpening of
the frequency response function of single units. For the case of
two-dimensional rate units with strong adaptation, we find that the network
exhibits a state of "resonant chaos", characterized by robust, narrow-band
stochastic oscillations. The coherence of stochastic oscillations is maximal at
the onset of chaos and their correlation time scales with the adaptation
timescale of single units. Surprisingly, the resonance frequency can be
predicted from the properties of isolated units, even in the presence of
heterogeneity in the adaptation parameters. In the presence of these
internally-generated chaotic fluctuations, the transmission of weak,
low-frequency signals is strongly enhanced by adaptation, whereas signal
transmission is not influenced by adaptation in the non-chaotic regime. Our
theoretical framework can be applied to other mechanisms at the level of single
nodes, such as synaptic filtering, refractoriness or spike synchronization.
These results advance our understanding of the interaction between the dynamics
of single units and recurrent connectivity, which is a fundamental step toward
the description of biologically realistic network models in the brain, or, more
generally, networks of other physical or man-made complex dynamical units
Synaptic mechanisms of interference in working memory
Information from preceding trials of cognitive tasks can bias performance in
the current trial, a phenomenon referred to as interference. Subjects
performing visual working memory tasks exhibit interference in their
trial-to-trial response correlations: the recalled target location in the
current trial is biased in the direction of the target presented on the
previous trial. We present modeling work that (a) develops a probabilistic
inference model of this history-dependent bias, and (b) links our probabilistic
model to computations of a recurrent network wherein short-term facilitation
accounts for the dynamics of the observed bias. Network connectivity is
reshaped dynamically during each trial, providing a mechanism for generating
predictions from prior trial observations. Applying timescale separation
methods, we can obtain a low-dimensional description of the trial-to-trial bias
based on the history of target locations. The model has response statistics
whose mean is centered at the true target location across many trials, typical
of such visual working memory tasks. Furthermore, we demonstrate task protocols
for which the plastic model performs better than a model with static
connectivity: repetitively presented targets are better retained in working
memory than targets drawn from uncorrelated sequences.Comment: 28 pages, 7 figure
Principles of Neuromorphic Photonics
In an age overrun with information, the ability to process reams of data has
become crucial. The demand for data will continue to grow as smart gadgets
multiply and become increasingly integrated into our daily lives.
Next-generation industries in artificial intelligence services and
high-performance computing are so far supported by microelectronic platforms.
These data-intensive enterprises rely on continual improvements in hardware.
Their prospects are running up against a stark reality: conventional
one-size-fits-all solutions offered by digital electronics can no longer
satisfy this need, as Moore's law (exponential hardware scaling),
interconnection density, and the von Neumann architecture reach their limits.
With its superior speed and reconfigurability, analog photonics can provide
some relief to these problems; however, complex applications of analog
photonics have remained largely unexplored due to the absence of a robust
photonic integration industry. Recently, the landscape for
commercially-manufacturable photonic chips has been changing rapidly and now
promises to achieve economies of scale previously enjoyed solely by
microelectronics.
The scientific community has set out to build bridges between the domains of
photonic device physics and neural networks, giving rise to the field of
\emph{neuromorphic photonics}. This article reviews the recent progress in
integrated neuromorphic photonics. We provide an overview of neuromorphic
computing, discuss the associated technology (microelectronic and photonic)
platforms and compare their metric performance. We discuss photonic neural
network approaches and challenges for integrated neuromorphic photonic
processors while providing an in-depth description of photonic neurons and a
candidate interconnection architecture. We conclude with a future outlook of
neuro-inspired photonic processing.Comment: 28 pages, 19 figure
- …