33,258 research outputs found
Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis
How can we tell whether two neural networks are utilizing the same internal
processes for a particular computation? This question is pertinent for multiple
subfields of both neuroscience and machine learning, including neuroAI,
mechanistic interpretability, and brain-machine interfaces. Standard approaches
for comparing neural networks focus on the spatial geometry of latent states.
Yet in recurrent networks, computations are implemented at the level of neural
dynamics, which do not have a simple one-to-one mapping with geometry. To
bridge this gap, we introduce a novel similarity metric that compares two
systems at the level of their dynamics. Our method incorporates two components:
Using recent advances in data-driven dynamical systems theory, we learn a
high-dimensional linear system that accurately captures core features of the
original nonlinear dynamics. Next, we compare these linear approximations via a
novel extension of Procrustes Analysis that accounts for how vector fields
change under orthogonal transformation. Via four case studies, we demonstrate
that our method effectively identifies and distinguishes dynamic structure in
recurrent neural networks (RNNs), whereas geometric methods fall short. We
additionally show that our method can distinguish learning rules in an
unsupervised manner. Our method therefore opens the door to novel data-driven
analyses of the temporal structure of neural computation, and to more rigorous
testing of RNNs as models of the brain.Comment: 21 pages, 10 figure
Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware
In recent years the field of neuromorphic low-power systems that consume
orders of magnitude less power gained significant momentum. However, their
wider use is still hindered by the lack of algorithms that can harness the
strengths of such architectures. While neuromorphic adaptations of
representation learning algorithms are now emerging, efficient processing of
temporal sequences or variable length-inputs remain difficult. Recurrent neural
networks (RNN) are widely used in machine learning to solve a variety of
sequence learning tasks. In this work we present a train-and-constrain
methodology that enables the mapping of machine learned (Elman) RNNs on a
substrate of spiking neurons, while being compatible with the capabilities of
current and near-future neuromorphic systems. This "train-and-constrain" method
consists of first training RNNs using backpropagation through time, then
discretizing the weights and finally converting them to spiking RNNs by
matching the responses of artificial neurons with those of the spiking neurons.
We demonstrate our approach by mapping a natural language processing task
(question classification), where we demonstrate the entire mapping process of
the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a
spike-based digital neuromorphic hardware architecture. TrueNorth imposes
specific constraints on connectivity, neural and synaptic parameters. To
satisfy these constraints, it was necessary to discretize the synaptic weights
and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find
that short synaptic delays are sufficient to implement the dynamical (temporal)
aspect of the RNN in the question classification task. The hardware-constrained
model achieved 74% accuracy in question classification while using less than
0.025% of the cores on one TrueNorth chip, resulting in an estimated power
consumption of ~17 uW
Neural Networks for Information Retrieval
Machine learning plays a role in many aspects of modern IR systems, and deep
learning is applied in all of them. The fast pace of modern-day research has
given rise to many different approaches for many different IR problems. The
amount of information available can be overwhelming both for junior students
and for experienced researchers looking for new research topics and directions.
Additionally, it is interesting to see what key insights into IR problems the
new technologies are able to give us. The aim of this full-day tutorial is to
give a clear overview of current tried-and-trusted neural methods in IR and how
they benefit IR research. It covers key architectures, as well as the most
promising future directions.Comment: Overview of full-day tutorial at SIGIR 201
- …