41,046 research outputs found
Interpreting recurrent neural networks behaviour via excitable network attractors
Introduction: Machine learning provides fundamental tools both for scientific
research and for the development of technologies with significant impact on
society. It provides methods that facilitate the discovery of regularities in
data and that give predictions without explicit knowledge of the rules
governing a system. However, a price is paid for exploiting such flexibility:
machine learning methods are typically black-boxes where it is difficult to
fully understand what the machine is doing or how it is operating. This poses
constraints on the applicability and explainability of such methods. Methods:
Our research aims to open the black-box of recurrent neural networks, an
important family of neural networks used for processing sequential data. We
propose a novel methodology that provides a mechanistic interpretation of
behaviour when solving a computational task. Our methodology uses mathematical
constructs called excitable network attractors, which are invariant sets in
phase space composed of stable attractors and excitable connections between
them. Results and Discussion: As the behaviour of recurrent neural networks
depends both on training and on inputs to the system, we introduce an algorithm
to extract network attractors directly from the trajectory of a neural network
while solving tasks. Simulations conducted on a controlled benchmark task
confirm the relevance of these attractors for interpreting the behaviour of
recurrent neural networks, at least for tasks that involve learning a finite
number of stable states and transitions between them.Comment: revised versio
Recurrent backpropagation and the dynamical approach to adaptive neural computation
Error backpropagation in feedforward neural network models is a popular learning algorithm that has its roots in nonlinear estimation and optimization. It is being used routinely to calculate error gradients in nonlinear systems with hundreds of thousands of parameters. However, the classical architecture for backpropagation has severe restrictions. The extension of backpropagation to networks with recurrent connections will be reviewed. It is now possible to efficiently compute the error gradients for networks that have temporal dynamics, which opens applications to a host of problems in systems identification and control
A geometrical analysis of global stability in trained feedback networks
Recurrent neural networks have been extensively studied in the context of
neuroscience and machine learning due to their ability to implement complex
computations. While substantial progress in designing effective learning
algorithms has been achieved in the last years, a full understanding of trained
recurrent networks is still lacking. Specifically, the mechanisms that allow
computations to emerge from the underlying recurrent dynamics are largely
unknown. Here we focus on a simple, yet underexplored computational setup: a
feedback architecture trained to associate a stationary output to a stationary
input. As a starting point, we derive an approximate analytical description of
global dynamics in trained networks which assumes uncorrelated connectivity
weights in the feedback and in the random bulk. The resulting mean-field theory
suggests that the task admits several classes of solutions, which imply
different stability properties. Different classes are characterized in terms of
the geometrical arrangement of the readout with respect to the input vectors,
defined in the high-dimensional space spanned by the network population. We
find that such approximate theoretical approach can be used to understand how
standard training techniques implement the input-output task in finite-size
feedback networks. In particular, our simplified description captures the local
and the global stability properties of the target solution, and thus predicts
training performance
full-FORCE: A Target-Based Method for Training Recurrent Networks
Trained recurrent networks are powerful tools for modeling dynamic neural
computations. We present a target-based method for modifying the full
connectivity matrix of a recurrent network to train it to perform tasks
involving temporally complex input/output transformations. The method
introduces a second network during training to provide suitable "target"
dynamics useful for performing the task. Because it exploits the full recurrent
connectivity, the method produces networks that perform tasks with fewer
neurons and greater noise robustness than traditional least-squares (FORCE)
approaches. In addition, we show how introducing additional input signals into
the target-generating network, which act as task hints, greatly extends the
range of tasks that can be learned and provides control over the complexity and
nature of the dynamics of the trained, task-performing network.Comment: 20 pages, 8 figure
- …