22,765 research outputs found
Richness of Deep Echo State Network Dynamics
Reservoir Computing (RC) is a popular methodology for the efficient design of
Recurrent Neural Networks (RNNs). Recently, the advantages of the RC approach
have been extended to the context of multi-layered RNNs, with the introduction
of the Deep Echo State Network (DeepESN) model. In this paper, we study the
quality of state dynamics in progressively higher layers of DeepESNs, using
tools from the areas of information theory and numerical analysis. Our
experimental results on RC benchmark datasets reveal the fundamental role
played by the strength of inter-reservoir connections to increasingly enrich
the representations developed in higher layers. Our analysis also gives
interesting insights into the possibility of effective exploitation of training
algorithms based on stochastic gradient descent in the RC field.Comment: Preprint of the paper accepted at IWANN 201
Short-term Memory of Deep RNN
The extension of deep learning towards temporal data processing is gaining an
increasing research interest. In this paper we investigate the properties of
state dynamics developed in successive levels of deep recurrent neural networks
(RNNs) in terms of short-term memory abilities. Our results reveal interesting
insights that shed light on the nature of layering as a factor of RNN design.
Noticeably, higher layers in a hierarchically organized RNN architecture
results to be inherently biased towards longer memory spans even prior to
training of the recurrent connections. Moreover, in the context of Reservoir
Computing framework, our analysis also points out the benefit of a layered
recurrent organization as an efficient approach to improve the memory skills of
reservoir models.Comment: This is a pre-print (pre-review) version of the paper accepted for
presentation at the 26th European Symposium on Artificial Neural Networks,
Computational Intelligence and Machine Learning (ESANN), Bruges (Belgium),
25-27 April 201
Deep Randomized Neural Networks
Randomized Neural Networks explore the behavior of neural systems where the
majority of connections are fixed, either in a stochastic or a deterministic
fashion. Typical examples of such systems consist of multi-layered neural
network architectures where the connections to the hidden layer(s) are left
untrained after initialization. Limiting the training algorithms to operate on
a reduced set of weights inherently characterizes the class of Randomized
Neural Networks with a number of intriguing features. Among them, the extreme
efficiency of the resulting learning processes is undoubtedly a striking
advantage with respect to fully trained architectures. Besides, despite the
involved simplifications, randomized neural systems possess remarkable
properties both in practice, achieving state-of-the-art results in multiple
domains, and theoretically, allowing to analyze intrinsic properties of neural
architectures (e.g. before training of the hidden layers' connections). In
recent years, the study of Randomized Neural Networks has been extended towards
deep architectures, opening new research directions to the design of effective
yet extremely efficient deep learning models in vectorial as well as in more
complex data domains. This chapter surveys all the major aspects regarding the
design and analysis of Randomized Neural Networks, and some of the key results
with respect to their approximation capabilities. In particular, we first
introduce the fundamentals of randomized neural models in the context of
feed-forward networks (i.e., Random Vector Functional Link and equivalent
models) and convolutional filters, before moving to the case of recurrent
systems (i.e., Reservoir Computing networks). For both, we focus specifically
on recent results in the domain of deep randomized systems, and (for recurrent
models) their application to structured domains
Dynamical systems as temporal feature spaces
Parameterized state space models in the form of recurrent networks are often
used in machine learning to learn from data streams exhibiting temporal
dependencies. To break the black box nature of such models it is important to
understand the dynamical features of the input driving time series that are
formed in the state space. We propose a framework for rigorous analysis of such
state representations in vanishing memory state space models such as echo state
networks (ESN). In particular, we consider the state space a temporal feature
space and the readout mapping from the state space a kernel machine operating
in that feature space. We show that: (1) The usual ESN strategy of randomly
generating input-to-state, as well as state coupling leads to shallow memory
time series representations, corresponding to cross-correlation operator with
fast exponentially decaying coefficients; (2) Imposing symmetry on dynamic
coupling yields a constrained dynamic kernel matching the input time series
with straightforward exponentially decaying motifs or exponentially decaying
motifs of the highest frequency; (3) Simple cycle high-dimensional reservoir
topology specified only through two free parameters can implement deep memory
dynamic kernels with a rich variety of matching motifs. We quantify richness of
feature representations imposed by dynamic kernels and demonstrate that for
dynamic kernel associated with cycle reservoir topology, the kernel richness
undergoes a phase transition close to the edge of stability.Comment: 45 pages, 17 figures, accepte
Deep Randomized Neural Networks
Randomized Neural Networks explore the behavior of neural systems where the majority of connections are fixed, either in a stochastic or a deterministic fashion. Typical examples of such systems consist of multi-layered neural network architectures where the connections to the hidden layer(s) are left untrained after initialization. Limiting the training algorithms to operate on a reduced set of weights inherently characterizes the class of Randomized Neural Networks with a number of intriguing features. Among them, the extreme efficiency of the resulting learning processes is undoubtedly a striking advantage with respect to fully trained architectures. Besides, despite the involved simplifications, randomized neural systems possess remarkable properties both in practice, achieving state-of-the-art results in multiple domains, and theoretically, allowing to analyze intrinsic properties of neural architectures (e.g. before training of the hidden layers’ connections). In recent years, the study of Randomized Neural Networks has been extended towards deep architectures, opening new research directions to the design of effective yet extremely efficient deep learning models in vectorial as well as in more complex data domains. This chapter surveys all the major aspects regarding the design and analysis of Randomized Neural Networks, and some of the key results with respect to their approximation capabilities. In particular, we first introduce the fundamentals of randomized neural models in the context of feed-forward networks (i.e., Random Vector Functional Link and equivalent models) and convolutional filters, before moving to the case of recurrent systems (i.e., Reservoir Computing networks). For both, we focus specifically on recent results in the domain of deep randomized systems, and (for recurrent models) their application to structured domains
Neuromorphic Few-Shot Learning: Generalization in Multilayer Physical Neural Networks
Neuromorphic computing leverages the complex dynamics of physical systems for
computation. The field has recently undergone an explosion in the range and
sophistication of implementations, with rapidly improving performance.
Neuromorphic schemes typically employ a single physical system, limiting the
dimensionality and range of available dynamics - restricting strong performance
to a few specific tasks. This is a critical roadblock facing the field,
inhibiting the power and versatility of neuromorphic schemes.
Here, we present a solution. We engineer a diverse suite of nanomagnetic
arrays and show how tuning microstate space and geometry enables a broad range
of dynamics and computing performance. We interconnect arrays in parallel,
series and multilayered neural network architectures, where each network node
is a distinct physical system. This networked approach grants extremely high
dimensionality and enriched dynamics enabling meta-learning to be implemented
on small training sets and exhibiting strong performance across a broad
taskset. We showcase network performance via few-shot learning, rapidly
adapting on-the-fly to previously unseen tasks
- …