15 research outputs found
Efficient Optimization of Echo State Networks for Time Series Datasets
Echo State Networks (ESNs) are recurrent neural networks that only train
their output layer, thereby precluding the need to backpropagate gradients
through time, which leads to significant computational gains. Nevertheless, a
common issue in ESNs is determining its hyperparameters, which are crucial in
instantiating a well performing reservoir, but are often set manually or using
heuristics. In this work we optimize the ESN hyperparameters using Bayesian
optimization which, given a limited budget of function evaluations, outperforms
a grid search strategy. In the context of large volumes of time series data,
such as light curves in the field of astronomy, we can further reduce the
optimization cost of ESNs. In particular, we wish to avoid tuning
hyperparameters per individual time series as this is costly; instead, we want
to find ESNs with hyperparameters that perform well not just on individual time
series but rather on groups of similar time series without sacrificing
predictive performance significantly. This naturally leads to a notion of
clusters, where each cluster is represented by an ESN tuned to model a group of
time series of similar temporal behavior. We demonstrate this approach both on
synthetic datasets and real world light curves from the MACHO survey. We show
that our approach results in a significant reduction in the number of ESN
models required to model a whole dataset, while retaining predictive
performance for the series in each cluster
Reservoir Topology in Deep Echo State Networks
Deep Echo State Networks (DeepESNs) recently extended the applicability of
Reservoir Computing (RC) methods towards the field of deep learning. In this
paper we study the impact of constrained reservoir topologies in the
architectural design of deep reservoirs, through numerical experiments on
several RC benchmarks. The major outcome of our investigation is to show the
remarkable effect, in terms of predictive performance gain, achieved by the
synergy between a deep reservoir construction and a structured organization of
the recurrent units in each layer. Our results also indicate that a
particularly advantageous architectural setting is obtained in correspondence
of DeepESNs where reservoir units are structured according to a permutation
recurrent matrix.Comment: Preprint of the paper published in the proceedings of ICANN 201
Designing a NISQ reservoir with maximal memory capacity for volatility forecasting
Forecasting the CBOE volatility index (VIX) is a highly non-linear and
memory-intensive task. In this paper, we use quantum reservoir computing to
forecast the VIX using S&P500 (SPX) time-series. Our reservoir is a hybrid
quantum-classical system executed on IBM's 53-qubit Rochester chip. We encode
the SPX values in the rotation angles and linearly combine the average spin of
the six-qubit register to predict the value of VIX at the next time step. Our
results demonstrate a potential application of noisy intermediate scale quantum
(NISQ) devices to complex, real world applications
Reservoir Memory Machines as Neural Computers
Differentiable neural computers extend artificial neural networks with an
explicit memory without interference, thus enabling the model to perform
classic computation tasks such as graph traversal. However, such models are
difficult to train, requiring long training times and large datasets. In this
work, we achieve some of the computational capabilities of differentiable
neural computers with a model that can be trained very efficiently, namely an
echo state network with an explicit memory without interference. This extension
enables echo state networks to recognize all regular languages, including those
that contractive echo state networks provably can not recognize. Further, we
demonstrate experimentally that our model performs comparably to its
fully-trained deep version on several typical benchmark tasks for
differentiable neural computers.Comment: In print at the special issue 'New Frontiers in Extremely Efficient
Reservoir Computing' of IEEE TNNL
Deep Randomized Neural Networks
Randomized Neural Networks explore the behavior of neural systems where the
majority of connections are fixed, either in a stochastic or a deterministic
fashion. Typical examples of such systems consist of multi-layered neural
network architectures where the connections to the hidden layer(s) are left
untrained after initialization. Limiting the training algorithms to operate on
a reduced set of weights inherently characterizes the class of Randomized
Neural Networks with a number of intriguing features. Among them, the extreme
efficiency of the resulting learning processes is undoubtedly a striking
advantage with respect to fully trained architectures. Besides, despite the
involved simplifications, randomized neural systems possess remarkable
properties both in practice, achieving state-of-the-art results in multiple
domains, and theoretically, allowing to analyze intrinsic properties of neural
architectures (e.g. before training of the hidden layers' connections). In
recent years, the study of Randomized Neural Networks has been extended towards
deep architectures, opening new research directions to the design of effective
yet extremely efficient deep learning models in vectorial as well as in more
complex data domains. This chapter surveys all the major aspects regarding the
design and analysis of Randomized Neural Networks, and some of the key results
with respect to their approximation capabilities. In particular, we first
introduce the fundamentals of randomized neural models in the context of
feed-forward networks (i.e., Random Vector Functional Link and equivalent
models) and convolutional filters, before moving to the case of recurrent
systems (i.e., Reservoir Computing networks). For both, we focus specifically
on recent results in the domain of deep randomized systems, and (for recurrent
models) their application to structured domains