6,022 research outputs found
Training Echo State Networks with Regularization through Dimensionality Reduction
In this paper we introduce a new framework to train an Echo State Network to
predict real valued time-series. The method consists in projecting the output
of the internal layer of the network on a space with lower dimensionality,
before training the output layer to learn the target task. Notably, we enforce
a regularization constraint that leads to better generalization capabilities.
We evaluate the performances of our approach on several benchmark tests, using
different techniques to train the readout of the network, achieving superior
predictive performance when using the proposed framework. Finally, we provide
an insight on the effectiveness of the implemented mechanics through a
visualization of the trajectory in the phase space and relying on the
methodologies of nonlinear time-series analysis. By applying our method on well
known chaotic systems, we provide evidence that the lower dimensional embedding
retains the dynamical properties of the underlying system better than the
full-dimensional internal states of the network
Impact of noise on a dynamical system: prediction and uncertainties from a swarm-optimized neural network
In this study, an artificial neural network (ANN) based on particle swarm
optimization (PSO) was developed for the time series prediction. The hybrid
ANN+PSO algorithm was applied on Mackey--Glass chaotic time series in the
short-term . The performance prediction was evaluated and compared with
another studies available in the literature. Also, we presented properties of
the dynamical system via the study of chaotic behaviour obtained from the
predicted time series. Next, the hybrid ANN+PSO algorithm was complemented with
a Gaussian stochastic procedure (called {\it stochastic} hybrid ANN+PSO) in
order to obtain a new estimator of the predictions, which also allowed us to
compute uncertainties of predictions for noisy Mackey--Glass chaotic time
series. Thus, we studied the impact of noise for several cases with a white
noise level () from 0.01 to 0.1.Comment: 11 pages, 8 figure
Complexity without chaos: Plasticity within random recurrent networks generates robust timing and motor control
It is widely accepted that the complex dynamics characteristic of recurrent
neural circuits contributes in a fundamental manner to brain function. Progress
has been slow in understanding and exploiting the computational power of
recurrent dynamics for two main reasons: nonlinear recurrent networks often
exhibit chaotic behavior and most known learning rules do not work in robust
fashion in recurrent networks. Here we address both these problems by
demonstrating how random recurrent networks (RRN) that initially exhibit
chaotic dynamics can be tuned through a supervised learning rule to generate
locally stable neural patterns of activity that are both complex and robust to
noise. The outcome is a novel neural network regime that exhibits both
transiently stable and chaotic trajectories. We further show that the recurrent
learning rule dramatically increases the ability of RRNs to generate complex
spatiotemporal motor patterns, and accounts for recent experimental data
showing a decrease in neural variability in response to stimulus onset
Comparison of echo state network output layer classification methods on noisy data
Echo state networks are a recently developed type of recurrent neural network
where the internal layer is fixed with random weights, and only the output
layer is trained on specific data. Echo state networks are increasingly being
used to process spatiotemporal data in real-world settings, including speech
recognition, event detection, and robot control. A strength of echo state
networks is the simple method used to train the output layer - typically a
collection of linear readout weights found using a least squares approach.
Although straightforward to train and having a low computational cost to use,
this method may not yield acceptable accuracy performance on noisy data.
This study compares the performance of three echo state network output layer
methods to perform classification on noisy data: using trained linear weights,
using sparse trained linear weights, and using trained low-rank approximations
of reservoir states. The methods are investigated experimentally on both
synthetic and natural datasets. The experiments suggest that using regularized
least squares to train linear output weights is superior on data with low
noise, but using the low-rank approximations may significantly improve accuracy
on datasets contaminated with higher noise levels.Comment: 8 pages. International Joint Conference on Neural Networks (IJCNN
2017
Exploring Transfer Function Nonlinearity in Echo State Networks
Supralinear and sublinear pre-synaptic and dendritic integration is
considered to be responsible for nonlinear computation power of biological
neurons, emphasizing the role of nonlinear integration as opposed to nonlinear
output thresholding. How, why, and to what degree the transfer function
nonlinearity helps biologically inspired neural network models is not fully
understood. Here, we study these questions in the context of echo state
networks (ESN). ESN is a simple neural network architecture in which a fixed
recurrent network is driven with an input signal, and the output is generated
by a readout layer from the measurements of the network states. ESN
architecture enjoys efficient training and good performance on certain
signal-processing tasks, such as system identification and time series
prediction. ESN performance has been analyzed with respect to the connectivity
pattern in the network structure and the input bias. However, the effects of
the transfer function in the network have not been studied systematically.
Here, we use an approach tanh on the Taylor expansion of a frequently used
transfer function, the hyperbolic tangent function, to systematically study the
effect of increasing nonlinearity of the transfer function on the memory,
nonlinear capacity, and signal processing performance of ESN. Interestingly, we
find that a quadratic approximation is enough to capture the computational
power of ESN with tanh function. The results of this study apply to both
software and hardware implementation of ESN.Comment: arXiv admin note: text overlap with arXiv:1502.0071
Investigating the Predictability of a Chaotic Time-Series Data using Reservoir Computing, Deep-Learning and Machine- Learning on the Short-, Medium- and Long-Term Pricing of Bitcoin and Ethereum.
This study will investigate the predictability of a Chaotic time-series data using Reservoir computing (Echo State Network), Deep-Learning(LSTM) and Machine- Learning(Linear, Bayesian, ElasticNetCV , Random Forest, XGBoost Regression and a machine learning Neural Network) on the short (1-day out prediction), medium (5-day out prediction) and long-term (30-day out prediction) pricing of Bitcoin and Ethereum Using a range of machine learning tools, to perform feature selection by permutation importance to select technical indicators on the individual cryptocurrencies, to ensure the datasets are the best for predictions per cryptocurrency while reducing noise within the models. The predictability of these two chaotic time-series is then compared to evaluate the models to find the best fit model. The models are fine-tuned, with hyperparameters, design of the network within the LSTM and the reservoir size within the Echo State Network being adjusted to improve accuracy and speed. This research highlights the effect of the trends within the cryptocurrency and its effect on predictive models, these models will then be optimized with hyperparameter tuning, and be evaluated to compare the models across the two currencies. It is found that the datasets for each cryptocurrency are different, due to the different permutation importance, which does not affect the overall predictability of the models with the short and medium-term predictions having the same models being the top performers. This research confirms that the chaotic data although can have positive results for shortand medium-term prediction, for long-term prediction, technical analysis basedprediction is not sufficient
- …