The Power of Linear Recurrent Neural Networks

Recurrent neural networks are a powerful means to cope with time series. We show how a type of linearly activated recurrent neural networks, which we call predictive neural networks, can approximate any time-dependent function f(t) given by a number of function values. The approximation can effectively be learned by simply solving a linear equation system; no backpropagation or similar methods are needed. Furthermore, the network size can be reduced by taking only most relevant components. Thus, in contrast to others, our approach not only learns network weights but also the network architecture. The networks have interesting properties: They end up in ellipse trajectories in the long run and allow the prediction of further values and compact representations of functions. We demonstrate this by several experiments, among them multiple superimposed oscillators (MSO), robotic soccer, and predicting stock prices. Predictive neural networks outperform the previous state-of-the-art for the MSO task with a minimal number of units.Comment: 22 pages, 14 figures and tables, revised implementatio

Excess risk bound for deep learning under weak dependence

This paper considers deep neural networks for learning weakly dependent processes in a general framework that includes, for instance, regression estimation, time series prediction, time series classification. The $\psi$-weak dependence structure considered is quite large and covers other conditions such as mixing, association,$\ldots$ Firstly, the approximation of smooth functions by deep neural networks with a broad class of activation functions is considered. We derive the required depth, width and sparsity of a deep neural network to approximate any H\"{o}lder smooth function, defined on any compact set \mx. Secondly, we establish a bound of the excess risk for the learning of weakly dependent observations by deep neural networks. When the target function is sufficiently smooth, this bound is close to the usual $\mathcal{O}(n^{-1/2})$

Minimum description length quality measurues for modular functional network architectures

Modular neural networks (MNNs) are increasingly popular models for dealing with complex problems constituted by a number of dependent subtasks. An important problem on MNNs is finding the optimal aggregation of the neural modules, each of them dealing with one of the subproblems. In this paper, we present a functional network approach, based on the minimum description length quality measure, to the problem of finding optimal modular network architectures for specific problems. Examples of function approximation and nonlinear time series prediction are used to illustrate the performance of these models when compared with standard functional and neural networks.Eje: Sistemas inteligentesRed de Universidades con Carreras en Informática (RedUNCI

Exploring Transfer Function Nonlinearity in Echo State Networks

Supralinear and sublinear pre-synaptic and dendritic integration is considered to be responsible for nonlinear computation power of biological neurons, emphasizing the role of nonlinear integration as opposed to nonlinear output thresholding. How, why, and to what degree the transfer function nonlinearity helps biologically inspired neural network models is not fully understood. Here, we study these questions in the context of echo state networks (ESN). ESN is a simple neural network architecture in which a fixed recurrent network is driven with an input signal, and the output is generated by a readout layer from the measurements of the network states. ESN architecture enjoys efficient training and good performance on certain signal-processing tasks, such as system identification and time series prediction. ESN performance has been analyzed with respect to the connectivity pattern in the network structure and the input bias. However, the effects of the transfer function in the network have not been studied systematically. Here, we use an approach tanh on the Taylor expansion of a frequently used transfer function, the hyperbolic tangent function, to systematically study the effect of increasing nonlinearity of the transfer function on the memory, nonlinear capacity, and signal processing performance of ESN. Interestingly, we find that a quadratic approximation is enough to capture the computational power of ESN with tanh function. The results of this study apply to both software and hardware implementation of ESN.Comment: arXiv admin note: text overlap with arXiv:1502.0071

Minimum description length quality measurues for modular functional network architectures

Modular neural networks (MNNs) are increasingly popular models for dealing with complex problems constituted by a number of dependent subtasks. An important problem on MNNs is finding the optimal aggregation of the neural modules, each of them dealing with one of the subproblems. In this paper, we present a functional network approach, based on the minimum description length quality measure, to the problem of finding optimal modular network architectures for specific problems. Examples of function approximation and nonlinear time series prediction are used to illustrate the performance of these models when compared with standard functional and neural networks.Eje: Sistemas inteligentesRed de Universidades con Carreras en Informática (RedUNCI

Advanced approach to numerical forecasting using artifi cial neural networks.

Abstract ŠTENCL, M., ŠŤASTNÝ, J.: Advanced approach to numerical forecasting using artifi cial neural networks. Acta univ. agric. et silvic. Mendel. Brun., 2009, LVII, No. 6, pp. 297-304 Current global market is driven by many factors, such as the information age, the time and amount of information distributed by many data channels it is practically impossible analyze all kinds of incoming information fl ows and transform them to data with classical methods. New requirements could be met by using other methods. Once trained on patterns artifi cial neural networks can be used for forecasting and they are able to work with extremely big data sets in reasonable time. The patterns used for learning process are samples of past data. This paper uses Radial Basis Functions neural network in comparison with Multi Layer Perceptron network with Back-propagation learning algorithm on prediction task. The task works with simplifi ed numerical time series and includes forty observations with prediction for next fi ve observations. The main topic of the article is the identifi cation of the main diff erences between used neural networks architectures together with numerical forecasting. Detected diff erences then verify on practical comparative example. Artifi cial Neural Networks, Radial basis function, Numerical Forecasting, Multi Layer Perceptron Network The knowledge of the future creates the advantage in all kind of business. The methods traditionally used for numerical forecasting are based on precise analysis of past values. The prognosis is then built as approximation of future values using functions estimated from dependencies founded by past values analysis. The statistical time series model is used as a traditional method for economical forecasting