Variational inference for robust sequential learning of multilayered perceptron neural network

U radu je prikazan i izveden novi sekvencijalni algoritam za obučavanje višeslojnog perceptrona u prisustvu autlajera. Autlajeri predstavljaju značajan problem, posebno ukoliko sprovodimo sekvencijalno obučavanje ili obučavanje u realnom vremenu. Linearizovani Kalmanov filtar robustan na autlajere (LKF-RA), je statistički generativni model u kome je matrica kovarijansi šuma merenja modelovana kao stohastički proces, a apriorna informacija usvojena kao inverzna Višartova raspodela. Izvođenje svih jednakosti je bazirano na prvim principima Bajesovske metodologije. Da bi se rešio korak modifikacije primenjen je varijacioni metod, u kome rešenje problema tražimo u familiji raspodela odgovarajuće funkcionalne forme. Eksperimentalni rezultati primene LKF-RA, dobijeni korišćenjem stvarnih vremenskih serija, pokazuju da je LKF-RA bolji od konvencionalnog linearizovanog Kalmanovog filtra u smislu generisanja niže greške na test skupu podataka. Prosečna vrednost poboljšanja određena u eksperimentalnom procesu je 7%.We derive a new sequential learning algorithm for Multilayered Perceptron (MLP) neural network robust to outliers. Presence of outliers in data results in failure of the model especially if data processing is performed on-line or in real time. Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is given as inverse Wishart distribution. Derivation of expressions comes straight form first principles, within Bayesian framework. Analytical intractability of Bayes' update step is solved using Variational Inference (VI). Experimental results obtained using real world stochastic data show that MLP network trained with proposed algorithm achieves low error and average improvement rate of 7% when compared directly to conventional EKF learning algorithm

Variational inference for robust sequential learning of multilayered perceptron neural network

U radu je prikazan i izveden novi sekvencijalni algoritam za obučavanje višeslojnog perceptrona u prisustvu autlajera. Autlajeri predstavljaju značajan problem, posebno ukoliko sprovodimo sekvencijalno obučavanje ili obučavanje u realnom vremenu. Linearizovani Kalmanov filtar robustan na autlajere (LKF-RA), je statistički generativni model u kome je matrica kovarijansi šuma merenja modelovana kao stohastički proces, a apriorna informacija usvojena kao inverzna Višartova raspodela. Izvođenje svih jednakosti je bazirano na prvim principima Bajesovske metodologije. Da bi se rešio korak modifikacije primenjen je varijacioni metod, u kome rešenje problema tražimo u familiji raspodela odgovarajuće funkcionalne forme. Eksperimentalni rezultati primene LKF-RA, dobijeni korišćenjem stvarnih vremenskih serija, pokazuju da je LKF-RA bolji od konvencionalnog linearizovanog Kalmanovog filtra u smislu generisanja niže greške na test skupu podataka. Prosečna vrednost poboljšanja određena u eksperimentalnom procesu je 7%.We derive a new sequential learning algorithm for Multilayered Perceptron (MLP) neural network robust to outliers. Presence of outliers in data results in failure of the model especially if data processing is performed on-line or in real time. Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is given as inverse Wishart distribution. Derivation of expressions comes straight form first principles, within Bayesian framework. Analytical intractability of Bayes' update step is solved using Variational Inference (VI). Experimental results obtained using real world stochastic data show that MLP network trained with proposed algorithm achieves low error and average improvement rate of 7% when compared directly to conventional EKF learning algorithm

Using an extended Kalman filter learning algorithm for feed-forward neural networks to describe tracer correlations

International audienceIn this study a new extended Kalman filter (EKF) learning algorithm for feed-forward neural networks (FFN) is used. With the EKF approach, the training of the FFN can be seen as state estimation for a non-linear stationary process. The EKF method gives excellent convergence performances provided that there is enough computer core memory and that the machine precision is high. Neural networks are ideally suited to describe the spatial and temporal dependence of tracer-tracer correlations. The neural network performs well even in regions where the correlations are less compact and normally a family of correlation curves would be required. For example, the CH4-N2O correlation can be well described using a neural network trained with the latitude, pressure, time of year, and CH4 volume mixing ratio (v.m.r.). The neural network was able to reproduce the CH4-N2O correlation with a correlation coefficient between simulated and training values of 0.9997. The neural network Fortran code used is available for download

Extended Kalman filter-based learning of interval type-2 intuitionistic fuzzy logic system

Fuzzy logic systems have been extensively applied for solving many real world application problems because they are found to be universal approximators and many methods, particularly, gradient descent (GD) methods have been widely adopted for the optimization of fuzzy membership functions. Despite its popularity, GD still suffers some drawbacks in terms of its slow learning and convergence. In this study, the use of decoupled extended Kalman filter (DEKF) to optimize the parameters of an interval type-2 intuitionistic fuzzy logic system of Tagagi-Sugeno-Kang (IT2IFLS-TSK) fuzzy inference is proposed and results compared with IT2IFLS gradient descent learning. The resulting systems are evaluated on a real world dataset from Australia’s electricity market. The IT2IFLS-DEKF is also compared with its type-1 variant and interval type-2 fuzzy logic system (IT2FLS). Analysis of results reveal performance superiority of IT2IFLS trained with DEKF (IT2IFLS-DEKF) over IT2IFLS trained with gradient descent (IT2IFLS-GD). The proposed IT2IFLS-DEKF also outperforms its type-1 variant and IT2FLS on the same learning platform

Query-Based Learning for Aerospace Applications

Models of real-world applications often include a large number of parameters with a wide dynamic range, which contributes to the difficulties of neural network training. Creating the training data set for such applications becomes costly, if not impossible. In order to overcome the challenge, one can employ an active learning technique known as query-based learning (QBL) to add performance-critical data to the training set during the learning phase, thereby efficiently improving the overall learning/generalization. The performance-critical data can be obtained using an inverse mapping called network inversion (discrete network inversion and continuous network inversion) followed by oracle query. This paper investigates the use of both inversion techniques for QBL learning, and introduces an original heuristic to select the inversion target values for continuous network inversion method. Efficiency and generalization was further enhanced by employing node decoupled extended Kalman filter (NDEKF) training and a causality index (CI) as a means to reduce the input search dimensionality. The benefits of the overall QBL approach are experimentally demonstrated in two aerospace applications: a classification problem with large input space and a control distribution problem

Application Of Cascade-Correlation Neural Networks In Developing Stock Selection Models For Global Equities

We investigate the potential of artificial neural networks (ANN) in the stock selection process of actively managed funds. Two ANN models are constructed to perform stock selection, using the Dow Jones (DJ) Sector Titans as the research database. The cascade-correlation algorithm of Fahlman and Lebiere (1990/1991) is combined with embedded learning rules, namely the backpropagation learning rule and the extended Kalman filter learning rule to forecast the cross-section of global equity returns. The main findings support the use of artificial neural networks for financial forecasting as an active portfolio management tool. In particular, fractile analysis and risk-adjusted return performance metrics provide evidence that the model trained via the extended Kalman filter rule had greater strength in identifying future top performers for global equities than the model trained via the backpropagation learning rule. There is no distinguishable difference between the performances of the bottom quartiles formed by both ANN models. The zero-investment portfolios formed by longing the top quartiles and simultaneously shorting the bottom quartiles or the market proxy exhibit statistically significant Jensen’s alpha and continues to accumulate positive returns over the out-of-sample period for both ANN models. On the other hand, the zero-investment portfolios formed by longing the bottom quartiles and simultaneously shorting the market proxy exhibit statistically significant Jensen’s alpha and continues to accumulate losses over the out-of-sample period for both ANN models. The implementation of the extended Kalman filter rule in training artificial neural networks for applications involving noisy financial data is recommended