Lattice dynamical wavelet neural networks implemented using particle swarm optimization for spatio-temporal system identification

In this brief, by combining an efficient wavelet representation with a coupled map lattice model, a new family of adaptive wavelet neural networks, called lattice dynamical wavelet neural networks (LDWNNs), is introduced for spatio-temporal system identification. A new orthogonal projection pursuit (OPP) method, coupled with a particle swarm optimization (PSO) algorithm, is proposed for augmenting the proposed network. A novel two-stage hybrid training scheme is developed for constructing a parsimonious network model. In the first stage, by applying the OPP algorithm, significant wavelet neurons are adaptively and successively recruited into the network, where adjustable parameters of the associated wavelet neurons are optimized using a particle swarm optimizer. The resultant network model, obtained in the first stage, however, may be redundant. In the second stage, an orthogonal least squares algorithm is then applied to refine and improve the initially trained network by removing redundant wavelet neurons from the network. An example for a real spatio-temporal system identification problem is presented to demonstrate the performance of the proposed new modeling framework

Generalised cellular neural networks (GCNNs) constructed using particle swarm optimisation for spatio-temporal evolutionary pattern identification

Particle swarm optimization (PSO) is introduced to implement a new constructive learning algorithm for training generalized cellular neural networks (GCNNs) for the identification of spatio-temporal evolutionary (STE) systems. The basic idea of the new PSO-based learning algorithm is to successively approximate the desired signal by progressively pursuing relevant orthogonal projections. This new algorithm will thus be referred to as the orthogonal projection pursuit (OPP) algorithm, which is in mechanism similar to the conventional projection pursuit approach. A novel two-stage hybrid training scheme is proposed for constructing a parsimonious GCNN model. In the first stage, the orthogonal projection pursuit algorithm is applied to adaptively and successively augment the network, where adjustable parameters of the associated units are optimized using a particle swarm optimizer. The resultant network model produced at the first stage may be redundant. In the second stage, a forward orthogonal regression (FOR) algorithm, aided by mutual information estimation, is applied to re. ne and improve the initially trained network. The effectiveness and performance of the proposed method is validated by applying the new modeling framework to a spatio-temporal evolutionary system identification problem

Generalised additive multiscale wavelet models constructed using particle swarm optimisation and mutual information for spatio-temporal evolutionary system representation

A new class of generalised additive multiscale wavelet models (GAMWMs) is introduced for high dimensional spatio-temporal evolutionary (STE) system identification. A novel two-stage hybrid learning scheme is developed for constructing such an additive wavelet model. In the first stage, a new orthogonal projection pursuit (OPP) method, implemented using a particle swarm optimisation(PSO) algorithm, is proposed for successively augmenting an initial coarse wavelet model, where relevant parameters of the associated wavelets are optimised using a particle swarm optimiser. The resultant network model, obtained in the first stage, may however be a redundant model. In the second stage, a forward orthogonal regression (FOR) algorithm, implemented using a mutual information method, is then applied to refine and improve the initially constructed wavelet model. The proposed two-stage hybrid method can generally produce a parsimonious wavelet model, where a ranked list of wavelet functions, according to the capability of each wavelet to represent the total variance in the desired system output signal is produced. The proposed new modelling framework is applied to real observed images, relative to a chemical reaction exhibiting a spatio-temporal evolutionary behaviour, and the associated identification results show that the new modelling framework is applicable and effective for handling high dimensional identification problems of spatio-temporal evolution sytems

A Projection Pursuit Approach to Cross Country Growth Data

The empirical modeling of the cross-country differences in growth behavior is undoubtedly one of the most predominant research topics in applied macro-econometrics.  However, despite the vast research effort it seems that there are only a few firm conclusions on the sources of cross-country differences.  Unlike the bulk of the literature which focuses on linear parametric models this paper studies a semi-parametric way of modelling.  In particular, it employs projection pursuit regression (PPR) to model the mean regression function of the growth process by a sum of unknown ridge functions (functions of linear combinations of covariates).  PPR model was proposed by Friedman and Stuetzle (1981) to approximate high dimensional functions by simpler functions that operate in low dimensional spaces-typically one-dimensional.  My findings identify non-linear relationships among the basic Solow-type variables.  In particular, initial income and human capital affect growth in a very nonlinear way. Furthermore, there is evidence of interaction effects between human capital and initial income as well as between initial income and population growth rates.   The findings suggest the presence of two steady-state equilibria that classify countries into two groups with different convergence characteristics.

Lattice dynamical wavelet neural networks implemented using particle swarm optimisation for spatio-temporal system identification

Starting from the basic concept of coupled map lattices, a new family of adaptive wavelet neural networks, called lattice dynamical wavelet neural networks (LDWNN), is introduced for spatiotemporal system identification, by combining an efficient wavelet representation with a coupled map lattice model. A new orthogonal projection pursuit (OPP) method, coupled with a particle swarm optimisation (PSO) algorithm, is proposed for augmenting the proposed network. A novel two-stage hybrid training scheme is developed for constructing a parsimonious network model. In the first stage, by applying the orthogonal projection pursuit algorithm, significant wavelet-neurons are adaptively and successively recruited into the network, where adjustable parameters of the associated waveletneurons are optimised using a particle swarm optimiser. The resultant network model, obtained in the first stage, may however be redundant. In the second stage, an orthogonal least squares (OLS) algorithm is then applied to refine and improve the initially trained network by removing redundant wavelet-neurons from the network. The proposed two-stage hybrid training procedure can generally produce a parsimonious network model, where a ranked list of wavelet-neurons, according to the capability of each neuron to represent the total variance in the system output signal is produced. Two spatio-temporal system identification examples are presented to demonstrate the performance of the proposed new modelling framework

Neural networks in geophysical applications

Neural networks are increasingly popular in geophysics. Because they are universal approximators, these tools can approximate any continuous function with an arbitrary precision. Hence, they may yield important contributions to finding solutions to a variety of geophysical applications. However, knowledge of many methods and techniques recently developed to increase the performance and to facilitate the use of neural networks does not seem to be widespread in the geophysical community. Therefore, the power of these tools has not yet been explored to their full extent. In this paper, techniques are described for faster training, better overall performance, i.e., generalization,and the automatic estimation of network size and architecture

Neuro-Fuzzy Computing System with the Capacity of Implementation on Memristor-Crossbar and Optimization-Free Hardware Training

In this paper, first we present a new explanation for the relation between logical circuits and artificial neural networks, logical circuits and fuzzy logic, and artificial neural networks and fuzzy inference systems. Then, based on these results, we propose a new neuro-fuzzy computing system which can effectively be implemented on the memristor-crossbar structure. One important feature of the proposed system is that its hardware can directly be trained using the Hebbian learning rule and without the need to any optimization. The system also has a very good capability to deal with huge number of input-out training data without facing problems like overtraining.Comment: 16 pages, 11 images, submitted to IEEE Trans. on Fuzzy system

Selección de perceptrones multicapa usando aprendizaje bayesiano

La Regularización Bayesiana de perceptrones multicapa pretende resolver el problema de optimización de los pesos de la red neuronal simultáneamente con el problema de generalización. En este trabajo se realiza un análisis de la regularización Bayesiana, que parece ser una de las más poderosas técnicas de entrenamiento de perceptrones multicapa, para luego hacer un comparativo con los resultados obtenidos usando Regla Delta Generalizada. Finalmente se discute alguna implicación de los resultados obtenidos respecto a la técnica basada en algoritmos constructivos para la selección final de neuronas en la capa oculta

Selección de perceptrones multicapa usando aprendizaje bayesiano

La Regularización Bayesiana de perceptrones multicapa pretende resolver el problema de optimización de los pesos de la red neuronal simultáneamente con el problema de generalización. En este trabajo se realiza un análisis de la regularización Bayesiana, que parece ser una de las más poderosas técnicas de entrenamiento de perceptrones multicapa, para luego hacer un comparativo con los resultados obtenidos usando Regla Delta Generalizada. Finalmente se discute alguna implicación de los resultados obtenidos respecto a la técnica basada en algoritmos constructivos para la selección final de neuronas en la capa oculta

Selección de perceptrones multicapa usando aprendizaje bayesiano

La Regularización Bayesiana de perceptrones multicapa pretende resolver el problema de optimización de los pesos de la red neuronal simultáneamente con el problema de generalización. En este trabajo se realiza un análisis de la regularización Bayesiana, que parece ser una de las más poderosas técnicas de entrenamiento de perceptrones multicapa, para luego hacer un comparativo con los resultados obtenidos usando Regla Delta Generalizada. Finalmente se discute alguna implicación de los resultados obtenidos respecto a la técnica basada en algoritmos constructivos para la selección final de neuronas en la capa oculta