Learning Nonlinear Functions with MLPs and SRNs

In this paper, nonlinear functions generated by randomly initialized multilayer perceptrons (MLPs) and simultaneous recurrent neural networks (SRNs) are learned by MLPs and SRNs. Training SRNs is a challenging task and a new learning algorithm - DEPSO is introduced. DEPSO is a standard particle swarm optimization (PSO) algorithm with the addition of a differential evolution step to aid in swarm convergence. The results from DEPSO are compared with the standard backpropagation (BP) and PSO algorithms. It is further verified that functions generated by SRNs are harder to learn than those generated by MLPs but DEPSO provides better learning capabilities for the functions generated by MLPs and SRNs as compared to BP and PSO. These three algorithms are also trained on several benchmark functions to confirm results

Learning Functions Generated by Randomly Initialized MLPs and SRNs

In this paper, nonlinear functions generated by randomly initialized multilayer perceptrons (MLPs) and simultaneous recurrent neural networks (SRNs) and two benchmark functions are learned by MLPs and SRNs. Training SRNs is a challenging task and a new learning algorithm - PSO-QI is introduced. PSO-QI is a standard particle swarm optimization (PSO) algorithm with the addition of a quantum step utilizing the probability density property of a quantum particle. The results from PSO-QI are compared with the standard backpropagation (BP) and PSO algorithms. It is further verified that functions generated by SRNs are harder to learn than those generated by MLPs but PSO-QI provides learning capabilities of these functions by MLPs and SRNs compared to BP and PSO

New Neural Network Design for Approximate Dynamic Programming and Optimal Multiuser Detection

In this thesis we demonstrate that a new neural network design can be used to solve a class of difficult function approximation problems which are crucial to the field of approximate dynamic programming (ADP). Although conventional neural networks have been proven to approximate smooth functions very well, the use of ADP for problems of intelligent control or planning requires the approximation of functions which are not so smooth. As an example, this thesis studies the problem of approximating the $J$ function of dynamic programming applied to the task of navigating mazes, in general, without the need to learn each individual maze. Conventional neural networks, like multi-layer perceptrons (MLPs), cannot learn this task. But a new type of neural network, simultaneous recurrent networks (SRNs), can accomplish the required learning as demonstrated by successful initial tests. In this thesis we also investigate the ability of recurrent neural networks to approximate MLPs and vice versa. Moreover, we present a comparison between using SRNs and MLPs to implement the optimal CDMA multiuser detector (OMD). This example is intended to demonstrate that SRNs can provide fast suboptimal solutions to hard combinatorial optimization problems, and achieve better bit- error-rate (BER) performance than MLPs

Cellular Simultanous Recurrent Networks for Image Processing

Artificial neural networks are inspired by the abilities of humans and animals to learn and adapt. Feed-forward networks are both fast and powerful, and are particularly useful for statistical pattern recognition. These networks are inspired by portions of the brain such as the visual cortex. However, feed-forward networks have been shown inadequate for complex applications such as long-term optimization, reinforced learning and image processing. Cellular Neural Networks (CNNs) are a type of recurrent network which have been used extensively for image processing. CNNs have shown limited success solving problems which involve topological relationships. Such problems include geometric transformations such as affine transformation and image registration. The Cellular Simultaneous Recurrent Network (CSRN) has been exploited to solve the 2D maze traversal problem, which is a long-term optimization problem with similar topological relations. From its inception, it has been speculated that the CSRN may have important implications in image processing. However, to date, very little work has been done to study CSRNs for image processing tasks. In this work, we investigate CSRNs for image processing. We propose a novel, generalized architecture for the CSRN suitable for generic image processing tasks. This architecture includes the use of sub-image processing which greatly improves the efficacy of CSRNs for image processing. We demonstrate the application of the CSRN with this generalized architecture across a variety of image processing problems including pixel level transformations, filtering, and geometric transformations. Results are evaluated and compared with standard MATLAB® functions. To better understand the inner workings of the CSRN we investigate the use of various CSRN cores including: 1) the original Generalized Multi-Layered Perceptron (GMLP) core used by Pang and Werbos to solve the 2D maze traversal problem, 2) the Elman Simultaneous Recurrent Network (ESRN), and 3) a novel ESRN core with multi-layered feedback. We compare the functionality of these cores in image processing applications. Further, we introduce the application of the unscented Kalman filter (UKF) for training of the CSRN. Results are compared with the standard Extended Kalman Filter (EKF) training method of CSRN. Finally, implications of current findings and proposed research directions are presented

A PSO with Quantum Infusion Algorithm for Training Simultaneous Recurrent Neural Networks

Simultaneous recurrent neural network (SRN) is one of the most powerful neural network architectures well suited for estimation and control of complex time varying nonlinear dynamic systems. SRN training is a difficult problem especially if multiple inputs and multiple outputs (MIMO) are involved. Particle swarm optimization with quantum infusion (PSO-QI) is introduced in this paper for training such SRNs. In order to illustrate the capability of the PSO-QI training algorithm, a wide area monitor (WAM) for a power system is developed using a multiple inputs multiple outputs Elman SRN. The SRN estimates speed deviations of four generators in a multimachine power system. Since MIMO structured SRNs are hard to train, a two step approach for training is presented with PSO-QI. The performance of PSO-QI is compared to that of the standard PSO algorithm. Results demonstrate that the SRN trained with the PSO-QI in the two step approach tracks the speed deviations of the generators with the minimum error

Neural apparent BRDF fields for multiview photometric stereo

We propose to tackle the multiview photometric stereo problem using an extension of Neural Radiance Fields (NeRFs), conditioned on light source direction. The geometric part of our neural representation predicts surface normal direction, allowing us to reason about local surface reflectance. The appearance part of our neural representation is decomposed into a neural bidirectional reflectance function (BRDF), learnt as part of the fitting process, and a shadow prediction network (conditioned on light source direction) allowing us to model the apparent BRDF. This balance of learnt components with inductive biases based on physical image formation models allows us to extrapolate far from the light source and viewer directions observed during training. We demonstrate our approach on a multiview photometric stereo benchmark and show that competitive performance can be obtained with the neural density representation of a NeRF.Comment: 9 pages, 6 figures, 1 tabl

From ADP to the Brain: Foundations, Roadmap, Challenges and Research Priorities

This paper defines and discusses Mouse Level Computational Intelligence (MLCI) as a grand challenge for the coming century. It provides a specific roadmap to reach that target, citing relevant work and review papers and discussing the relation to funding priorities in two NSF funding activities: the ongoing Energy, Power and Adaptive Systems program (EPAS) and the recent initiative in Cognitive Optimization and Prediction (COPN). It elaborates on the first step, vector intelligence, a challenge in the development of universal learning systems, which itself will require considerable new research to attain. This in turn is a crucial prerequisite to true functional understanding of how mammal brains achieve such general learning capabilities.Comment: 5p, to appear in Proc. of the Interantional Joint Conference on Neural Networks 201

The Unstoppable Rise of Computational Linguistics in Deep Learning

In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the challenges facing research in deep learning architectures for natural language understanding.Comment: 13 pages. Accepted for publication at ACL 2020, in the theme trac

Does money matter in inflation forecasting?.

This paper provides the most fully comprehensive evidence to date on whether or not monetary aggregates are valuable for forecasting US inflation in the early to mid 2000s. We explore a wide range of different definitions of money, including different methods of aggregation and different collections of included monetary assets. In our forecasting experiment we use two non-linear techniques, namely, recurrent neural networks and kernel recursive least squares regression - techniques that are new to macroeconomics. Recurrent neural networks operate with potentially unbounded input memory, while the kernel regression technique is a finite memory predictor. The two methodologies compete to find the best fitting US inflation forecasting models and are then compared to forecasts from a naive random walk model. The best models were non-linear autoregressive models based on kernel methods. Our findings do not provide much support for the usefulness of monetary aggregates in forecasting inflation