Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Estimation of physical variables from multichannel remotely sensed imagery using a neural network: Application to rainfall estimation

Satellite-based remotely sensed data have the potential to provide hydrologically relevant information about spatially and temporally varying physical variables. A methodology for estimating such variables from multichannel remotely sensed data is presented; the approach is based on a modified counterpropagation neural network (MCPN) and is both effective and efficient at building complex nonlinear input-output function mappings from large amounts of data. An application to high-resolution estimation of the spatial and temporal variation of surface rainfall using geostationary satellite infrared and visible imagery is presented. Test results also indicate that spatially and temporally sparse ground-based observations can be assimilated via an adaptive implementation of the MCPN method, thereby allowing on-line improvement of the estimates

Control of Complex Dynamic Systems by Neural Networks

This paper considers the use of neural networks (NN's) in controlling a nonlinear, stochastic system with unknown process equations. The NN is used to model the resulting unknown control law. The approach here is based on using the output error of the system to train the NN controller without the need to construct a separate model (NN or other type) for the unknown process dynamics. To implement such a direct adaptive control approach, it is required that connection weights in the NN be estimated while the system is being controlled. As a result of the feedback of the unknown process dynamics, however, it is not possible to determine the gradient of the loss function for use in standard (back-propagation-type) weight estimation algorithms. Therefore, this paper considers the use of a new stochastic approximation algorithm for this weight estimation, which is based on a 'simultaneous perturbation' gradient approximation that only requires the system output error. It is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite-difference gradient approximations

A survey on modern trainable activation functions

In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.Comment: Published in "Neural Networks" journal (Elsevier

Exploring the use of neural network-based band selection on hyperspectral imagery to identify informative wavelengths for improving classifier task performance

Hyperspectral imagery is a highly dimensional type of data resulting in high computational costs during analysis. Band selection aims to reduce the original hyperspectral image to a smaller subset that reduces these costs while preserving the maximum amount of spectral information within the data. This thesis explores various types of band selection techniques used in hyperspectral image processing. Modifying Neural network-based techniques and observing the effects on the band selection process due to the change in network architecture or objective are of particular focus in this thesis. Herein, a generalized neural network-based band selection technique is developed and compared to state-of-the-art algorithms that are applied to a unique dataset and the Pavia City Center dataset where the subsequent selected bands are fed into a classifier to gather comparison results