Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Development Of An Intelligent System For River Water Quality Classification Based On Algae Composition

Throughout the years, many researches have been conducted on the potential applications of Artificial Intelligence (AI) in the biological monitoring of river quality. This project will provide an overview regarding the feasibility of the application of neural networks for direct classification of river water quality based on algae composition. A brief introduction to neural networks and the suitability of neural network for use in river water quality determination will be investigated. In this project, several neural networks will be developed and their performance are compared to yield the most suitable network that will be used to model the classification system for determination of river water quality based on algae composition. Among the types of neural network that will be developed are Multilayer Perceptron network (MLP), Radial Basis Function (RBF) network and Hybrid Multilayer Perceptron (HMLP) network. This study proves that the HMLP network trained using the MRPE algorithm achieves the best performance as compared to the MLP and RBF network. The HMLP network produces 90% accuracy. In this study, an intelligent system is developed for the classification of river water quality using the HMLP network. The proposed system provides several advantages in terms of its applicability, high accuracy, user-friendliness and as well as yields faster results compared to conventional system

ECG Classification with an Adaptive Neuro-Fuzzy Inference System

Heart signals allow for a comprehensive analysis of the heart. Electrocardiography (ECG or EKG) uses electrodes to measure the electrical activity of the heart. Extracting ECG signals is a non-invasive process that opens the door to new possibilities for the application of advanced signal processing and data analysis techniques in the diagnosis of heart diseases. With the help of today’s large database of ECG signals, a computationally intelligent system can learn and take the place of a cardiologist. Detection of various abnormalities in the patient’s heart to identify various heart diseases can be made through an Adaptive Neuro-Fuzzy Inference System (ANFIS) preprocessed by subtractive clustering. Six types of heartbeats are classified: normal sinus rhythm, premature ventricular contraction (PVC), atrial premature contraction (APC), left bundle branch block (LBBB), right bundle branch block (RBBB), and paced beats. The goal is to detect important characteristics of an ECG signal to determine if the patient’s heartbeat is normal or irregular. The results from three trials indicate an average accuracy of 98.10%, average sensitivity of 94.99%, and average specificity of 98.87%. These results are comparable to two artificial neural network (ANN) algorithms: gradient descent and Levenberg Marquardt, as well as the ANFIS preprocessed by grid partitioning

Adaptive Estimation and Heuristic Optimization of Nonlinear Spacecraft Attitude Dynamics

For spacecraft conducting on-orbit operations, changes to the structure of the spacecraft are not uncommon. These planned or unanticipated changes in inertia properties couple with the spacecraft\u27s attitude dynamics and typically require estimation. For systems with time-varying inertia parameters, multiple model adaptive estimation (MMAE) routines can be utilized for parameter and state estimates. MMAE algorithms involve constructing a bank of recursive estimators, each assuming a different hypothesis for the systems dynamics. This research has three distinct, but related, contributions to satellite attitude dynamics and estimation. In the first part of this research, MMAE routines employing parallel banks of unscented attitude filters are applied to analytical models of spacecraft with time-varying mass moments of inertia (MOI), with the objective of estimating the MOI and classifying the spacecraft\u27s behavior. New adaptive estimation techniques were either modified or developed that can detect discontinuities in MOI up to 98 of the time in the specific problem scenario.Second, heuristic optimization techniques and numerical methods are applied to Wahba\u27s single-frame attitude estimation problem,decreasing computation time by an average of nearly 67 . Finally, this research poses MOI estimation as an ODE parameter identification problem, achieving successful numerical estimates through shooting methods and exploiting the polhodes of rigid body motion with results, on average, to be within 1 to 5 of the true MOI values

Noise Induces Biased Estimation of the Correction Gain

The detection of an error in the motor output and the correction in the next movement are critical components of any form of motor learning. Accordingly, a variety of iterative learning models have assumed that a fraction of the error is adjusted in the next trial. This critical fraction, the correction gain, learning rate, or feedback gain, has been frequently estimated via least-square regression of the obtained data set. Such data contain not only the inevitable noise from motor execution, but also noise from measurement. It is generally assumed that this noise averages out with large data sets and does not affect the parameter estimation. This study demonstrates that this is not the case and that in the presence of noise the conventional estimate of the correction gain has a significant bias, even with the simplest model. Furthermore, this bias does not decrease with increasing length of the data set. This study reveals this limitation of current system identification methods and proposes a new method that overcomes this limitation. We derive an analytical form of the bias from a simple regression method (Yule-Walker) and develop an improved identification method. This bias is discussed as one of other examples for how the dynamics of noise can introduce significant distortions in data analysis.OAIID:RECH_ACHV_DSTSH_NO:T201700064RECH_ACHV_FG:RR00200001ADJUST_YN:EMP_ID:A080358CITE_RATE:3.057FILENAME:journal.pone.0158466.PDFDEPT_NM:체육교육과EMAIL:[email protected]_YN:YFILEURL:https://srnd.snu.ac.kr/eXrepEIR/fws/file/99ceacea-ee11-4779-89ed-79b478e92d46/linkCONFIRM:

A generalised feedforward neural network architecture and its applications to classification and regression

Shunting inhibition is a powerful computational mechanism that plays an important role in sensory neural information processing systems. It has been extensively used to model some important visual and cognitive functions. It equips neurons with a gain control mechanism that allows them to operate as adaptive non-linear filters. Shunting Inhibitory Artificial Neural Networks (SIANNs) are biologically inspired networks where the basic synaptic computations are based on shunting inhibition. SIANNs were designed to solve difficult machine learning problems by exploiting the inherent non-linearity mediated by shunting inhibition. The aim was to develop powerful, trainable networks, with non-linear decision surfaces, for classification and non-linear regression tasks. This work enhances and extends the original SIANN architecture to a more general form called the Generalised Feedforward Neural Network (GFNN) architecture, which contains as subsets both SIANN and the conventional Multilayer Perceptron (MLP) architectures. The original SIANN structure has the number of shunting neurons in the hidden layers equal to the number of inputs, due to the neuron model that is used having a single direct excitatory input. This was found to be too restrictive, often resulting in inadequately small or inordinately large network structures

Applications of neural networks to control systems

Tese de dout., Engenharia Electrónica, School of Electronic Engineering Science, Univ. of Wales, Bangor, 1992This work investigates the applicability of artificial neural networks to control systems. The following properties of neural networks are identified as of major interest to this field: their ability to implement nonlinear mappings, their massively parallel structure and their capacity to adapt. Exploiting the first feature, a new method is proposed for PID autotuning. Based on integral measures of the open or closed loop step response, multilayer perceptrons (MLPs) are used to supply PID parameter values to a standard PID controller. Before being used on-line, the MLPs are trained offline, to provide PID parameter values based on integral performance criteria. Off-line simulations, where a plant with time-varying parameters and time varying transfer function is considered, show that well damped responses are obtained. The neural PID autotuner is subsequently implemented in real-time. Extensive experimentation confirms the good results obtained in the off-line simulations. To reduce the training time incurred when using the error back-propagation algorithm, three possibilities are investigated. A comparative study of higherorder methods of optimization identifies the Levenberg-Marquardt (LM)algorithm as the best method. When used for function approximation purposes, the neurons in the output layer of the MLPs have a linear activation function. Exploiting this linearity, the standard training criterion can be replaced by a new, yet equivalent, criterion. Using the LM algorithm to minimize this new criterion, together with an alternative form of Jacobian matrix, a new learning algorithm is obtained. This algorithm is subsequently parallelized. Its main blocks of computation are identified, separately parallelized, and finally connected together. The training time of MLPs is reduced by a factor greater than 70 executing the new learning algorithm on 7 Inmos transputers

Optimization Algorithms for Machine Learning Designed for Parallel and Distributed Environments

This thesis proposes several optimization methods that utilize parallel algorithms for large-scale machine learning problems. The overall theme is network-based machine learning algorithms; in particular, we consider two machine learning models: graphical models and neural networks. Graphical models are methods categorized under unsupervised machine learning, aiming at recovering conditional dependencies among random variables from observed samples of a multivariable distribution. Neural networks, on the other hand, are methods that learn an implicit approximation to underlying true nonlinear functions based on sample data and utilize that information to generalize to validation data. The goal of finding the best methods relies on an optimization problem tasked with training such models. Improvements in current methods of solving the optimization problem for graphical models are obtained by parallelization and the use of a new update and a new step-size selection rule in the coordinate descent algorithms designed for large-scale problems. For training deep neural networks, we consider the second-order optimization algorithms within trust-region-like optimization frameworks. Deep networks are represented using large-scale vectors of weights and are trained based on very large datasets. Hence, obtaining second-order information is very expensive for these networks. In this thesis, we undertake an extensive exploration of algorithms that use a small number of curvature evaluations and are hence faster than other existing methods

Dynamic payload estimation in four wheel drive loaders

Knowledge of the mass of the manipulated load (i.e. payload) in off-highway machines is useful information for a variety of reasons ranging from knowledge of machine stability to ensuring compliance with transportion regulations. This knowledge is difficult to ascertain however. This dissertation concerns itself with delineating the motivations for, and difficulties in development of a dynamic payload weighing algorithm. The dissertation will describe how the new type of dynamic payload weighing algorithm was developed and progressively overcame some of these difficulties. The payload mass estimate is dependent upon many different variables within the off-highway vehicle. These variables include static variability such as machining tolerances of the revolute joints in the linkage, mass of the linkage members, etc as well as dynamic variability such as whole-machine accelerations, hydraulic cylinder friction, pin joint friction, etc. Some initial effort was undertaken to understand the static variables in this problem first by studying the effects of machining tolerances on the working linkage kinematics in a four-wheel-drive loader. This effort showed that if the linkage members were machined within the tolerances prescribed by the design of the linkage components, the tolerance stack-up of the machining variability had very little impact on overall linkage kinematics. Once some of the static dependent variables were understood in greater detail significant effort was undertaken to understand and compensate for the dynamic dependent variables of the estimation problem. The first algorithm took a simple approach of using the kinematic linkage model coupled with hydraulic cylinder pressure information to calculate a payload estimate directly. This algorithm did not account for many of the aforementioned dynamic variables (joint friction, machine acceleration, etc) but was computationally expedient. This work however produced payload estimates with error far greater than the 1% full scale value being targeted. Since this initial simplistic effort met with failure, a second algorithm was needed. The second algorithm was developed upon the information known about the limitations of the first algorithm. A suitable method of compensating for the non-linear dependent dynamic variables was needed. To address this dilemma, an artificial neural network approach was taken for the second algorithm. The second algorithm’s construction was to utilise an artificial neural network to capture the kinematic linkage characteristics and all other dynamic dependent variable behaviour and estimate the payload information based upon the linkage position and hydraulic cylinder pressures. This algorithm was trained using emperically collected data and then subjected to actual use in the field. This experiment showed that that the dynamic complexity of the estimation problem was too large for a small (and computationally feasible) artificial neural network to characterize such that the error estimate was less than the 1% full scale requirement. A third algorithm was required due to the failures of the first two. The third algorithm was constructed to ii take advantage of the kinematic model developed and utilise the artificial neural network’s ability to perform nonlinear mapping. As such, the third algorithm developed uses the kinematic model output as an input to the artificial neural network. This change from the second algorithm keeps the network from having to characterize the linkage kinematics and only forces the network to compensate for the dependent dynamic variables excluded by the kinematic linkage model. This algorithm showed significant improvement over the previous two but still did not meet the required 1% full scale requirement. The promise shown by this algorithm however was convincing enough that further effort was spent in trying to refine it to improve the accuracy. The fourth algorithm developed proceeded with improving the third algorithm. This was accomplished by adding additional inputs to the artificial neural network that allowed the network to better compensate for the variables present in the problem. This effort produced an algorithm that, when subjected to actual field use, produced results very near the 1% full scale accuracy requirement. This algorithm could be improved upon slightly with better input data filtering and possibly adding additional network inputs. The final algorithm produced results very near the desired accuracy. This algorithm was also novel in that for this estimation, the artificial neural network was not used soley as the means to characterize the problem for estimation purposes. Instead, much of the responsibility for the mathematical characterization of the problem was placed upon a kinematic linkage model that then fed it’s own payload estimate into the neural network where the estimate was further refined during network training with calibration data and additional inputs. This method of nonlinear state estimation (i.e. utilising a neural network to compensate for nonlinear effects in conjunction with a first principles model) has not been seen previously in the literature