Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Conditioning of extreme learning machine for noisy data using heuristic optimization

This article provides a tool that can be used in the exact sciences to obtain good approximations to reality when noisy data is inevitable. Two heuristic optimization algorithms are implemented: Simulated Annealing and Particle Swarming for the determination of the extreme learning machine output weights. The first operates in a large search space and at each iteration it probabilistically decides between staying at its current state or moving to another. The swarm of particles, it optimizes a problem from a population of candidate solutions, moving them throughout the search space according to position and speed. The methodology consists of building data sets around a polynomial function, implementing the heuristic algorithms and comparing the errors with the traditional computation method using the Moore–Penrose inverse. The results show that the heuristic optimization algorithms implemented improve the estimation of the output weights when the input have highly noisy data

A simulation data-driven design approach for rapid product optimization

Traditional design optimization is an iterative process of design, simulation, and redesign, which requires extensive calculations and analysis. The designer needs to adjust and evaluate the design parameters manually and continually based on the simulation results until a satisfactory design is obtained. However, the expensive computational costs and large resource consumption of complex products hinder the wide application of simulation in industry. It is not an easy task to search the optimal design solution intelligently and efficiently. Therefore, a simulation data-driven design approach which combines dynamic simulation data mining and design optimization is proposed to achieve this purpose in this study. The dynamic simulation data mining algorithm—on-line sequential extreme learning machine with adaptive weights (WadaptiveOS-ELM)—is adopted to train the dynamic prediction model to effectively evaluate the merits of new design solutions in the optimization process. Meanwhile, the prediction model is updated incrementally by combining new “good” data set to reduce the modeling cost and improve the prediction accuracy. Furthermore, the improved heuristic optimization algorithm—adaptive and weighted center particle swarm optimization (AWCPSO)—is introduced to guide the design change direction intelligently to improve the search efficiency. In this way, the optimal design solution can be searched automatically with less actual simulation iterations and higher optimization efficiency, and thus supporting the rapid product optimization effectively. The experimental results demonstrate the feasibility and effectiveness of the proposed approach

Radar Target Classification Using an Evolutionary Extreme Learning Machine Based on Improved Quantum-Behaved Particle Swarm Optimization

A novel evolutionary extreme learning machine (ELM) based on improved quantum-behaved particle swarm optimization (IQPSO) for radar target classification is presented in this paper. Quantum-behaved particle swarm optimization (QPSO) has been used in ELM to solve the problem that ELM needs more hidden nodes than conventional tuning-based learning algorithms due to the random set of input weights and hidden biases. But the method for calculating the characteristic length of Delta potential well of QPSO may reduce the global search ability of the algorithm. To solve this issue, a new method to calculate the characteristic length of Delta potential well is proposed in this paper. Experimental results based on the benchmark functions validate the better performance of IQPSO against QPSO in most cases. The novel algorithm is also evaluated by using real-world datasets and radar data; the experimental results indicate that the proposed algorithm is more effective than BP, SVM, ELM, QPSO-ELM, and so on, in terms of real-time performance and accuracy

IMPLEMENTASI ALGORITMA EXTREME LEARNING MACHINE PADA PREDIKSI AKTIVITAS BADAI GEOMAGNETIK

Badai geomagnetik merupakan gangguan yang terjadi di magnetosfer bumi, akibat adanya aktivitas dari matahari. Dalam rangka peringatan dini, Lembaga Penerbangan dan Antariksa Nasional (LAPAN) di Indonesia memiliki kegiatan rutin untuk memprediksi kemungkinan terjadinya badai tersebut dalam rentang waktu 24 jam ke depan. Namun pada tahun 2015, hasil prediksi badai geomagnetik yang dilakukan secara manual oleh LAPAN hanya mendapatkan akurasi sebesar 57,14%. Oleh karena itu, penelitian ini mengusulkan pemanfaatan metode Extreme Learning Machine (ELM) dalam melakukan prediksi badai geomagnetik, dengan tujuan untuk mendapatkan akurasi yang lebih baik. Data penelitian yang digunakan meliputi data coronal hole, coronal mass ejection, solar wind dan indeks Dst pada tahun 2011 hingga 2016. Hasil penelitian ini menunjukkan bahwa algoritma ELM memiliki tingkat akurasi yang lebih besar dalam memprediksi badai geomagnetik tahun 2015, dengan perolehan nilai 57,80822%. Meskipun memiliki selisih akurasi yang kecil, namun pemanfaatan ELM ini dapat membantu prediksi badai geomagnetik secara otomatis. Secara umum, algoritma ELM yang dibangun dalam penelitian ini memiliki nilai rata-rata akurasi prediksi tertinggi sebesar 69,9055%.---------- The geomagnetic storm is a disturbance that occurs in the earth’s magnetosphere, as the result of the activity of the sun. In case for early warning, National Institute of Aeronautics and Space Agency (LAPAN) in Indonesia has a routine activity to predict the probability of geomagnetic storm appearance for the next 24 hours. But in 2015, the geomagnetic storm prediction results are done manually just managed to get the accuracy of 57.14%. Therefore, this research proposes the utilization method of Extreme Learning Machine (ELM) for geomagnetic storm prediction, in order to get better accuracy. Research data that used include data on coronal holes, coronal mass ejection, solar wind and the Dst index from 2011 to 2016. The results of this research show that the ELM algorithm has a greater accuracy in prediction the 2015 geomagnetic storm activity, with the acquisition of 57.80822% value. Despite the difference in accuracy is small, but the utilization of ELM can help predicting geomagnetic storm automatically. In general, the ELM algorithm built in this research have the average value of the highest prediction accuracy of 69.9055%

Generalised additive multiscale wavelet models constructed using particle swarm optimisation and mutual information for spatio-temporal evolutionary system representation

A new class of generalised additive multiscale wavelet models (GAMWMs) is introduced for high dimensional spatio-temporal evolutionary (STE) system identification. A novel two-stage hybrid learning scheme is developed for constructing such an additive wavelet model. In the first stage, a new orthogonal projection pursuit (OPP) method, implemented using a particle swarm optimisation(PSO) algorithm, is proposed for successively augmenting an initial coarse wavelet model, where relevant parameters of the associated wavelets are optimised using a particle swarm optimiser. The resultant network model, obtained in the first stage, may however be a redundant model. In the second stage, a forward orthogonal regression (FOR) algorithm, implemented using a mutual information method, is then applied to refine and improve the initially constructed wavelet model. The proposed two-stage hybrid method can generally produce a parsimonious wavelet model, where a ranked list of wavelet functions, according to the capability of each wavelet to represent the total variance in the desired system output signal is produced. The proposed new modelling framework is applied to real observed images, relative to a chemical reaction exhibiting a spatio-temporal evolutionary behaviour, and the associated identification results show that the new modelling framework is applicable and effective for handling high dimensional identification problems of spatio-temporal evolution sytems

Radial Basis Function Neural Networks : A Review

Radial Basis Function neural networks (RBFNNs) represent an attractive alternative to other neural network models. One reason is that they form a unifying link between function approximation, regularization, noisy interpolation, classification and density estimation. It is also the case that training RBF neural networks is faster than training multi-layer perceptron networks. RBFNN learning is usually split into an unsupervised part, where center and widths of the Gaussian basis functions are set, and a linear supervised part for weight computation. This paper reviews various learning methods for determining centers, widths, and synaptic weights of RBFNN. In addition, we will point to some applications of RBFNN in various fields. In the end, we name software that can be used for implementing RBFNNs

Optimizing Weights And Biases in MLP Using Whale Optimization Algorithm

Artificial Neural Networks are intelligent and non-parametric mathematical models inspired by the human nervous system. They have been widely studied and applied for classification, pattern recognition and forecasting problems. The main challenge of training an Artificial Neural network is its learning process, the nonlinear nature and the unknown best set of main controlling parameters (weights and biases). When the Artificial Neural Networks are trained using the conventional training algorithm, they get caught in the local optima stagnation and slow convergence speed; this makes the stochastic optimization algorithm a definitive alternative to alleviate the drawbacks. This thesis proposes an algorithm based on the recently proposed Whale Optimization Algorithm(WOA). The algorithm has proven to solve a wide range of optimization problems and outperform existing algorithms. The successful implementation of this algorithm motivated our attempts to benchmark its performance in training feed-forward neural networks. We have taken a set of 20 datasets with different difficulty levels and tested the proposed WOA-MLP based trainer. Further, the results are verified by comparing WOA-MLP with the back propagation algorithms and six evolutionary techniques. The results have proved that the proposed trainer can outperform the current algorithms on the majority of datasets in terms of local optima avoidance and convergence speed