An enhanced artificial neural network with a shuffled complex evolutionary global optimization with principal component analysis

The classical Back-Propagation (BP) scheme with gradient-based optimization in training Artificial Neural Networks (ANNs) suffers from many drawbacks, such as the premature convergence, and the tendency of being trapped in local optimums. Therefore, as an alternative for the BP and gradient-based optimization schemes, various Evolutionary Algorithms (EAs), i.e., Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Simulated Annealing (SA), and Differential Evolution (DE), have gained popularity in the field of ANN weight training. This study applied a new efficient and effective Shuffled Complex Evolutionary Global Optimization Algorithm with Principal Component Analysis – University of California Irvine (SP-UCI) to the weight training process of a three-layer feed-forward ANN. A large-scale numerical comparison is conducted among the SP-UCI-, PSO-, GA-, SA-, and DE-based ANNs on 17 benchmark, complex, and real-world datasets. Results show that SP-UCI-based ANN outperforms other EA-based ANNs in the context of convergence and generalization. Results suggest that the SP-UCI algorithm possesses good potential in support of the weight training of ANN in real-word problems. In addition, the suitability of different kinds of EAs on training ANN is discussed. The large-scale comparison experiments conducted in this paper are fundamental references for selecting proper ANN weight training algorithms in practice

Freeze-drying modeling and monitoring using a new neuro-evolutive technique

This paper is focused on the design of a black-box model for the process of freeze-drying of pharmaceuticals. A new methodology based on a self-adaptive differential evolution scheme is combined with a back-propagation algorithm, as local search method, for the simultaneous structural and parametric optimization of the model represented by a neural network. Using the model of the freeze-drying process, both the temperature and the residual ice content in the product vs. time can be determine off-line, given the values of the operating conditions (the temperature of the heating shelf and the pressure in the drying chamber). This makes possible to understand if the maximum temperature allowed by the product is trespassed and when the sublimation drying is complete, thus providing a valuable tool for recipe design and optimization. Besides, the black box model can be applied to monitor the freeze-drying process: in this case, the measurement of product temperature is used as input variable of the neural network in order to provide in-line estimation of the state of the product (temperature and residual amount of ice). Various examples are presented and discussed, thus pointing out the strength of the too

Training a Feed-forward Neural Network with Artificial Bee Colony Based Backpropagation Method

Back-propagation algorithm is one of the most widely used and popular techniques to optimize the feed forward neural network training. Nature inspired meta-heuristic algorithms also provide derivative-free solution to optimize complex problem. Artificial bee colony algorithm is a nature inspired meta-heuristic algorithm, mimicking the foraging or food source searching behaviour of bees in a bee colony and this algorithm is implemented in several applications for an improved optimized outcome. The proposed method in this paper includes an improved artificial bee colony algorithm based back-propagation neural network training method for fast and improved convergence rate of the hybrid neural network learning method. The result is analysed with the genetic algorithm based back-propagation method, and it is another hybridized procedure of its kind. Analysis is performed over standard data sets, reflecting the light of efficiency of proposed method in terms of convergence speed and rate.Comment: 14 Pages, 11 figure

Limited Evaluation Cooperative Co-evolutionary Differential Evolution for Large-scale Neuroevolution

Many real-world control and classification tasks involve a large number of features. When artificial neural networks (ANNs) are used for modeling these tasks, the network architectures tend to be large. Neuroevolution is an effective approach for optimizing ANNs; however, there are two bottlenecks that make their application challenging in case of high-dimensional networks using direct encoding. First, classic evolutionary algorithms tend not to scale well for searching large parameter spaces; second, the network evaluation over a large number of training instances is in general time-consuming. In this work, we propose an approach called the Limited Evaluation Cooperative Co-evolutionary Differential Evolution algorithm (LECCDE) to optimize high-dimensional ANNs. The proposed method aims to optimize the pre-synaptic weights of each post-synaptic neuron in different subpopulations using a Cooperative Co-evolutionary Differential Evolution algorithm, and employs a limited evaluation scheme where fitness evaluation is performed on a relatively small number of training instances based on fitness inheritance. We test LECCDE on three datasets with various sizes, and our results show that cooperative co-evolution significantly improves the test error comparing to standard Differential Evolution, while the limited evaluation scheme facilitates a significant reduction in computing time

Robust learning with implicit residual networks

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power and allows to control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique and provide numerical results in support of our findings

Multivariate time series analysis for short-term forecasting of ground level ozone (O3) in Malaysia

The declining of air quality mostly affects the elderly, children, people with asthma, as well as a restriction on outdoor activities. Therefore, there is an importance to provide a statistical modelling to forecast the future values of surface layer ozone (O3) concentration. The objectives of this study are to obtain the best multivariate time series (MTS) model and develop an online air quality forecasting system for O3 concentration in Malaysia. The implementations of MTS model improve the recent statistical model on air quality for short-term prediction. Ten air quality monitoring stations situated at four (4) different types of location were selected in this study. The first type is industrial represent by Pasir Gudang, Perai, and Nilai, second type is urban represent by Kuala Terengganu, Kota Bharu, and Alor Setar. The third is suburban located in Banting, Kangar, and Tanjung Malim, also the only background station at Jerantut. The hourly record data from 2010 to 2017 were used to assess the characteristics and behaviour of O3 concentration. Meanwhile, the monthly record data of O3, particulate matter (PM10), nitrogen dioxide (NO2), sulphur dioxide (SO2), carbon monoxide (CO), temperature (T), wind speed (WS), and relative humidity (RH) were used to examine the best MTS models. Three methods of MTS namely vector autoregressive (VAR), vector moving average (VMA), and vector autoregressive moving average (VARMA), has been applied in this study. Based on the performance error, the most appropriate MTS model located in Pasir Gudang, Kota Bharu and Kangar is VAR(1), Kuala Terengganu and Alor Setar for VAR(2), Perai and Nilai for VAR(3), Tanjung Malim for VAR(4) and Banting for VAR(5). Only Jerantut obtained the VMA(2) as the best model. The lowest root mean square error (RMSE) and normalized absolute error is 0.0053 and <0.0001 which is for MTS model in Perai and Kuala Terengganu, respectively. Meanwhile, for mean absolute error (MAE), the lowest is in Banting and Jerantut at 0.0013. The online air quality forecasting system for O3 was successfully developed based on the best MTS models to represent each monitoring station